You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by R W <ai...@gmail.com> on 2014/04/11 09:58:42 UTC

Is it possible for Flume-NG to transfer binary files?

Hi All

We have an App will generate huge amount of binary based file, it's not
like text file which we may have new line separator for it, we want to
collect these binary files and store them into a Hadoop cluster, I'm new to
Flume, so could anyone share some ideas here on how to do it with Flume,
thanks in advance.

Cheers
aij

Re: Is it possible for Flume-NG to transfer binary files?

Posted by Ehsan ul Haq <m....@gmail.com>.
What is the use case? Where in hadoop you want to store? Where is the
source coming from is it in files?

What is the issue with BlobDeserializer?

Ehsan
On 11 Apr 2014 12:07, "R W" <ai...@gmail.com> wrote:

> Hi Ehsan
>
> Thanks for your quick responding, is it BlobDeserializer the only way to
> go? Thanks.
>
> Cheers
> aij
>
>
> On Fri, Apr 11, 2014 at 4:27 PM, Ehsan ul Haq <m....@gmail.com>wrote:
>
>> Yes it is possible to transfer binary files into Hadoop (HDFS/Hbase etc)
>> using flume.
>> If your binary files are already being stored in some local file system
>> in some folders then you can use SpoolingDirectory source "
>> https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source".
>> The default deserializer is a LINE based text deserializer which is not for
>> binary data. You can use BlobDeserializer "
>> https://flume.apache.org/FlumeUserGuide.html#blobdeserializer" which can
>> handle binary data.
>>
>> Cheers
>> Ehsan
>>
>>
>> On Fri, Apr 11, 2014 at 9:58 AM, R W <ai...@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> We have an App will generate huge amount of binary based file, it's not
>>> like text file which we may have new line separator for it, we want to
>>> collect these binary files and store them into a Hadoop cluster, I'm new to
>>> Flume, so could anyone share some ideas here on how to do it with Flume,
>>> thanks in advance.
>>>
>>> Cheers
>>> aij
>>>
>>
>>
>

Re: Is it possible for Flume-NG to transfer binary files?

Posted by R W <ai...@gmail.com>.
Hi Ehsan

Thanks for your quick responding, is it BlobDeserializer the only way to
go? Thanks.

Cheers
aij


On Fri, Apr 11, 2014 at 4:27 PM, Ehsan ul Haq <m....@gmail.com> wrote:

> Yes it is possible to transfer binary files into Hadoop (HDFS/Hbase etc)
> using flume.
> If your binary files are already being stored in some local file system in
> some folders then you can use SpoolingDirectory source "
> https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source".
> The default deserializer is a LINE based text deserializer which is not for
> binary data. You can use BlobDeserializer "
> https://flume.apache.org/FlumeUserGuide.html#blobdeserializer" which can
> handle binary data.
>
> Cheers
> Ehsan
>
>
> On Fri, Apr 11, 2014 at 9:58 AM, R W <ai...@gmail.com> wrote:
>
>> Hi All
>>
>> We have an App will generate huge amount of binary based file, it's not
>> like text file which we may have new line separator for it, we want to
>> collect these binary files and store them into a Hadoop cluster, I'm new to
>> Flume, so could anyone share some ideas here on how to do it with Flume,
>> thanks in advance.
>>
>> Cheers
>> aij
>>
>
>

Re: Is it possible for Flume-NG to transfer binary files?

Posted by Ehsan ul Haq <m....@gmail.com>.
Yes it is possible to transfer binary files into Hadoop (HDFS/Hbase etc)
using flume.
If your binary files are already being stored in some local file system in
some folders then you can use SpoolingDirectory source "
https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source".
The default deserializer is a LINE based text deserializer which is not for
binary data. You can use BlobDeserializer "
https://flume.apache.org/FlumeUserGuide.html#blobdeserializer" which can
handle binary data.

Cheers
Ehsan


On Fri, Apr 11, 2014 at 9:58 AM, R W <ai...@gmail.com> wrote:

> Hi All
>
> We have an App will generate huge amount of binary based file, it's not
> like text file which we may have new line separator for it, we want to
> collect these binary files and store them into a Hadoop cluster, I'm new to
> Flume, so could anyone share some ideas here on how to do it with Flume,
> thanks in advance.
>
> Cheers
> aij
>