You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Meyer, Dennis" <de...@adtech.com> on 2011/10/19 13:22:15 UTC

Using Flume agent on binary Avro file

Hi,

We've got some use case I couldn't find good information about:

We use AVRO binary logging format and log the data out in binary compressed AVRO format as we want to kep the data small. Those files we'd like a Flume agent to listen and stream the data into hadoop. The current process needs the intermediate file – so no direct to Hadoop streaming is possible.

Is there any way to do this, would wonder if it's not supported, but can't find infos?

Thanks,
Dennis

Re: Using Flume agent on binary Avro file

Posted by "Meyer, Dennis" <de...@adtech.com>.
Hi,

Would it be possible to write directly to the Flume TCP socket using the binary AVRO format? This could be an option in our use-case.

Thanks,
Dennis



Von: Eric Sammer <es...@cloudera.com>>
Antworten an: <fl...@incubator.apache.org>>
Datum: Wed, 19 Oct 2011 10:55:38 -0700
An: <fl...@incubator.apache.org>>
Betreff: Re: Using Flume agent on binary Avro file

Dennis:

There's no good way to tail an Avro file without writing a custom source. Your options are probably to either move the complete file (without Flume) or have the application write directly to the Avro source (using the log4j appender or something similar).

On Wed, Oct 19, 2011 at 4:22 AM, Meyer, Dennis <de...@adtech.com>> wrote:
Hi,

We've got some use case I couldn't find good information about:

We use AVRO binary logging format and log the data out in binary compressed AVRO format as we want to kep the data small. Those files we'd like a Flume agent to listen and stream the data into hadoop. The current process needs the intermediate file – so no direct to Hadoop streaming is possible.

Is there any way to do this, would wonder if it's not supported, but can't find infos?

Thanks,
Dennis



--
Eric Sammer
twitter: esammer
data: www.cloudera.com<http://www.cloudera.com>

Re: Using Flume agent on binary Avro file

Posted by Eric Sammer <es...@cloudera.com>.
Dennis:

There's no good way to tail an Avro file without writing a custom source.
Your options are probably to either move the complete file (without Flume)
or have the application write directly to the Avro source (using the log4j
appender or something similar).

On Wed, Oct 19, 2011 at 4:22 AM, Meyer, Dennis <de...@adtech.com>wrote:

>   Hi,
>
>  We've got some use case I couldn't find good information about:
>
>  We use AVRO binary logging format and log the data out in binary
> compressed AVRO format as we want to kep the data small. Those files we'd
> like a Flume agent to listen and stream the data into hadoop. The current
> process needs the intermediate file – so no direct to Hadoop streaming is
> possible.
>
>  Is there any way to do this, would wonder if it's not supported, but
> can't find infos?
>
>  Thanks,
> Dennis
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com