You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by バーチャル クリストファー <bi...@infoscience.co.jp> on 2012/08/10 11:22:51 UTC

Allowing custom FlumeFormatter to be used when writing HDFS SequenceFiles

Hi,

I have rewritten the HDFS event sink's HDFSFormatterFactory so that I
can use my own FlumeFormatter implementation to write events to
SequenceFiles. I did this because the standard HDFSWritableFormatter
discards all headers apart from the timestamp, whereas I want to write
all headers to HDFS.

The code is available for review
here:<https://github.com/cb372/flume/tree/custom-hdfs-formatter>

https://github.com/cb372/flume/tree/custom-hdfs-formatter

To use it, you just pass in the FlumeFormatter implementation's
classname in the config, similar to the way you specify a custom
EventSerializer.

e.g.
agent_foo.sinks.hdfs-sink.writeFormat=com.mycompany.flume.MyCustomFormatter

The class must have a public zero-argument constructor.

Please let me know what you think,

Chris.

PS: I would have filed a Jira (maybe there already is one?), but the
Jira server is down at the moment.

Re: Allowing custom FlumeFormatter to be used when writing HDFS SequenceFiles

Posted by バーチャル クリストファー <bi...@infoscience.co.jp>.
Mike,

Thanks for the reply.

Once JIRA was back up I found that this feature was already being 
tracked as FLUME-1100. I have attached a patch there.

Chris.


On 2012/08/16 3:09, Mike Percy wrote:
> Hi Chris,
> It would be great if you could file a JIRA to track this. JIRA is up again.
>
> Regards,
> Mike
>
> On Fri, Aug 10, 2012 at 2:22 AM, バーチャル クリストファー
> <bi...@infoscience.co.jp>wrote:
>
>> Hi,
>>
>> I have rewritten the HDFS event sink's HDFSFormatterFactory so that I
>> can use my own FlumeFormatter implementation to write events to
>> SequenceFiles. I did this because the standard HDFSWritableFormatter
>> discards all headers apart from the timestamp, whereas I want to write
>> all headers to HDFS.
>>
>> The code is available for review
>> here:<https://github.com/cb372/flume/tree/custom-hdfs-formatter>
>>
>> https://github.com/cb372/flume/tree/custom-hdfs-formatter
>>
>> To use it, you just pass in the FlumeFormatter implementation's
>> classname in the config, similar to the way you specify a custom
>> EventSerializer.
>>
>> e.g.
>> agent_foo.sinks.hdfs-sink.writeFormat=com.mycompany.flume.MyCustomFormatter
>>
>> The class must have a public zero-argument constructor.
>>
>> Please let me know what you think,
>>
>> Chris.
>>
>> PS: I would have filed a Jira (maybe there already is one?), but the
>> Jira server is down at the moment.
>>



Re: Allowing custom FlumeFormatter to be used when writing HDFS SequenceFiles

Posted by Mike Percy <mp...@apache.org>.
Hi Chris,
It would be great if you could file a JIRA to track this. JIRA is up again.

Regards,
Mike

On Fri, Aug 10, 2012 at 2:22 AM, バーチャル クリストファー
<bi...@infoscience.co.jp>wrote:

> Hi,
>
> I have rewritten the HDFS event sink's HDFSFormatterFactory so that I
> can use my own FlumeFormatter implementation to write events to
> SequenceFiles. I did this because the standard HDFSWritableFormatter
> discards all headers apart from the timestamp, whereas I want to write
> all headers to HDFS.
>
> The code is available for review
> here:<https://github.com/cb372/flume/tree/custom-hdfs-formatter>
>
> https://github.com/cb372/flume/tree/custom-hdfs-formatter
>
> To use it, you just pass in the FlumeFormatter implementation's
> classname in the config, similar to the way you specify a custom
> EventSerializer.
>
> e.g.
> agent_foo.sinks.hdfs-sink.writeFormat=com.mycompany.flume.MyCustomFormatter
>
> The class must have a public zero-argument constructor.
>
> Please let me know what you think,
>
> Chris.
>
> PS: I would have filed a Jira (maybe there already is one?), but the
> Jira server is down at the moment.
>