You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by João Ascenso <jo...@gmail.com> on 2012/08/20 18:03:58 UTC
Parsing log data
Greetings,
I'm currently storing log data using flume -> hdfs and was looking for a
way to alter the log data from the normal string format to a json
representation (and possibly in the future store in hbase).
Is there any component in flume that i can use for this modification? i
remember seeing in the old documentation that you could use decorators, but
never found a good example or even if i can still use that in flume 1.x
-------------------
João Ascenso
Re: Parsing log data
Posted by João Ascenso <jo...@gmail.com>.
Thank you, i'll check the 2 interceptor present to try to build my own json
interceptor.
As recommendation, should i use flume 1.3 for interceptors and hbase
serializers or the 1.2 already has it mature enough?
--------------------------
João Ascenso
2012/8/20 Brock Noland <br...@cloudera.com>
> You should be able to use "Flume Interceptors" (
> http://flume.apache.org/FlumeUserGuide.html) to do this. Also, the HBase
> sink allows you to specify a serializer to help match events to Put records.
>
> Brock
>
>
> On Mon, Aug 20, 2012 at 11:03 AM, João Ascenso <
> joaoricardoascenso@gmail.com> wrote:
>
>> Greetings,
>>
>> I'm currently storing log data using flume -> hdfs and was looking for a
>> way to alter the log data from the normal string format to a json
>> representation (and possibly in the future store in hbase).
>>
>> Is there any component in flume that i can use for this modification? i
>> remember seeing in the old documentation that you could use decorators, but
>> never found a good example or even if i can still use that in flume 1.x
>>
>> -------------------
>> João Ascenso
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: Parsing log data
Posted by Brock Noland <br...@cloudera.com>.
You should be able to use "Flume Interceptors" (
http://flume.apache.org/FlumeUserGuide.html) to do this. Also, the HBase
sink allows you to specify a serializer to help match events to Put records.
Brock
On Mon, Aug 20, 2012 at 11:03 AM, João Ascenso <joaoricardoascenso@gmail.com
> wrote:
> Greetings,
>
> I'm currently storing log data using flume -> hdfs and was looking for a
> way to alter the log data from the normal string format to a json
> representation (and possibly in the future store in hbase).
>
> Is there any component in flume that i can use for this modification? i
> remember seeing in the old documentation that you could use decorators, but
> never found a good example or even if i can still use that in flume 1.x
>
> -------------------
> João Ascenso
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/