You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by IT CTO <go...@gmail.com> on 2015/09/21 17:03:01 UTC
Interceptor vrs Serialeztion
I want to read files and write them to hdfs but I want to write the to a
date partitioned folder based on a date value IN THE ROW.
Should I write a custom interceptor or custom serializer?
Eran
--
Eran | "You don't need eyes to see, you need vision" (Faithless)
elatiscsearch sink test case failure
Posted by 이승진 <sw...@navercorp.com>.
Hello all,
I pulled trunk branch and tried to build jar by mvn package.
But it failed because of a single test case failure.
Below is the log regarding that, I think this is due to the difference in locale and should be fixed
Failed tests: shouldUseUtcAsBasisForDateFormat(org.apache.flume.sink.elasticsearch.TestElasticSearchIndexRequestBuilderFactory): expected:<[Coordinated Universal Time]> but was:<[세계 표준시]>
Re: Interceptor vrs Serialeztion
Posted by IT CTO <go...@gmail.com>.
Thanks, I will try it.
בתאריך יום ב׳, 21 בספט׳ 2015, 18:15 מאת Ahmed Vila <av...@devlogic.eu>:
> I think that interceptor is a way to go and you can use regex_extractor
> interceptor instead of building your own - it simplifies deployment.
> https://flume.apache.org/FlumeUserGuide.html#regex-extractor-interceptor
>
> Further, you can use RegexExtractorInterceptorMillisSerializer for this
> interceptor's type in order to do datetime parsing and do the formatting on
> fly that will be suitable for your hdfs folder (f.e. extract year-month).
> https://flume.apache.org/FlumeUserGuide.html#example-2
>
> Later on, you can use header replacement string in the hdfs path to inject
> a header defined with serializer's name ("timestamp" in the example above).
> Hdfs Sink supports timestamp header out of the box and applies escape
> sequences against it by default, unless hdfs.useLocalTimeStamp = true.
> So, a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S will have
> replacement values from the timestamp header produced by regex extractor.
>
>
>
>
> On Mon, Sep 21, 2015 at 5:03 PM, IT CTO <go...@gmail.com> wrote:
>
>> I want to read files and write them to hdfs but I want to write the to a
>> date partitioned folder based on a date value IN THE ROW.
>> Should I write a custom interceptor or custom serializer?
>> Eran
>> --
>> Eran | "You don't need eyes to see, you need vision" (Faithless)
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu
> E-mail : avila@devlogic.eu
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
--
Eran | "You don't need eyes to see, you need vision" (Faithless)
Re: Interceptor vrs Serialeztion
Posted by Ahmed Vila <av...@devlogic.eu>.
I think that interceptor is a way to go and you can use regex_extractor
interceptor instead of building your own - it simplifies deployment.
https://flume.apache.org/FlumeUserGuide.html#regex-extractor-interceptor
Further, you can use RegexExtractorInterceptorMillisSerializer for this
interceptor's type in order to do datetime parsing and do the formatting on
fly that will be suitable for your hdfs folder (f.e. extract year-month).
https://flume.apache.org/FlumeUserGuide.html#example-2
Later on, you can use header replacement string in the hdfs path to inject
a header defined with serializer's name ("timestamp" in the example above).
Hdfs Sink supports timestamp header out of the box and applies escape
sequences against it by default, unless hdfs.useLocalTimeStamp = true.
So, a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S will have
replacement values from the timestamp header produced by regex extractor.
On Mon, Sep 21, 2015 at 5:03 PM, IT CTO <go...@gmail.com> wrote:
> I want to read files and write them to hdfs but I want to write the to a
> date partitioned folder based on a date value IN THE ROW.
> Should I write a custom interceptor or custom serializer?
> Eran
> --
> Eran | "You don't need eyes to see, you need vision" (Faithless)
>
--
Best regards,
Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina
Office : +387 33 942 123
Mobile: +387 62 139 348
Website: www.devlogic.eu
E-mail : avila@devlogic.eu
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.
--
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.