You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Juhani Connolly <ju...@cyberagent.co.jp> on 2012/07/04 04:19:37 UTC
Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
To use the date related escape sequences you need to provide a timestamp
header.
The easiest way to do this would be configuring a timestamp interceptor
onto your source, ours looks something like:
sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
On 07/04/2012 11:00 AM, Shara Shi wrote:
>
> HI
>
> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>
> I got following error messages
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
> at java.lang.Long.parseLong(Long.java:375)
>
> at java.lang.Long.valueOf(Long.java:525)
>
> at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
> at java.lang.Long.parseLong(Long.java:375)
>
> at java.lang.Long.valueOf(Long.java:525)
>
> at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
> ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
> at java.lang.Long.parseLong(Long.java:375)
>
> at java.lang.Long.valueOf(Long.java:525)
>
> at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
> at java.lang.Thread.run(Thread.java:662)
>
> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
>
> org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.lang.NumberFormatException: null
>
> at java.lang.Long.parseLong(Long.java:375)
>
> at java.lang.Long.valueOf(Long.java:525)
>
> at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>
> at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>
> ... 3 more
>
> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>
> java.lang.NumberFormatException: null
>
> at java.lang.Long.parseLong(Long.java:375)
>
> at java.lang.Long.valueOf(Long.java:525)
>
> =============
>
> collector2.channels.ch2.type = memory
>
> collector2.sources.avro-source1.channels = ch2
>
> collector2.sources.avro-source1.type = avro
>
> collector2.sources.avro-source1.bind = 0.0.0.0
>
> collector2.sources.avro-source1.port = 41414
>
> collector2.sinks.hdfs.channel = ch2
>
> collector2.sinks.hdfs.type= hdfs
>
> *collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/*
>
> collector2.sinks.hdfs.batchsize=1
>
> collector2.sinks.hdfs.runner.type=polling
>
> collector2.sinks.hdfs.runner.polling.interval = 60
>
> collector2.sinks.hdfs.hdfs.rollInterval = 300
>
> collector2.sinks.hdfs.hdfs.rollSize = 0
>
> collector2.sinks.hdfs.hdfs.rollCount = 0
>
> collector2.channels = ch2
>
> collector2.sources = avro-source1
>
> collector2.sinks = hdfs
>
> Ruihong Shi
>
答复: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
Posted by Shara Shi <sh...@dhgate.com>.
Hi
It works well , thanks
Ruihong Shi
发件人: Juhani Connolly [mailto:juhani_connolly@cyberagent.co.jp]
发送时间: 2012年7月4日 10:20
收件人: flume-user@incubator.apache.org
主题: Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
To use the date related escape sequences you need to provide a timestamp
header.
The easiest way to do this would be configuring a timestamp interceptor onto
your source, ours looks something like:
sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
On 07/04/2012 11:00 AM, Shara Shi wrote:
HI
I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
I got following error messages
12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception
follows.
org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
null
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
... 3 more
12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception
follows.
org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
null
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)
at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
... 3 more
12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:375)
at java.lang.Long.valueOf(Long.java:525)
=============
collector2.channels.ch2.type = memory
collector2.sources.avro-source1.channels = ch2
collector2.sources.avro-source1.type = avro
collector2.sources.avro-source1.bind = 0.0.0.0
collector2.sources.avro-source1.port = 41414
collector2.sinks.hdfs.channel = ch2
collector2.sinks.hdfs.type= hdfs
collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/
collector2.sinks.hdfs.batchsize=1
collector2.sinks.hdfs.runner.type=polling
collector2.sinks.hdfs.runner.polling.interval = 60
collector2.sinks.hdfs.hdfs.rollInterval = 300
collector2.sinks.hdfs.hdfs.rollSize = 0
collector2.sinks.hdfs.hdfs.rollCount = 0
collector2.channels = ch2
collector2.sources = avro-source1
collector2.sinks = hdfs
Ruihong Shi
Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
Posted by Will McQueen <wi...@cloudera.com>.
Right, you can specify the FQCN, or the alias 'TIMESTAMP'
Cheers,
Will
On Jul 3, 2012, at 7:19 PM, Juhani Connolly <ju...@cyberagent.co.jp> wrote:
> To use the date related escape sequences you need to provide a timestamp header.
>
> The easiest way to do this would be configuring a timestamp interceptor onto your source, ours looks something like:
>
> sender1.sources.tail.interceptors = ts
> sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
>
>
> On 07/04/2012 11:00 AM, Shara Shi wrote:
>> HI
>>
>> I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs.
>> I got following error messages
>>
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> 12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>> at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>> ... 3 more
>> 12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed
>> java.lang.NumberFormatException: null
>> at java.lang.Long.parseLong(Long.java:375)
>> at java.lang.Long.valueOf(Long.java:525)
>>
>> =============
>>
>> collector2.channels.ch2.type = memory
>>
>> collector2.sources.avro-source1.channels = ch2
>> collector2.sources.avro-source1.type = avro
>> collector2.sources.avro-source1.bind = 0.0.0.0
>> collector2.sources.avro-source1.port = 41414
>>
>> collector2.sinks.hdfs.channel = ch2
>> collector2.sinks.hdfs.type= hdfs
>> collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/
>> collector2.sinks.hdfs.batchsize=1
>> collector2.sinks.hdfs.runner.type=polling
>> collector2.sinks.hdfs.runner.polling.interval = 60
>>
>> collector2.sinks.hdfs.hdfs.rollInterval = 300
>> collector2.sinks.hdfs.hdfs.rollSize = 0
>> collector2.sinks.hdfs.hdfs.rollCount = 0
>>
>> collector2.channels = ch2
>> collector2.sources = avro-source1
>> collector2.sinks = hdfs
>>
>>
>> Ruihong Shi
>>
>
>