You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by khadar basha <kh...@gmail.com> on 2012/07/26 13:32:40 UTC
ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Hi
we are using flume-1.1.0-cdh4.0.1 . we have the following configurations in
flume-conf.properties.
agent1Test1.sources = seqGenSrc
agent1Test1.channels = memoryChannel
agent1Test1.sinks = loggerSink
# For each one of the sources, the type is defined
agent1Test1.sources.seqGenSrc.type = exec
agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out
# The channel can be defined as follows.
agent1Test1.sources.seqGenSrc.channels = memoryChannel
# Each sink's type must be defined
agent1Test1.sinks.loggerSink.type = hdfs
agent1Test1.sinks.loggerSink.hdfs.path =
hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
#Specify the channel the sink should use
agent1Test1.sinks.loggerSink.channel = memoryChannel
# Each channel's type is defined.
agent1Test1.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent1Test1.channels.memoryChannel.capacity = 1000
I am getting the following exception. Anybody facing this issue.
*2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
*java.lang.NumberFormatException: null*
* at java.lang.Long.parseLong(Long.java:375)*
* at java.lang.Long.valueOf(Long.java:525)*
* at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
*
* at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
*
* at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
* at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
*
* at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
* at java.lang.Thread.run(Thread.java:662)*
*2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.*
*org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
null*
* at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)*
* at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
*
* at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
* at java.lang.Thread.run(Thread.java:662)*
*Caused by: java.lang.NumberFormatException: null*
* at java.lang.Long.parseLong(Long.java:375)*
* at java.lang.Long.valueOf(Long.java:525)*
* at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
*
* at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
*
* at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
* ... 3 more*
--
Thanks,
Khadar
Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Posted by Brock Noland <br...@cloudera.com>.
It does http://flume.apache.org/releases/1.1.0.html but the root of the
issue is that the source doesn't add the timestamp header to the event.
Also 1.2 has just been released which has Interceptors which I think can
add the timestamp for you.
Brock
On Thu, Jul 26, 2012 at 12:59 PM, khadar basha <kh...@gmail.com> wrote:
> Hi Brock.
>
> flume-1.1.0 doesn't support this escape sequence options?
>
> -Khadar
>
> On Thu, Jul 26, 2012 at 10:34 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> The flume user doc here http://flume.apache.org/releases/1.2.0.html has
>> many examples.
>>
>> Brock
>>
>>
>> On Thu, Jul 26, 2012 at 11:37 AM, khadar basha <kh...@gmail.com>wrote:
>>
>>> Hi Brock,
>>>
>>> could you please provide me the sample configuration?
>>>
>>> Thanks,
>>> Khadar
>>>
>>>
>>> On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> If you want to use a bucketed paths your events need to have a
>>>> "timestamp" header which tail source does not appear to add.
>>>>
>>>> Brock
>>>>
>>>> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <kh...@gmail.com>wrote:
>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>> we are using flume-1.1.0-cdh4.0.1 . we have the following
>>>>> configurations in flume-conf.properties.
>>>>>
>>>>> agent1Test1.sources = seqGenSrc
>>>>> agent1Test1.channels = memoryChannel
>>>>> agent1Test1.sinks = loggerSink
>>>>>
>>>>> # For each one of the sources, the type is defined
>>>>> agent1Test1.sources.seqGenSrc.type = exec
>>>>> agent1Test1.sources.seqGenSrc.command= tail -f
>>>>> /tomcat/logs/catalina.out
>>>>>
>>>>>
>>>>> # The channel can be defined as follows.
>>>>> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>>>>>
>>>>> # Each sink's type must be defined
>>>>> agent1Test1.sinks.loggerSink.type = hdfs
>>>>> agent1Test1.sinks.loggerSink.hdfs.path =
>>>>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>>>>>
>>>>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>>>>>
>>>>> #Specify the channel the sink should use
>>>>> agent1Test1.sinks.loggerSink.channel = memoryChannel
>>>>>
>>>>> # Each channel's type is defined.
>>>>> agent1Test1.channels.memoryChannel.type = memory
>>>>>
>>>>> # Other config values specific to each type of channel(sink or source)
>>>>> # can be defined as well
>>>>> # In this case, it specifies the capacity of the memory channel
>>>>> agent1Test1.channels.memoryChannel.capacity = 1000
>>>>>
>>>>>
>>>>> I am getting the following exception. Anybody facing this issue.
>>>>>
>>>>>
>>>>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
>>>>> *java.lang.NumberFormatException: null*
>>>>> * at java.lang.Long.parseLong(Long.java:375)*
>>>>> * at java.lang.Long.valueOf(Long.java:525)*
>>>>> * at
>>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>>> * at java.lang.Thread.run(Thread.java:662)*
>>>>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver
>>>>> event. Exception follows.*
>>>>> *org.apache.flume.EventDeliveryException:
>>>>> java.lang.NumberFormatException: null*
>>>>> * at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>>> * at java.lang.Thread.run(Thread.java:662)*
>>>>> *Caused by: java.lang.NumberFormatException: null*
>>>>> * at java.lang.Long.parseLong(Long.java:375)*
>>>>> * at java.lang.Long.valueOf(Long.java:525)*
>>>>> * at
>>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>>> *
>>>>> * at
>>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>>> *
>>>>> * ... 3 more*
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Khadar
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce -
>>>> http://incubator.apache.org/mrunit/
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Khadar
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
>
> --
> Thanks,
> Khadar
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Posted by khadar basha <kh...@gmail.com>.
Hi Brock.
flume-1.1.0 doesn't support this escape sequence options?
-Khadar
On Thu, Jul 26, 2012 at 10:34 PM, Brock Noland <br...@cloudera.com> wrote:
> The flume user doc here http://flume.apache.org/releases/1.2.0.html has
> many examples.
>
> Brock
>
>
> On Thu, Jul 26, 2012 at 11:37 AM, khadar basha <kh...@gmail.com>wrote:
>
>> Hi Brock,
>>
>> could you please provide me the sample configuration?
>>
>> Thanks,
>> Khadar
>>
>>
>> On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> If you want to use a bucketed paths your events need to have a
>>> "timestamp" header which tail source does not appear to add.
>>>
>>> Brock
>>>
>>> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <kh...@gmail.com>wrote:
>>>
>>>>
>>>> Hi
>>>>
>>>> we are using flume-1.1.0-cdh4.0.1 . we have the following
>>>> configurations in flume-conf.properties.
>>>>
>>>> agent1Test1.sources = seqGenSrc
>>>> agent1Test1.channels = memoryChannel
>>>> agent1Test1.sinks = loggerSink
>>>>
>>>> # For each one of the sources, the type is defined
>>>> agent1Test1.sources.seqGenSrc.type = exec
>>>> agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out
>>>>
>>>>
>>>> # The channel can be defined as follows.
>>>> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>>>>
>>>> # Each sink's type must be defined
>>>> agent1Test1.sinks.loggerSink.type = hdfs
>>>> agent1Test1.sinks.loggerSink.hdfs.path =
>>>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>>>>
>>>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>>>>
>>>> #Specify the channel the sink should use
>>>> agent1Test1.sinks.loggerSink.channel = memoryChannel
>>>>
>>>> # Each channel's type is defined.
>>>> agent1Test1.channels.memoryChannel.type = memory
>>>>
>>>> # Other config values specific to each type of channel(sink or source)
>>>> # can be defined as well
>>>> # In this case, it specifies the capacity of the memory channel
>>>> agent1Test1.channels.memoryChannel.capacity = 1000
>>>>
>>>>
>>>> I am getting the following exception. Anybody facing this issue.
>>>>
>>>>
>>>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
>>>> *java.lang.NumberFormatException: null*
>>>> * at java.lang.Long.parseLong(Long.java:375)*
>>>> * at java.lang.Long.valueOf(Long.java:525)*
>>>> * at
>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>> *
>>>> * at
>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>> *
>>>> * at
>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>> *
>>>> * at
>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>> *
>>>> * at
>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>> * at java.lang.Thread.run(Thread.java:662)*
>>>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver
>>>> event. Exception follows.*
>>>> *org.apache.flume.EventDeliveryException:
>>>> java.lang.NumberFormatException: null*
>>>> * at
>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>>> *
>>>> * at
>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>> *
>>>> * at
>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>>> * at java.lang.Thread.run(Thread.java:662)*
>>>> *Caused by: java.lang.NumberFormatException: null*
>>>> * at java.lang.Long.parseLong(Long.java:375)*
>>>> * at java.lang.Long.valueOf(Long.java:525)*
>>>> * at
>>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>>> *
>>>> * at
>>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>>> *
>>>> * at
>>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>>> *
>>>> * ... 3 more*
>>>>
>>>> --
>>>> Thanks,
>>>> Khadar
>>>>
>>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>>
>>
>>
>> --
>> Thanks,
>> Khadar
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
Thanks,
Khadar
Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Posted by Brock Noland <br...@cloudera.com>.
The flume user doc here http://flume.apache.org/releases/1.2.0.html has
many examples.
Brock
On Thu, Jul 26, 2012 at 11:37 AM, khadar basha <kh...@gmail.com> wrote:
> Hi Brock,
>
> could you please provide me the sample configuration?
>
> Thanks,
> Khadar
>
>
> On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> If you want to use a bucketed paths your events need to have a
>> "timestamp" header which tail source does not appear to add.
>>
>> Brock
>>
>> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <kh...@gmail.com>wrote:
>>
>>>
>>> Hi
>>>
>>> we are using flume-1.1.0-cdh4.0.1 . we have the following configurations
>>> in flume-conf.properties.
>>>
>>> agent1Test1.sources = seqGenSrc
>>> agent1Test1.channels = memoryChannel
>>> agent1Test1.sinks = loggerSink
>>>
>>> # For each one of the sources, the type is defined
>>> agent1Test1.sources.seqGenSrc.type = exec
>>> agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out
>>>
>>>
>>> # The channel can be defined as follows.
>>> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>>>
>>> # Each sink's type must be defined
>>> agent1Test1.sinks.loggerSink.type = hdfs
>>> agent1Test1.sinks.loggerSink.hdfs.path =
>>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>>>
>>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>>>
>>> #Specify the channel the sink should use
>>> agent1Test1.sinks.loggerSink.channel = memoryChannel
>>>
>>> # Each channel's type is defined.
>>> agent1Test1.channels.memoryChannel.type = memory
>>>
>>> # Other config values specific to each type of channel(sink or source)
>>> # can be defined as well
>>> # In this case, it specifies the capacity of the memory channel
>>> agent1Test1.channels.memoryChannel.capacity = 1000
>>>
>>>
>>> I am getting the following exception. Anybody facing this issue.
>>>
>>>
>>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
>>> *java.lang.NumberFormatException: null*
>>> * at java.lang.Long.parseLong(Long.java:375)*
>>> * at java.lang.Long.valueOf(Long.java:525)*
>>> * at
>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>> *
>>> * at
>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>> *
>>> * at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>> *
>>> * at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> *
>>> * at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>> * at java.lang.Thread.run(Thread.java:662)*
>>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver
>>> event. Exception follows.*
>>> *org.apache.flume.EventDeliveryException:
>>> java.lang.NumberFormatException: null*
>>> * at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
>>> *
>>> * at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> *
>>> * at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>>> * at java.lang.Thread.run(Thread.java:662)*
>>> *Caused by: java.lang.NumberFormatException: null*
>>> * at java.lang.Long.parseLong(Long.java:375)*
>>> * at java.lang.Long.valueOf(Long.java:525)*
>>> * at
>>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>>> *
>>> * at
>>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>>> *
>>> * at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>>> *
>>> * ... 3 more*
>>>
>>> --
>>> Thanks,
>>> Khadar
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
>
> --
> Thanks,
> Khadar
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Posted by khadar basha <kh...@gmail.com>.
Hi Brock,
could you please provide me the sample configuration?
Thanks,
Khadar
On Thu, Jul 26, 2012 at 7:23 PM, Brock Noland <br...@cloudera.com> wrote:
> If you want to use a bucketed paths your events need to have a "timestamp"
> header which tail source does not appear to add.
>
> Brock
>
> On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <kh...@gmail.com> wrote:
>
>>
>> Hi
>>
>> we are using flume-1.1.0-cdh4.0.1 . we have the following configurations
>> in flume-conf.properties.
>>
>> agent1Test1.sources = seqGenSrc
>> agent1Test1.channels = memoryChannel
>> agent1Test1.sinks = loggerSink
>>
>> # For each one of the sources, the type is defined
>> agent1Test1.sources.seqGenSrc.type = exec
>> agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out
>>
>>
>> # The channel can be defined as follows.
>> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>>
>> # Each sink's type must be defined
>> agent1Test1.sinks.loggerSink.type = hdfs
>> agent1Test1.sinks.loggerSink.hdfs.path =
>> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>>
>> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>>
>> #Specify the channel the sink should use
>> agent1Test1.sinks.loggerSink.channel = memoryChannel
>>
>> # Each channel's type is defined.
>> agent1Test1.channels.memoryChannel.type = memory
>>
>> # Other config values specific to each type of channel(sink or source)
>> # can be defined as well
>> # In this case, it specifies the capacity of the memory channel
>> agent1Test1.channels.memoryChannel.capacity = 1000
>>
>>
>> I am getting the following exception. Anybody facing this issue.
>>
>>
>> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
>> *java.lang.NumberFormatException: null*
>> * at java.lang.Long.parseLong(Long.java:375)*
>> * at java.lang.Long.valueOf(Long.java:525)*
>> * at
>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> *
>> * at
>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> *
>> * at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
>> * at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> *
>> * at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>> * at java.lang.Thread.run(Thread.java:662)*
>> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver
>> event. Exception follows.*
>> *org.apache.flume.EventDeliveryException:
>> java.lang.NumberFormatException: null*
>> * at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)*
>> * at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> *
>> * at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
>> * at java.lang.Thread.run(Thread.java:662)*
>> *Caused by: java.lang.NumberFormatException: null*
>> * at java.lang.Long.parseLong(Long.java:375)*
>> * at java.lang.Long.valueOf(Long.java:525)*
>> * at
>> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>> *
>> * at
>> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>> *
>> * at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
>> * ... 3 more*
>>
>> --
>> Thanks,
>> Khadar
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
Thanks,
Khadar
Re: ERROR hdfs.HDFSEventSink: process failed. ERROR flume.SinkRunner:
Unable to deliver event. Exception follows.
Posted by Brock Noland <br...@cloudera.com>.
If you want to use a bucketed paths your events need to have a "timestamp"
header which tail source does not appear to add.
Brock
On Thu, Jul 26, 2012 at 6:32 AM, khadar basha <kh...@gmail.com> wrote:
>
> Hi
>
> we are using flume-1.1.0-cdh4.0.1 . we have the following configurations
> in flume-conf.properties.
>
> agent1Test1.sources = seqGenSrc
> agent1Test1.channels = memoryChannel
> agent1Test1.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> agent1Test1.sources.seqGenSrc.type = exec
> agent1Test1.sources.seqGenSrc.command= tail -f /tomcat/logs/catalina.out
>
>
> # The channel can be defined as follows.
> agent1Test1.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent1Test1.sinks.loggerSink.type = hdfs
> agent1Test1.sinks.loggerSink.hdfs.path =
> hdfs://hdfshost:port/data/%Y/%m/%{host}/mylogs
>
> agent1Test1.sinks.loggerSink.hdfs.fileType = DataStream
>
> #Specify the channel the sink should use
> agent1Test1.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent1Test1.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> agent1Test1.channels.memoryChannel.capacity = 1000
>
>
> I am getting the following exception. Anybody facing this issue.
>
>
> *2012-07-26 13:07:52,002 ERROR hdfs.HDFSEventSink: process failed*
> *java.lang.NumberFormatException: null*
> * at java.lang.Long.parseLong(Long.java:375)*
> * at java.lang.Long.valueOf(Long.java:525)*
> * at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
> *
> * at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
> *
> * at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
> * at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> *
> * at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
> * at java.lang.Thread.run(Thread.java:662)*
> *2012-07-26 13:07:52,003 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.*
> *org.apache.flume.EventDeliveryException:
> java.lang.NumberFormatException: null*
> * at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)*
> * at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> *
> * at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
> * at java.lang.Thread.run(Thread.java:662)*
> *Caused by: java.lang.NumberFormatException: null*
> * at java.lang.Long.parseLong(Long.java:375)*
> * at java.lang.Long.valueOf(Long.java:525)*
> * at
> org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
> *
> * at
> org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
> *
> * at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)*
> * ... 3 more*
>
> --
> Thanks,
> Khadar
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/