You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Deepak Subhramanian <de...@gmail.com> on 2013/10/23 17:25:18 UTC
Not able to set date %d%m%Y in Sink Directory name
Hi ,
I am trying to store my logs in folders named with date for my file_roll
and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
is not working . Any thoughts .
My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
tier1.sinks.filesink1.type = file_roll
tier1.sinks.filesink1.channel = c2
tier1.sinks.filesink1.batchSize = 100000
tier1.sinks.filesink1.rollInterval = 0
#tier1.sinks.filesink1.rollInterval = 86400
tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.
org.apache.flume.EventDeliveryException: Failed to open file
/projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException:
/projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(Unknown Source)
at java.io.FileOutputStream.<init>(Unknown Source)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
Thanks,
Deepak Subhramanian
Re: Not able to set date %d%m%Y in Sink Directory name
Posted by Deepak Subhramanian <de...@gmail.com>.
Thanks David.
On Thu, Oct 24, 2013 at 2:47 PM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:
> You could do that or add the TimestampInterceptor to your source as well.
>
>
> On Thu, Oct 24, 2013 at 5:16 AM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler
>> which reads a log message from HTTP Request and create a Flume event. May
>> be I have to add the header to the event with the timestamp.
>>
>> String message = (String)request.getParameter("log");
>>
>> //LOG.info(message );
>>
>> LOG.info("Msg Read" );
>>
>>
>> Event event = EventBuilder.withBody(message.getBytes());
>>
>>
>> eventList.add(event);
>>
>>
>> On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair <
>> dsinclair@chariotsolutions.com> wrote:
>>
>>> The sink is dependent on a header with the key "timestamp" being present
>>> in the event for this to work. What source are you using?
>>>
>>>
>>> On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
>>> deepak.subhramanian@gmail.com> wrote:
>>>
>>>> Hi ,
>>>>
>>>> I am trying to store my logs in folders named with date for my
>>>> file_roll and hdfssink. For some reason when I pass %d%m%Y in the sink
>>>> directory it is not working . Any thoughts .
>>>>
>>>> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>>>>
>>>> tier1.sinks.filesink1.type = file_roll
>>>> tier1.sinks.filesink1.channel = c2
>>>> tier1.sinks.filesink1.batchSize = 100000
>>>> tier1.sinks.filesink1.rollInterval = 0
>>>> #tier1.sinks.filesink1.rollInterval = 86400
>>>> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>>>>
>>>>
>>>> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
>>>> Exception follows.
>>>> org.apache.flume.EventDeliveryException: Failed to open file
>>>> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
>>>> at
>>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
>>>> at
>>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>> at java.lang.Thread.run(Unknown Source)
>>>> Caused by: java.io.FileNotFoundException:
>>>> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
>>>> directory)
>>>> at java.io.FileOutputStream.open(Native Method)
>>>> at java.io.FileOutputStream.<init>(Unknown Source)
>>>> at java.io.FileOutputStream.<init>(Unknown Source)
>>>> at
>>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>>>
>>>>
>>>> Thanks,
>>>> Deepak Subhramanian
>>>>
>>>
>>>
>>
>>
>> --
>> Deepak Subhramanian
>>
>
>
--
Deepak Subhramanian
Re: Not able to set date %d%m%Y in Sink Directory name
Posted by David Sinclair <ds...@chariotsolutions.com>.
You could do that or add the TimestampInterceptor to your source as well.
On Thu, Oct 24, 2013 at 5:16 AM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler
> which reads a log message from HTTP Request and create a Flume event. May
> be I have to add the header to the event with the timestamp.
>
> String message = (String)request.getParameter("log");
>
> //LOG.info(message );
>
> LOG.info("Msg Read" );
>
>
> Event event = EventBuilder.withBody(message.getBytes());
>
>
> eventList.add(event);
>
>
> On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair <
> dsinclair@chariotsolutions.com> wrote:
>
>> The sink is dependent on a header with the key "timestamp" being present
>> in the event for this to work. What source are you using?
>>
>>
>> On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
>> deepak.subhramanian@gmail.com> wrote:
>>
>>> Hi ,
>>>
>>> I am trying to store my logs in folders named with date for my file_roll
>>> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
>>> is not working . Any thoughts .
>>>
>>> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>>>
>>> tier1.sinks.filesink1.type = file_roll
>>> tier1.sinks.filesink1.channel = c2
>>> tier1.sinks.filesink1.batchSize = 100000
>>> tier1.sinks.filesink1.rollInterval = 0
>>> #tier1.sinks.filesink1.rollInterval = 86400
>>> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>>>
>>>
>>> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
>>> Exception follows.
>>> org.apache.flume.EventDeliveryException: Failed to open file
>>> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
>>> at
>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Unknown Source)
>>> Caused by: java.io.FileNotFoundException:
>>> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
>>> directory)
>>> at java.io.FileOutputStream.open(Native Method)
>>> at java.io.FileOutputStream.<init>(Unknown Source)
>>> at java.io.FileOutputStream.<init>(Unknown Source)
>>> at
>>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>>
>>>
>>> Thanks,
>>> Deepak Subhramanian
>>>
>>
>>
>
>
> --
> Deepak Subhramanian
>
Re: Not able to set date %d%m%Y in Sink Directory name
Posted by Deepak Subhramanian <de...@gmail.com>.
I am using a SimpleHTTPEventHandler which implements HTTPSourceHandler
which reads a log message from HTTP Request and create a Flume event. May
be I have to add the header to the event with the timestamp.
String message = (String)request.getParameter("log");
//LOG.info(message );
LOG.info("Msg Read" );
Event event = EventBuilder.withBody(message.getBytes());
eventList.add(event);
On Wed, Oct 23, 2013 at 6:40 PM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:
> The sink is dependent on a header with the key "timestamp" being present
> in the event for this to work. What source are you using?
>
>
> On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
> deepak.subhramanian@gmail.com> wrote:
>
>> Hi ,
>>
>> I am trying to store my logs in folders named with date for my file_roll
>> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
>> is not working . Any thoughts .
>>
>> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>>
>> tier1.sinks.filesink1.type = file_roll
>> tier1.sinks.filesink1.channel = c2
>> tier1.sinks.filesink1.batchSize = 100000
>> tier1.sinks.filesink1.rollInterval = 0
>> #tier1.sinks.filesink1.rollInterval = 86400
>> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>>
>>
>> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
>> Exception follows.
>> org.apache.flume.EventDeliveryException: Failed to open file
>> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
>> at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Unknown Source)
>> Caused by: java.io.FileNotFoundException:
>> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
>> directory)
>> at java.io.FileOutputStream.open(Native Method)
>> at java.io.FileOutputStream.<init>(Unknown Source)
>> at java.io.FileOutputStream.<init>(Unknown Source)
>> at
>> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>
>>
>> Thanks,
>> Deepak Subhramanian
>>
>
>
--
Deepak Subhramanian
Re: Not able to set date %d%m%Y in Sink Directory name
Posted by David Sinclair <ds...@chariotsolutions.com>.
The sink is dependent on a header with the key "timestamp" being present in
the event for this to work. What source are you using?
On Wed, Oct 23, 2013 at 11:25 AM, Deepak Subhramanian <
deepak.subhramanian@gmail.com> wrote:
> Hi ,
>
> I am trying to store my logs in folders named with date for my file_roll
> and hdfssink. For some reason when I pass %d%m%Y in the sink directory it
> is not working . Any thoughts .
>
> My Flume source is a simple HTTP Handler extended from HTTPSourceHandler
>
> tier1.sinks.filesink1.type = file_roll
> tier1.sinks.filesink1.channel = c2
> tier1.sinks.filesink1.batchSize = 100000
> tier1.sinks.filesink1.rollInterval = 0
> #tier1.sinks.filesink1.rollInterval = 86400
> tier1.sinks.filesink1.sink.directory = /projects/test/logs/%d%m%Y
>
>
> 13/10/23 16:20:36 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.EventDeliveryException: Failed to open file
> /projects/test/logs/%d%m%Y/1382541251307-1 while delivering event
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.FileNotFoundException:
> /projects/ETHAN_FLUME/logs/%d%m%Y/1382541251307-1 (No such file or
> directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(Unknown Source)
> at java.io.FileOutputStream.<init>(Unknown Source)
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>
>
> Thanks,
> Deepak Subhramanian
>