You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by rafeeq shanavaz <ra...@gmail.com> on 2015/05/24 15:00:55 UTC
Flume Roll File Sink - unable to create YY/mm/dd directory
Hi,
I am using Flume Roll-File-Sink to write data in text file on local
directory.
I am trying to use /%Y/%m/%d in directory to maintain my data in daily
basis.
*Agent.sinks.rollSink.sink.directory =
/home/cloudera/flume_events/%Y/%m/%d/*
I am using timeStampInterceptor to achive above directory structure as
below:
Agent.sources.sample.interceptors = i2
Agent.sources.sample.interceptors.i2.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
#Agent.sources.sample.interceptors.i2.type = timestamp
Agent.sources.rollSink.filePrefix = FlumeData.%Y/%m/%d
But I am unable to create directory based on the timestamp and also i am
sending timestamp with event header.
Below exception is thrown:
Caused by: java.io.FileNotFoundException:
/home/cloudera/flume_events/%Y/%m/%d/1432468505170-1 (No such file or
directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
*Please help me to create directory based on the timestamp in Roll File
Sink.*
Regards,
Rafeeq Shanavaz
“The virtue of a man ought to be measured, not by his extraordinary
exertions, but by his everyday conduct.” Blaise Pascal
Re: Flume Roll File Sink - unable to create YY/mm/dd directory
Posted by rafeeq shanavaz <ra...@gmail.com>.
To achieve it, which sink need to be used. I want to write my events on
text file on a local directory not on hdfs.
Please suggest your valuable answer.
Regards,
Rafeeq Shanavaz
“The virtue of a man ought to be measured, not by his extraordinary
exertions, but by his everyday conduct.” Blaise Pascal
On Sun, May 24, 2015 at 5:22 PM, Johny Rufus <jr...@cloudera.com> wrote:
> The Roll File Sink does not support escape sequences in the directory
> parameter and it also does not support filePrefix config.
>
> Thanks,
> Rufus
>
> On Sun, May 24, 2015 at 6:00 AM, rafeeq shanavaz <rafeeqshanavaz@gmail.com
> > wrote:
>
>> Hi,
>>
>> I am using Flume Roll-File-Sink to write data in text file on local
>> directory.
>>
>> I am trying to use /%Y/%m/%d in directory to maintain my data in daily
>> basis.
>>
>> *Agent.sinks.rollSink.sink.directory =
>> /home/cloudera/flume_events/%Y/%m/%d/*
>>
>> I am using timeStampInterceptor to achive above directory structure as
>> below:
>>
>> Agent.sources.sample.interceptors = i2
>> Agent.sources.sample.interceptors.i2.type =
>> org.apache.flume.interceptor.TimestampInterceptor$Builder
>> #Agent.sources.sample.interceptors.i2.type = timestamp
>> Agent.sources.rollSink.filePrefix = FlumeData.%Y/%m/%d
>>
>> But I am unable to create directory based on the timestamp and also i am
>> sending timestamp with event header.
>>
>>
>> Below exception is thrown:
>>
>> Caused by: java.io.FileNotFoundException:
>> /home/cloudera/flume_events/%Y/%m/%d/1432468505170-1 (No such file or
>> directory)
>> at java.io.FileOutputStream.open(Native Method)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
>> at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
>> at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>>
>>
>> *Please help me to create directory based on the timestamp in Roll File
>> Sink.*
>>
>> Regards,
>> Rafeeq Shanavaz
>>
>>
>> “The virtue of a man ought to be measured, not by his extraordinary
>> exertions, but by his everyday conduct.” Blaise Pascal
>>
>
>
Re: Flume Roll File Sink - unable to create YY/mm/dd directory
Posted by Johny Rufus <jr...@cloudera.com>.
The Roll File Sink does not support escape sequences in the directory
parameter and it also does not support filePrefix config.
Thanks,
Rufus
On Sun, May 24, 2015 at 6:00 AM, rafeeq shanavaz <ra...@gmail.com>
wrote:
> Hi,
>
> I am using Flume Roll-File-Sink to write data in text file on local
> directory.
>
> I am trying to use /%Y/%m/%d in directory to maintain my data in daily
> basis.
>
> *Agent.sinks.rollSink.sink.directory =
> /home/cloudera/flume_events/%Y/%m/%d/*
>
> I am using timeStampInterceptor to achive above directory structure as
> below:
>
> Agent.sources.sample.interceptors = i2
> Agent.sources.sample.interceptors.i2.type =
> org.apache.flume.interceptor.TimestampInterceptor$Builder
> #Agent.sources.sample.interceptors.i2.type = timestamp
> Agent.sources.rollSink.filePrefix = FlumeData.%Y/%m/%d
>
> But I am unable to create directory based on the timestamp and also i am
> sending timestamp with event header.
>
>
> Below exception is thrown:
>
> Caused by: java.io.FileNotFoundException:
> /home/cloudera/flume_events/%Y/%m/%d/1432468505170-1 (No such file or
> directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
> at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
>
>
> *Please help me to create directory based on the timestamp in Roll File
> Sink.*
>
> Regards,
> Rafeeq Shanavaz
>
>
> “The virtue of a man ought to be measured, not by his extraordinary
> exertions, but by his everyday conduct.” Blaise Pascal
>