You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Lenin Raj <em...@gmail.com> on 2013/06/14 21:04:44 UTC
Can we use escape sequences in file_roll sink?
Hello,
I tried to use %Y in my sink.directory in the file_roll sink but it
resulted in exception:
15 Jun 2013 00:26:20,596 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to open file
/var/emaillenin/card/data/%Y/1371236170080-1 while delivering event
at
org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException:
/var/emaillenin/card/data/%Y/1371236170080-1 (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at
org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
... 3 more
When I remove %Y from the path and tried again, the files get created
successfuly. Are there any ways to use these escape sequences in file_roll
sink?
Thanks,
Lenin
Re: Can we use escape sequences in file_roll sink?
Posted by Hari Shreedharan <hs...@cloudera.com>.
The main implementation is in the BucketPath class. It should not be too difficult to create the path based on that.
Cheers,
Hari
On Friday, June 14, 2013 at 12:22 PM, Lenin Raj wrote:
> Hi Hari,
>
> Here is the ticket - https://issues.apache.org/jira/browse/FLUME-1902
>
> I will see how it is done in HDFS sink.
>
>
> Thanks,
> Lenin
>
>
> On Sat, Jun 15, 2013 at 12:40 AM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > Currently escaping is not supported by File Roll Sink, but I think there is a jira to enable it - perhaps you want to submit a patch?
> >
> >
> > Cheers,
> > Hari
> >
> >
> > On Friday, June 14, 2013 at 12:04 PM, Lenin Raj wrote:
> >
> > > Hello,
> > >
> > > I tried to use %Y in my sink.directory in the file_roll sink but it resulted in exception:
> > >
> > > 15 Jun 2013 00:26:20,596 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
> > > org.apache.flume.EventDeliveryException: Failed to open file /var/emaillenin/card/data/%Y/1371236170080-1 while delivering event
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> > > at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:662)
> > > Caused by: java.io.FileNotFoundException: /var/emaillenin/card/data/%Y/1371236170080-1 (No such file or directory)
> > > at java.io.FileOutputStream.open(Native Method)
> > > at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
> > > at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
> > > at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
> > > ... 3 more
> > >
> > > When I remove %Y from the path and tried again, the files get created successfuly. Are there any ways to use these escape sequences in file_roll sink?
> > >
> > > Thanks,
> > > Lenin
> >
>
Re: Can we use escape sequences in file_roll sink?
Posted by Lenin Raj <em...@gmail.com>.
Hi Hari,
Here is the ticket - https://issues.apache.org/jira/browse/FLUME-1902
I will see how it is done in HDFS sink.
Thanks,
Lenin
On Sat, Jun 15, 2013 at 12:40 AM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:
> Currently escaping is not supported by File Roll Sink, but I think there
> is a jira to enable it - perhaps you want to submit a patch?
>
>
> Cheers,
> Hari
>
> On Friday, June 14, 2013 at 12:04 PM, Lenin Raj wrote:
>
> Hello,
>
> I tried to use %Y in my sink.directory in the file_roll sink but it
> resulted in exception:
>
> 15 Jun 2013 00:26:20,596 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to open file
> /var/emaillenin/card/data/%Y/1371236170080-1 while delivering event
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException:
> /var/emaillenin/card/data/%Y/1371236170080-1 (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
> ... 3 more
>
> When I remove %Y from the path and tried again, the files get created
> successfuly. Are there any ways to use these escape sequences in file_roll
> sink?
>
> Thanks,
> Lenin
>
>
>
Re: Can we use escape sequences in file_roll sink?
Posted by Hari Shreedharan <hs...@cloudera.com>.
Currently escaping is not supported by File Roll Sink, but I think there is a jira to enable it - perhaps you want to submit a patch?
Cheers,
Hari
On Friday, June 14, 2013 at 12:04 PM, Lenin Raj wrote:
> Hello,
>
> I tried to use %Y in my sink.directory in the file_roll sink but it resulted in exception:
>
> 15 Jun 2013 00:26:20,596 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to open file /var/emaillenin/card/data/%Y/1371236170080-1 while delivering event
> at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException: /var/emaillenin/card/data/%Y/1371236170080-1 (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
> at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
> ... 3 more
>
> When I remove %Y from the path and tried again, the files get created successfuly. Are there any ways to use these escape sequences in file_roll sink?
>
> Thanks,
> Lenin
Re: Can we use escape sequences in file_roll sink?
Posted by Lenin Raj <em...@gmail.com>.
Hi Juhani,
I have not tried with that. But that is not the recommended way right? We
might have additional properties for file_roll sink in the future, maybe.
Thanks,
Lenin
On Mon, Jun 17, 2013 at 7:45 AM, Juhani Connolly <
juhani_connolly@cyberagent.co.jp> wrote:
> Hi Lenin,
>
> Instead of using the file_roll sink, we just use the HDFS sink with a
> local path and it works fine. Could that work for you
>
>
> On 06/15/2013 04:04 AM, Lenin Raj wrote:
>
>> Hello,
>>
>> I tried to use %Y in my sink.directory in the file_roll sink but it
>> resulted in exception:
>>
>> 15 Jun 2013 00:26:20,596 ERROR [SinkRunner-PollingRunner-**DefaultSinkProcessor]
>> (org.apache.flume.SinkRunner$**PollingRunner.run:160) - Unable to
>> deliver event. Exception follows.
>> org.apache.flume.**EventDeliveryException: Failed to open file
>> /var/emaillenin/card/data/%Y/**1371236170080-1 while delivering event
>> at org.apache.flume.sink.**RollingFileSink.process(**
>> RollingFileSink.java:177)
>> at org.apache.flume.sink.**DefaultSinkProcessor.process(**
>> DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$**PollingRunner.run(SinkRunner.**
>> java:147)
>> at java.lang.Thread.run(Thread.**java:662)
>> Caused by: java.io.FileNotFoundException: /var/emaillenin/card/data/%Y/**1371236170080-1
>> (No such file or directory)
>> at java.io.FileOutputStream.open(**Native Method)
>> at java.io.FileOutputStream.<**init>(FileOutputStream.java:**194)
>> at java.io.FileOutputStream.<**init>(FileOutputStream.java:**145)
>> at org.apache.flume.sink.**RollingFileSink.process(**
>> RollingFileSink.java:169)
>> ... 3 more
>>
>> When I remove %Y from the path and tried again, the files get created
>> successfuly. Are there any ways to use these escape sequences in file_roll
>> sink?
>>
>> Thanks,
>> Lenin
>>
>
>
Re: Can we use escape sequences in file_roll sink?
Posted by Juhani Connolly <ju...@cyberagent.co.jp>.
Hi Lenin,
Instead of using the file_roll sink, we just use the HDFS sink with a
local path and it works fine. Could that work for you
On 06/15/2013 04:04 AM, Lenin Raj wrote:
> Hello,
>
> I tried to use %Y in my sink.directory in the file_roll sink but it
> resulted in exception:
>
> 15 Jun 2013 00:26:20,596 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to
> deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to open file
> /var/emaillenin/card/data/%Y/1371236170080-1 while delivering event
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:177)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException:
> /var/emaillenin/card/data/%Y/1371236170080-1 (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
> at
> org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:169)
> ... 3 more
>
> When I remove %Y from the path and tried again, the files get created
> successfuly. Are there any ways to use these escape sequences in
> file_roll sink?
>
> Thanks,
> Lenin