You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Tzur Turkenitz <tz...@vision.bi> on 2013/01/31 17:56:41 UTC
SpoolDir sub-directories
Hi All,
I am experimenting with Flumes` 1.3 new SpoolDir adapter.
I was wondering is there an option to tell the adapter to walk down
directories and search for files.
Currently, if I have a directory within the source location I get java
exceptions.
13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
/home/fl/source/dir
java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileReader.<init>(FileReader.java:55)
at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Thanks!
--
Regards,
Tzur Turkenitz
Vision.BI
http://www.vision.bi/
"*Facts are stubborn things, but statistics are more pliable*"
-Mark Twain
RE: SpoolDir sub-directories
Posted by Tzur Turkenitz <tz...@vision.bi>.
Will do. Thanks!
From: Brock Noland [mailto:brock@cloudera.com]
Sent: Monday, February 4, 2013 11:14 AM
To: user@flume.apache.org
Subject: Re: SpoolDir sub-directories
Hi,
I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?
Brock
On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <tzurt@vision.bi
<ma...@vision.bi> > wrote:
Hi,
I couldn't find any information about this in the Wiki / User guide.
Are sub directories for the Flume SpoolDir supported in 1.3x?
Thanks!
On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <tzurt@vision.bi
<ma...@vision.bi> > wrote:
Hi All,
I am experimenting with Flumes` 1.3 new SpoolDir adapter.
I was wondering is there an option to tell the adapter to walk down
directories and search for files.
Currently, if I have a directory within the source location I get java
exceptions.
13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
/home/fl/source/dir
java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileReader.<init>(FileReader.java:55)
at
org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFile
LineReader.java:322)
at
org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLi
neReader.java:172)
at
org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(Spoo
lDirectorySource.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
at java.lang.Thread.run(Thread.java:662)
Thanks!
--
Regards,
Tzur Turkenitz
Vision.BI
<http://www.vision.bi/> http://www.vision.bi/
"Facts are stubborn things, but statistics are more pliable"
-Mark Twain
--
Regards,
Tzur Turkenitz
Vision.BI
<http://www.vision.bi/> http://www.vision.bi/
"Facts are stubborn things, but statistics are more pliable"
-Mark Twain
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: SpoolDir sub-directories
Posted by Brock Noland <br...@cloudera.com>.
Hi,
I don't believe sub directories are supported at this time. Perhaps you
should file a JIRA for this improvement?
Brock
On Mon, Feb 4, 2013 at 10:01 AM, Tzur Turkenitz <tz...@vision.bi> wrote:
> Hi,
> I couldn't find any information about this in the Wiki / User guide.
> Are sub directories for the Flume SpoolDir supported in 1.3x?
> Thanks!
>
>
> On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <tz...@vision.bi> wrote:
>
>> Hi All,
>>
>> I am experimenting with Flumes` 1.3 new SpoolDir adapter.
>> I was wondering is there an option to tell the adapter to walk down
>> directories and search for files.
>>
>> Currently, if I have a directory within the source location I get java
>> exceptions.
>> 13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
>> /home/fl/source/dir
>> java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
>> at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>> at java.io.FileReader.<init>(FileReader.java:55)
>> at
>> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
>> at
>> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
>> at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Thanks!
>>
>>
>>
>> --
>> Regards,
>> Tzur Turkenitz
>> Vision.BI
>> http://www.vision.bi/
>>
>> "*Facts are stubborn things, but statistics are more pliable*"
>> -Mark Twain
>>
>
>
>
> --
> Regards,
> Tzur Turkenitz
> Vision.BI
> http://www.vision.bi/
>
> "*Facts are stubborn things, but statistics are more pliable*"
> -Mark Twain
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: SpoolDir sub-directories
Posted by Tzur Turkenitz <tz...@vision.bi>.
Hi,
I couldn't find any information about this in the Wiki / User guide.
Are sub directories for the Flume SpoolDir supported in 1.3x?
Thanks!
On Thu, Jan 31, 2013 at 11:56 AM, Tzur Turkenitz <tz...@vision.bi> wrote:
> Hi All,
>
> I am experimenting with Flumes` 1.3 new SpoolDir adapter.
> I was wondering is there an option to tell the adapter to walk down
> directories and search for files.
>
> Currently, if I have a directory within the source location I get java
> exceptions.
> 13/01/30 17:41:30 WARN avro.SpoolingFileLineReader: Could not find file:
> /home/fl/source/dir
> java.io.FileNotFoundException: /home/fl/source/dir (Is a directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at java.io.FileReader.<init>(FileReader.java:55)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> at
> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Thanks!
>
>
>
> --
> Regards,
> Tzur Turkenitz
> Vision.BI
> http://www.vision.bi/
>
> "*Facts are stubborn things, but statistics are more pliable*"
> -Mark Twain
>
--
Regards,
Tzur Turkenitz
Vision.BI
http://www.vision.bi/
"*Facts are stubborn things, but statistics are more pliable*"
-Mark Twain