You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Dan Young <da...@gmail.com> on 2012/11/13 22:42:37 UTC
Spooling...
I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
downloaded the branch from git, ran the mvn install, but when I try to
setup/run with an agent.source.x.type = spooldir (per the 1.3
documentation), I'm getting the following exception. Any advice/insight
would be greatly appreciated....
org.apache.flume.FlumeException: Unable to load source type: spooldir,
class: org.apache.flume.conf.source.SpoolDirectorySource
at
org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
at
org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
at
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException:
org.apache.flume.conf.source.SpoolDirectorySource
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at
org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
... 14 more
Thanx,
Dano
Re: Spooling...
Posted by Brock Noland <br...@cloudera.com>.
Good catch https://issues.apache.org/jira/browse/FLUME-1705 yes we'll
need to get this into 1.3.
On Tue, Nov 13, 2012 at 4:10 PM, Will McQueen <wi...@cloudera.com> wrote:
> more background info...
>
> SourceType.java says:
>
> SPOOLDIR("org.apache.flume.conf.source.SpoolDirectorySource");
>
> ...but should be:
>
> SPOOLDIR("org.apache.flume.source.SpoolDirectorySource");
>
> ...so for now, just specify this in your conf file:
>
> a1.sources.r1.type = org.apache.flume.source.SpoolDirectorySource
>
> ...instead of this:
>
> a1.sources.r1.type = SPOOLDIR
>
> I think we should get this change in for 1.3.0
>
>
> On Tue, Nov 13, 2012 at 2:08 PM, Will McQueen <wi...@cloudera.com> wrote:
>>
>> SourceType class needs to be updated to point to the correct FQCN. For
>> now, just specify the FQCN as the type.
>>
>>
>> On Tue, Nov 13, 2012 at 1:42 PM, Dan Young <da...@gmail.com> wrote:
>>>
>>>
>>>
>>> I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
>>> downloaded the branch from git, ran the mvn install, but when I try to
>>> setup/run with an agent.source.x.type = spooldir (per the 1.3
>>> documentation), I'm getting the following exception. Any advice/insight
>>> would be greatly appreciated....
>>>
>>>
>>>
>>>
>>>
>>> org.apache.flume.FlumeException: Unable to load source type: spooldir,
>>> class: org.apache.flume.conf.source.SpoolDirectorySource
>>> at
>>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
>>> at
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
>>> at
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:680)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.flume.conf.source.SpoolDirectorySource
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
>>> ... 14 more
>>>
>>>
>>> Thanx,
>>>
>>> Dano
>>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Spooling...
Posted by Will McQueen <wi...@cloudera.com>.
more background info...
SourceType.java says:
SPOOLDIR("org.apache.flume.conf.source.SpoolDirectorySource");
...but should be:
SPOOLDIR("org.apache.flume.source.SpoolDirectorySource");
...so for now, just specify this in your conf file:
a1.sources.r1.*type* = org.apache.flume.source.SpoolDirectorySource
...instead of this:
a1.sources.r1.*type* = SPOOLDIR
I think we should get this change in for 1.3.0
On Tue, Nov 13, 2012 at 2:08 PM, Will McQueen <wi...@cloudera.com> wrote:
> SourceType class needs to be updated to point to the correct FQCN. For
> now, just specify the FQCN as the type.
>
>
> On Tue, Nov 13, 2012 at 1:42 PM, Dan Young <da...@gmail.com> wrote:
>
>>
>>
>> I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
>> downloaded the branch from git, ran the mvn install, but when I try to
>> setup/run with an agent.source.x.type = spooldir (per the 1.3
>> documentation), I'm getting the following exception. Any advice/insight
>> would be greatly appreciated....
>>
>>
>>
>>
>>
>> org.apache.flume.FlumeException: Unable to load source type: spooldir,
>> class: org.apache.flume.conf.source.SpoolDirectorySource
>> at
>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
>> at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
>> at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.flume.conf.source.SpoolDirectorySource
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at
>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
>> ... 14 more
>>
>>
>> Thanx,
>>
>> Dano
>>
>>
>
Re: Spooling...
Posted by Dan Young <da...@gmail.com>.
I got it working....pounding on it now :) Thanx guys....yea, I saw that
typo conf....
On Tue, Nov 13, 2012 at 3:32 PM, Dan Young <da...@gmail.com> wrote:
> Great , thank you. I'll try that.
>
> Regards
>
> Dano
> On Nov 13, 2012 3:09 PM, "Will McQueen" <wi...@cloudera.com> wrote:
>
>> SourceType class needs to be updated to point to the correct FQCN. For
>> now, just specify the FQCN as the type.
>>
>> On Tue, Nov 13, 2012 at 1:42 PM, Dan Young <da...@gmail.com> wrote:
>>
>>>
>>>
>>> I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
>>> downloaded the branch from git, ran the mvn install, but when I try to
>>> setup/run with an agent.source.x.type = spooldir (per the 1.3
>>> documentation), I'm getting the following exception. Any advice/insight
>>> would be greatly appreciated....
>>>
>>>
>>>
>>>
>>>
>>> org.apache.flume.FlumeException: Unable to load source type: spooldir,
>>> class: org.apache.flume.conf.source.SpoolDirectorySource
>>> at
>>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
>>> at
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
>>> at
>>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>>> at
>>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:680)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.flume.conf.source.SpoolDirectorySource
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> at java.lang.Class.forName0(Native Method)
>>> at java.lang.Class.forName(Class.java:169)
>>> at
>>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
>>> ... 14 more
>>>
>>>
>>> Thanx,
>>>
>>> Dano
>>>
>>>
>>
Re: Spooling...
Posted by Dan Young <da...@gmail.com>.
Great , thank you. I'll try that.
Regards
Dano
On Nov 13, 2012 3:09 PM, "Will McQueen" <wi...@cloudera.com> wrote:
> SourceType class needs to be updated to point to the correct FQCN. For
> now, just specify the FQCN as the type.
>
> On Tue, Nov 13, 2012 at 1:42 PM, Dan Young <da...@gmail.com> wrote:
>
>>
>>
>> I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
>> downloaded the branch from git, ran the mvn install, but when I try to
>> setup/run with an agent.source.x.type = spooldir (per the 1.3
>> documentation), I'm getting the following exception. Any advice/insight
>> would be greatly appreciated....
>>
>>
>>
>>
>>
>> org.apache.flume.FlumeException: Unable to load source type: spooldir,
>> class: org.apache.flume.conf.source.SpoolDirectorySource
>> at
>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
>> at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
>> at
>> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>> at
>> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.flume.conf.source.SpoolDirectorySource
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:169)
>> at
>> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
>> ... 14 more
>>
>>
>> Thanx,
>>
>> Dano
>>
>>
>
Re: Spooling...
Posted by Will McQueen <wi...@cloudera.com>.
SourceType class needs to be updated to point to the correct FQCN. For now,
just specify the FQCN as the type.
On Tue, Nov 13, 2012 at 1:42 PM, Dan Young <da...@gmail.com> wrote:
>
>
> I'm curious where FLUME-1425 stands? I see it resolved in 1.3, so I
> downloaded the branch from git, ran the mvn install, but when I try to
> setup/run with an agent.source.x.type = spooldir (per the 1.3
> documentation), I'm getting the following exception. Any advice/insight
> would be greatly appreciated....
>
>
>
>
>
> org.apache.flume.FlumeException: Unable to load source type: spooldir,
> class: org.apache.flume.conf.source.SpoolDirectorySource
> at
> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:93)
> at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:319)
> at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:222)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flume.conf.source.SpoolDirectorySource
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:169)
> at
> org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:91)
> ... 14 more
>
>
> Thanx,
>
> Dano
>
>