You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Steve Johnson <st...@webninja.com> on 2012/08/29 17:38:30 UTC

Unable to start FileChannel

Good morning.  I'm trying to test Flume for use for our companies log
collection.  These are application transaction logs (not syslogs) so I need
the agents channel data to be fully recoverable.  I have done some initial
setup testing using the memory channel for my initial tests and it's been
going well.  However, I'm unable to get the FileChannel operable.  Below is
the exception I'm getting.  I'm currently running this on OSX 10.7.4
(Darwin Kernel Version 11.4.0).  I've installed using the flume-ng 1.2.0
binary.

I searched the flume mail lists and didn't find anything, but I was able to
find something on google here, but I didn't see any more updates on the
issues since late June:
http://www.mail-archive.com/flume-dev@incubator.apache.org/msg07419.html

I've pasted the exception below I'm getting at startup.  Also, here is my
confiugration, thanks in advance for any help:
testagent.sources = netcatSource
testagent.channels = fileChannel
testagent.sinks = fileSink

# For each one of the sources, the type is defined
testagent.sources.netcatSource.type = netcat

# The channel can be defined as follows.
testagent.sources.netcatSource.channels = fileChannel

# Each sink's type must be defined
testagent.sinks.fileSink.type = FILE_ROLL

#Specify the channel the sink should use
testagent.sinks.fileSink.channel = fileChannel

# Each channel's type is defined.
testagent.channels.fileChannel.type = file

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
testagent.sources.netcatSource.bind = 0.0.0.0
testagent.sources.netcatSource.port = 6166
testagent.sources.netcatSource.max-line-length = 32768
testagent.channels.fileChannel.transactionCapacity = 1024000
testagent.channels.fileChannel.maxFileSize = 10240000000
testagent.channels.fileChannel.checkpointDir =
/opt/dotomi/flume/file_channel/checkpoint
testagent.channels.fileChannel.dataDirs =
/opt/dotomi/flume/file_channel/data
testagent.channels.fileChannel.capacity = 512000000
testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir
testagent.sinks.fileSink.sink.rollInterval = 0


2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
Starting new configuration:{
sourceRunners:{netcatSource=EventDrivenSourceRunner: { source:o
rg.apache.flume.source.NetcatSource@3c50507 }}
sinkRunners:{fileSink=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107f counterGroup:{
nam
e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel {
dataDirs: [/opt/dotomi/flume/file_channel/data] }} }
2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
Starting Channel fileChannel
2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Waiting
for channel: fileChannel to start. Sleeping for 500 ms
2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel
fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to
start FileChannel fileChannel { dataDirs:
[/opt/dotomi/flume/file_channel/data] } - Excep
tion follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
        at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at org.apache.flume.channel.file.Log$Builder.build(Log.java:139)
        at
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226)
        at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        ... 24 more
2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel
fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...

-- 
Steve Johnson
steve@webninja.com

Re: Unable to start FileChannel

Posted by Steve Johnson <st...@webninja.com>.
Thanks, easy enough solution, this worked for me.

On Wed, Aug 29, 2012 at 10:42 AM, Brock Noland <br...@cloudera.com> wrote:

> Hi,
>
> This is my fault. At this point FC is dependent on Hadoop. We want to
> remove it, but it will take some time. This is in
> https://issues.apache.org/jira/browse/FLUME-1285
>
> The simplest way to solve this is to put the hadoop command in your
> path before you start flume.
>
> Brock
>
> On Wed, Aug 29, 2012 at 10:38 AM, Steve Johnson <st...@webninja.com>
> wrote:
> > Good morning.  I'm trying to test Flume for use for our companies log
> > collection.  These are application transaction logs (not syslogs) so I
> need
> > the agents channel data to be fully recoverable.  I have done some
> initial
> > setup testing using the memory channel for my initial tests and it's been
> > going well.  However, I'm unable to get the FileChannel operable.  Below
> is
> > the exception I'm getting.  I'm currently running this on OSX 10.7.4
> (Darwin
> > Kernel Version 11.4.0).  I've installed using the flume-ng 1.2.0 binary.
> >
> > I searched the flume mail lists and didn't find anything, but I was able
> to
> > find something on google here, but I didn't see any more updates on the
> > issues since late June:
> > http://www.mail-archive.com/flume-dev@incubator.apache.org/msg07419.html
> >
> > I've pasted the exception below I'm getting at startup.  Also, here is my
> > confiugration, thanks in advance for any help:
> > testagent.sources = netcatSource
> > testagent.channels = fileChannel
> > testagent.sinks = fileSink
> >
> > # For each one of the sources, the type is defined
> > testagent.sources.netcatSource.type = netcat
> >
> > # The channel can be defined as follows.
> > testagent.sources.netcatSource.channels = fileChannel
> >
> > # Each sink's type must be defined
> > testagent.sinks.fileSink.type = FILE_ROLL
> >
> > #Specify the channel the sink should use
> > testagent.sinks.fileSink.channel = fileChannel
> >
> > # Each channel's type is defined.
> > testagent.channels.fileChannel.type = file
> >
> > # Other config values specific to each type of channel(sink or source)
> > # can be defined as well
> > # In this case, it specifies the capacity of the memory channel
> > testagent.sources.netcatSource.bind = 0.0.0.0
> > testagent.sources.netcatSource.port = 6166
> > testagent.sources.netcatSource.max-line-length = 32768
> > testagent.channels.fileChannel.transactionCapacity = 1024000
> > testagent.channels.fileChannel.maxFileSize = 10240000000
> > testagent.channels.fileChannel.checkpointDir =
> > /opt/dotomi/flume/file_channel/checkpoint
> > testagent.channels.fileChannel.dataDirs =
> > /opt/dotomi/flume/file_channel/data
> > testagent.channels.fileChannel.capacity = 512000000
> > testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir
> > testagent.sinks.fileSink.sink.rollInterval = 0
> >
> >
> > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> > new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner:
> {
> > source:o
> > rg.apache.flume.source.NetcatSource@3c50507 }}
> > sinkRunners:{fileSink=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107fcounterGroup:{
> > nam
> > e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel {
> > dataDirs: [/opt/dotomi/flume/file_channel/data] }} }
> > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
> Starting
> > Channel fileChannel
> > 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager:
> Waiting
> > for channel: fileChannel to start. Sleeping for 500 ms
> > 2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel
> > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
> > 2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to
> start
> > FileChannel fileChannel { dataDirs:
> [/opt/dotomi/flume/file_channel/data] }
> > - Excep
> > tion follows.
> > java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
> >         at java.lang.ClassLoader.defineClass1(Native Method)
> >         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
> >         at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
> >         at
> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >         at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >         at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >         at org.apache.flume.channel.file.Log$Builder.build(Log.java:139)
> >         at
> > org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226)
> >         at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >         at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >         at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >         at java.lang.Thread.run(Thread.java:680)
> > Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.io.Writable
> >         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >         ... 24 more
> > 2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel
> > fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
> >
> > --
> > Steve Johnson
> > steve@webninja.com
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>



-- 
Steve Johnson
steve@webninja.com

Re: Unable to start FileChannel

Posted by Brock Noland <br...@cloudera.com>.
Hi,

This is my fault. At this point FC is dependent on Hadoop. We want to
remove it, but it will take some time. This is in
https://issues.apache.org/jira/browse/FLUME-1285

The simplest way to solve this is to put the hadoop command in your
path before you start flume.

Brock

On Wed, Aug 29, 2012 at 10:38 AM, Steve Johnson <st...@webninja.com> wrote:
> Good morning.  I'm trying to test Flume for use for our companies log
> collection.  These are application transaction logs (not syslogs) so I need
> the agents channel data to be fully recoverable.  I have done some initial
> setup testing using the memory channel for my initial tests and it's been
> going well.  However, I'm unable to get the FileChannel operable.  Below is
> the exception I'm getting.  I'm currently running this on OSX 10.7.4 (Darwin
> Kernel Version 11.4.0).  I've installed using the flume-ng 1.2.0 binary.
>
> I searched the flume mail lists and didn't find anything, but I was able to
> find something on google here, but I didn't see any more updates on the
> issues since late June:
> http://www.mail-archive.com/flume-dev@incubator.apache.org/msg07419.html
>
> I've pasted the exception below I'm getting at startup.  Also, here is my
> confiugration, thanks in advance for any help:
> testagent.sources = netcatSource
> testagent.channels = fileChannel
> testagent.sinks = fileSink
>
> # For each one of the sources, the type is defined
> testagent.sources.netcatSource.type = netcat
>
> # The channel can be defined as follows.
> testagent.sources.netcatSource.channels = fileChannel
>
> # Each sink's type must be defined
> testagent.sinks.fileSink.type = FILE_ROLL
>
> #Specify the channel the sink should use
> testagent.sinks.fileSink.channel = fileChannel
>
> # Each channel's type is defined.
> testagent.channels.fileChannel.type = file
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> testagent.sources.netcatSource.bind = 0.0.0.0
> testagent.sources.netcatSource.port = 6166
> testagent.sources.netcatSource.max-line-length = 32768
> testagent.channels.fileChannel.transactionCapacity = 1024000
> testagent.channels.fileChannel.maxFileSize = 10240000000
> testagent.channels.fileChannel.checkpointDir =
> /opt/dotomi/flume/file_channel/checkpoint
> testagent.channels.fileChannel.dataDirs =
> /opt/dotomi/flume/file_channel/data
> testagent.channels.fileChannel.capacity = 512000000
> testagent.sinks.fileSink.sink.directory = /opt/dotomi/flume/sinkdir
> testagent.sinks.fileSink.sink.rollInterval = 0
>
>
> 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting
> new configuration:{ sourceRunners:{netcatSource=EventDrivenSourceRunner: {
> source:o
> rg.apache.flume.source.NetcatSource@3c50507 }}
> sinkRunners:{fileSink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@7bcd107f counterGroup:{
> nam
> e:null counters:{} } }} channels:{fileChannel=FileChannel fileChannel {
> dataDirs: [/opt/dotomi/flume/file_channel/data] }} }
> 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel fileChannel
> 2012-08-28 16:32:39,162 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> for channel: fileChannel to start. Sleeping for 500 ms
> 2012-08-28 16:32:39,162 INFO file.FileChannel: Starting FileChannel
> fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
> 2012-08-28 16:32:39,165 ERROR lifecycle.LifecycleSupervisor: Unable to start
> FileChannel fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }
> - Excep
> tion follows.
> java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
>         at java.lang.ClassLoader.defineClass1(Native Method)
>         at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>         at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>         at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         at org.apache.flume.channel.file.Log$Builder.build(Log.java:139)
>         at
> org.apache.flume.channel.file.FileChannel.start(FileChannel.java:226)
>         at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:237)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         ... 24 more
> 2012-08-28 16:32:39,167 INFO file.FileChannel: Stopping FileChannel
> fileChannel { dataDirs: [/opt/dotomi/flume/file_channel/data] }...
>
> --
> Steve Johnson
> steve@webninja.com



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/