You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "lexsoto@gmail.com" <le...@gmail.com> on 2012/05/11 22:02:45 UTC

Unable to use HDFS as sink

Hello,

I am new to Flume.  I am trying to use hdfs as a sink.  My first
problem is that flume hangs during initialization.
Same problem as:

  https://issues.apache.org/jira/browse/FLUME-1077

and

  https://issues.apache.org/jira/browse/FLUME-1093

My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
"flume-ng-1.1.0-cdh3u4"
I noticed that Hadoop and Flume have different versions of Guava.

If I remove Flume's Guava jar and use Hadoop's:

<code>
bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
--classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
</code>

I get this error:

<code>
Exception in thread "main" java.lang.NoClassDefFoundError:
com/google/common/util/concurrent/ThreadFactoryBuilder
	at org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
	at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
	at org.apache.flume.node.Application.run(Application.java:135)
	at org.apache.flume.node.Application.main(Application.java:70)
Caused by: java.lang.ClassNotFoundException:
com.google.common.util.concurrent.ThreadFactoryBuilder
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 4 more
</code>

If I put back Flume's Guava jar:

<code>
bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
--classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
</code>

I get this error:

<code>
Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
java.lang.NoClassDefFoundError:
org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
	at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
	at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
	at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
	at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
	at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
	at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
	at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 21 more
</code>

Has anybody been able to successfully run Flume with a HDFS sink?
What's the magic combination?

-- 
Alex Soto
lexsoto@gmail.com

-
Amicus Plato, sed magis amica veritas.

Re: Unable to use HDFS as sink

Posted by Mike Percy <mp...@cloudera.com>.
Great! Thanks for the update Alex.

Mike 


On Wednesday, May 16, 2012 at 9:30 AM, Alex Soto wrote:

> Just to let everyone know;  after letting it rest for a few days, it is magically working again.  I do not know what was the issue since I tested again with fully qualified paths and with relative paths and it works both ways.
> 
> Thanks again
> -Alex 
> 
> 
> --
> Amicus Plato, sed magis amica veritas.
> 
> 
> On May 14, 2012, at 5:34 PM, Mike Percy <mpercy@cloudera.com (mailto:mpercy@cloudera.com)> wrote:
> 
> > Another useful thing to troubleshoot this would be if you include the output from invoking flume. When you invoke the avro client it should dump the command line that is being used to invoke it. 
> > 
> > Mike 
> > 
> > On Monday, May 14, 2012 at 1:15 PM, Eric Sammer wrote:
> > 
> > > We've seen some odd behavior if the value of HADOOP_HOME isn't an absolute path. I think I already filed a JIRA about this.
> > > 
> > > On Mon, May 14, 2012 at 6:59 AM, lexsoto@gmail.com (mailto:lexsoto@gmail.com) <lexsoto@gmail.com (mailto:lexsoto@gmail.com)> wrote:
> > > > Thanks for the help.  This is working now after setting the
> > > > HADOOP_HOME variable.
> > > > Now the problem is testing it.
> > > > 
> > > > In a different terminal I am typing:
> > > > 
> > > >  export HADOOP_HOME=~/hadoop-0.20.2-cdh3u4
> > > >  bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F server.log
> > > > 
> > > > but I am getting the exception:
> > > > 
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > com/google/common/collect/Lists
> > > >        at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:170)
> > > >        at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> > > > Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists
> > > >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > >        at java.security.AccessController.doPrivileged(Native Method)
> > > >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > >        ... 2 more
> > > > 
> > > > Any idea?
> > > > 
> > > > Thanks
> > > > 
> > > > On Fri, May 11, 2012 at 9:25 PM, Mike Percy <mpercy@cloudera.com (mailto:mpercy@cloudera.com)> wrote:
> > > > > Alright, well cdh-user@ is publicly viewable I believe but here is the most
> > > > > important bit of what I told Alex:
> > > > >
> > > > > You need to set HADOOP_HOME variable or else Flume will not be able to find
> > > > > the HDFS jars if you have put then in some non-standard place such as your
> > > > > home directory.
> > > > >
> > > > > Also it turns out that he is running Hadoop on Mac which is not something
> > > > > I'm really familiar with but CDH doesn't target it from a QA perspective.
> > > > >
> > > > > HTH,
> > > > > Mike
> > > > >
> > > > > On May 11, 2012, at 2:09 PM, Ralph Goers wrote:
> > > > >
> > > > > A word of caution here.  If these issues are also relevant to the Apache
> > > > > distribution then some form of the discussion should come back here so that
> > > > > it benefits users of the vanilla distribution.
> > > > >
> > > > > Ralph
> > > > >
> > > > > On May 11, 2012, at 1:17 PM, Mike Percy wrote:
> > > > >
> > > > > bcc: flume-user@incubator.apache.org (mailto:flume-user@incubator.apache.org)
> > > > >
> > > > > This issue involves a version of Flume distributed with CDH, so moving to
> > > > > cdh-user@cloudera.org (mailto:cdh-user@cloudera.org) which is the proper support channel.
> > > > >
> > > > > Thanks,
> > > > > Mike
> > > > >
> > > > > On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com (mailto:lexsoto@gmail.com) wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I am new to Flume. I am trying to use hdfs as a sink. My first
> > > > > problem is that flume hangs during initialization.
> > > > > Same problem as:
> > > > >
> > > > > https://issues.apache.org/jira/browse/FLUME-1077
> > > > >
> > > > > and
> > > > >
> > > > > https://issues.apache.org/jira/browse/FLUME-1093
> > > > >
> > > > > My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
> > > > > "flume-ng-1.1.0-cdh3u4"
> > > > > I noticed that Hadoop and Flume have different versions of Guava.
> > > > >
> > > > > If I remove Flume's Guava jar and use Hadoop's:
> > > > >
> > > > > <code>
> > > > > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > > > > --classpath
> > > > > ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
> > > > > </code>
> > > > >
> > > > > I get this error:
> > > > >
> > > > > <code>
> > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > com/google/common/util/concurrent/ThreadFactoryBuilder
> > > > > at
> > > > > org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
> > > > > at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
> > > > > at org.apache.flume.node.Application.run(Application.java:135)
> > > > > at org.apache.flume.node.Application.main(Application.java:70)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > com.google.common.util.concurrent.ThreadFactoryBuilder
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > > > ... 4 more
> > > > > </code>
> > > > >
> > > > > If I put back Flume's Guava jar:
> > > > >
> > > > > <code>
> > > > > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > > > > --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
> > > > > </code>
> > > > >
> > > > > I get this error:
> > > > >
> > > > > <code>
> > > > > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> > > > > java.lang.NoClassDefFoundError:
> > > > > org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
> > > > > at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
> > > > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
> > > > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
> > > > > at
> > > > > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > > > > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
> > > > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> > > > > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
> > > > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
> > > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
> > > > > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
> > > > > at
> > > > > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
> > > > > at
> > > > > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
> > > > > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
> > > > > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
> > > > > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
> > > > > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
> > > > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > at java.lang.Thread.run(Thread.java:680)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
> > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > > > ... 21 more
> > > > > </code>
> > > > >
> > > > > Has anybody been able to successfully run Flume with a HDFS sink?
> > > > > What's the magic combination?
> > > > >
> > > > > --
> > > > > Alex Soto
> > > > > lexsoto@gmail.com (mailto:lexsoto@gmail.com)
> > > > >
> > > > > -
> > > > > Amicus Plato, sed magis amica veritas.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > 
> > > > 
> > > > 
> > > > --
> > > > Alex Soto
> > > > lexsoto@gmail.com (mailto:lexsoto@gmail.com)
> > > > 
> > > > -
> > > > Amicus Plato, sed magis amica veritas.
> > > 
> > > 
> > > 
> > > -- 
> > > Eric Sammer
> > > twitter: esammer
> > > data: www.cloudera.com (http://www.cloudera.com)
> >

Re: Unable to use HDFS as sink

Posted by Alex Soto <le...@gmail.com>.
Just to let everyone know;  after letting it rest for a few days, it is magically working again.  I do not know what was the issue since I tested again with fully qualified paths and with relative paths and it works both ways.

Thanks again
-Alex 


--
Amicus Plato, sed magis amica veritas.

On May 14, 2012, at 5:34 PM, Mike Percy <mp...@cloudera.com> wrote:

> Another useful thing to troubleshoot this would be if you include the output from invoking flume. When you invoke the avro client it should dump the command line that is being used to invoke it.
> 
> Mike
> On Monday, May 14, 2012 at 1:15 PM, Eric Sammer wrote:
> 
>> We've seen some odd behavior if the value of HADOOP_HOME isn't an absolute path. I think I already filed a JIRA about this.
>> 
>> On Mon, May 14, 2012 at 6:59 AM, lexsoto@gmail.com <le...@gmail.com> wrote:
>>> Thanks for the help.  This is working now after setting the
>>> HADOOP_HOME variable.
>>> Now the problem is testing it.
>>> 
>>> In a different terminal I am typing:
>>> 
>>>  export HADOOP_HOME=~/hadoop-0.20.2-cdh3u4
>>>  bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F server.log
>>> 
>>> but I am getting the exception:
>>> 
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/common/collect/Lists
>>>        at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:170)
>>>        at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
>>> Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists
>>>        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>        ... 2 more
>>> 
>>> Any idea?
>>> 
>>> Thanks
>>> 
>>> On Fri, May 11, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
>>> > Alright, well cdh-user@ is publicly viewable I believe but here is the most
>>> > important bit of what I told Alex:
>>> >
>>> > You need to set HADOOP_HOME variable or else Flume will not be able to find
>>> > the HDFS jars if you have put then in some non-standard place such as your
>>> > home directory.
>>> >
>>> > Also it turns out that he is running Hadoop on Mac which is not something
>>> > I'm really familiar with but CDH doesn't target it from a QA perspective.
>>> >
>>> > HTH,
>>> > Mike
>>> >
>>> > On May 11, 2012, at 2:09 PM, Ralph Goers wrote:
>>> >
>>> > A word of caution here.  If these issues are also relevant to the Apache
>>> > distribution then some form of the discussion should come back here so that
>>> > it benefits users of the vanilla distribution.
>>> >
>>> > Ralph
>>> >
>>> > On May 11, 2012, at 1:17 PM, Mike Percy wrote:
>>> >
>>> > bcc: flume-user@incubator.apache.org
>>> >
>>> > This issue involves a version of Flume distributed with CDH, so moving to
>>> > cdh-user@cloudera.org which is the proper support channel.
>>> >
>>> > Thanks,
>>> > Mike
>>> >
>>> > On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:
>>> >
>>> > Hello,
>>> >
>>> > I am new to Flume. I am trying to use hdfs as a sink. My first
>>> > problem is that flume hangs during initialization.
>>> > Same problem as:
>>> >
>>> > https://issues.apache.org/jira/browse/FLUME-1077
>>> >
>>> > and
>>> >
>>> > https://issues.apache.org/jira/browse/FLUME-1093
>>> >
>>> > My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
>>> > "flume-ng-1.1.0-cdh3u4"
>>> > I noticed that Hadoop and Flume have different versions of Guava.
>>> >
>>> > If I remove Flume's Guava jar and use Hadoop's:
>>> >
>>> > <code>
>>> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>>> > --classpath
>>> > ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
>>> > </code>
>>> >
>>> > I get this error:
>>> >
>>> > <code>
>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>> > com/google/common/util/concurrent/ThreadFactoryBuilder
>>> > at
>>> > org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
>>> > at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
>>> > at org.apache.flume.node.Application.run(Application.java:135)
>>> > at org.apache.flume.node.Application.main(Application.java:70)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> > com.google.common.util.concurrent.ThreadFactoryBuilder
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> > ... 4 more
>>> > </code>
>>> >
>>> > If I put back Flume's Guava jar:
>>> >
>>> > <code>
>>> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>>> > --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
>>> > </code>
>>> >
>>> > I get this error:
>>> >
>>> > <code>
>>> > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
>>> > java.lang.NoClassDefFoundError:
>>> > org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
>>> > at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
>>> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
>>> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
>>> > at
>>> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>>> > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
>>> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
>>> > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
>>> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
>>> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>>> > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>>> > at
>>> > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
>>> > at
>>> > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
>>> > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
>>> > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
>>> > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
>>> > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
>>> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:680)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> > org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> > ... 21 more
>>> > </code>
>>> >
>>> > Has anybody been able to successfully run Flume with a HDFS sink?
>>> > What's the magic combination?
>>> >
>>> > --
>>> > Alex Soto
>>> > lexsoto@gmail.com
>>> >
>>> > -
>>> > Amicus Plato, sed magis amica veritas.
>>> >
>>> >
>>> >
>>> >
>>> 
>>> 
>>> 
>>> --
>>> Alex Soto
>>> lexsoto@gmail.com
>>> 
>>> -
>>> Amicus Plato, sed magis amica veritas.
>> 
>> 
>> 
>> -- 
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>

Re: Unable to use HDFS as sink

Posted by Mike Percy <mp...@cloudera.com>.
Another useful thing to troubleshoot this would be if you include the output from invoking flume. When you invoke the avro client it should dump the command line that is being used to invoke it. 

Mike 


On Monday, May 14, 2012 at 1:15 PM, Eric Sammer wrote:

> We've seen some odd behavior if the value of HADOOP_HOME isn't an absolute path. I think I already filed a JIRA about this.
> 
> On Mon, May 14, 2012 at 6:59 AM, lexsoto@gmail.com (mailto:lexsoto@gmail.com) <lexsoto@gmail.com (mailto:lexsoto@gmail.com)> wrote:
> > Thanks for the help.  This is working now after setting the
> > HADOOP_HOME variable.
> > Now the problem is testing it.
> > 
> > In a different terminal I am typing:
> > 
> >  export HADOOP_HOME=~/hadoop-0.20.2-cdh3u4
> >  bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F server.log
> > 
> > but I am getting the exception:
> > 
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/google/common/collect/Lists
> >        at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:170)
> >        at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> > Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >        ... 2 more
> > 
> > Any idea?
> > 
> > Thanks
> > 
> > On Fri, May 11, 2012 at 9:25 PM, Mike Percy <mpercy@cloudera.com (mailto:mpercy@cloudera.com)> wrote:
> > > Alright, well cdh-user@ is publicly viewable I believe but here is the most
> > > important bit of what I told Alex:
> > >
> > > You need to set HADOOP_HOME variable or else Flume will not be able to find
> > > the HDFS jars if you have put then in some non-standard place such as your
> > > home directory.
> > >
> > > Also it turns out that he is running Hadoop on Mac which is not something
> > > I'm really familiar with but CDH doesn't target it from a QA perspective.
> > >
> > > HTH,
> > > Mike
> > >
> > > On May 11, 2012, at 2:09 PM, Ralph Goers wrote:
> > >
> > > A word of caution here.  If these issues are also relevant to the Apache
> > > distribution then some form of the discussion should come back here so that
> > > it benefits users of the vanilla distribution.
> > >
> > > Ralph
> > >
> > > On May 11, 2012, at 1:17 PM, Mike Percy wrote:
> > >
> > > bcc: flume-user@incubator.apache.org (mailto:flume-user@incubator.apache.org)
> > >
> > > This issue involves a version of Flume distributed with CDH, so moving to
> > > cdh-user@cloudera.org (mailto:cdh-user@cloudera.org) which is the proper support channel.
> > >
> > > Thanks,
> > > Mike
> > >
> > > On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com (mailto:lexsoto@gmail.com) wrote:
> > >
> > > Hello,
> > >
> > > I am new to Flume. I am trying to use hdfs as a sink. My first
> > > problem is that flume hangs during initialization.
> > > Same problem as:
> > >
> > > https://issues.apache.org/jira/browse/FLUME-1077
> > >
> > > and
> > >
> > > https://issues.apache.org/jira/browse/FLUME-1093
> > >
> > > My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
> > > "flume-ng-1.1.0-cdh3u4"
> > > I noticed that Hadoop and Flume have different versions of Guava.
> > >
> > > If I remove Flume's Guava jar and use Hadoop's:
> > >
> > > <code>
> > > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > > --classpath
> > > ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
> > > </code>
> > >
> > > I get this error:
> > >
> > > <code>
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > com/google/common/util/concurrent/ThreadFactoryBuilder
> > > at
> > > org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
> > > at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
> > > at org.apache.flume.node.Application.run(Application.java:135)
> > > at org.apache.flume.node.Application.main(Application.java:70)
> > > Caused by: java.lang.ClassNotFoundException:
> > > com.google.common.util.concurrent.ThreadFactoryBuilder
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > ... 4 more
> > > </code>
> > >
> > > If I put back Flume's Guava jar:
> > >
> > > <code>
> > > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > > --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
> > > </code>
> > >
> > > I get this error:
> > >
> > > <code>
> > > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
> > > at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
> > > at
> > > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
> > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> > > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
> > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
> > > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
> > > at
> > > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
> > > at
> > > org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
> > > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
> > > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
> > > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
> > > at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
> > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:680)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > > ... 21 more
> > > </code>
> > >
> > > Has anybody been able to successfully run Flume with a HDFS sink?
> > > What's the magic combination?
> > >
> > > --
> > > Alex Soto
> > > lexsoto@gmail.com (mailto:lexsoto@gmail.com)
> > >
> > > -
> > > Amicus Plato, sed magis amica veritas.
> > >
> > >
> > >
> > >
> > 
> > 
> > 
> > --
> > Alex Soto
> > lexsoto@gmail.com (mailto:lexsoto@gmail.com)
> > 
> > -
> > Amicus Plato, sed magis amica veritas.
> 
> 
> 
> -- 
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com (http://www.cloudera.com)

Re: Unable to use HDFS as sink

Posted by Eric Sammer <es...@cloudera.com>.
We've seen some odd behavior if the value of HADOOP_HOME isn't an absolute
path. I think I already filed a JIRA about this.

On Mon, May 14, 2012 at 6:59 AM, lexsoto@gmail.com <le...@gmail.com>wrote:

> Thanks for the help.  This is working now after setting the
> HADOOP_HOME variable.
> Now the problem is testing it.
>
> In a different terminal I am typing:
>
>  export HADOOP_HOME=~/hadoop-0.20.2-cdh3u4
>  bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F server.log
>
> but I am getting the exception:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/collect/Lists
>        at
> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:170)
>        at
> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.collect.Lists
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>         ... 2 more
>
> Any idea?
>
> Thanks
>
> On Fri, May 11, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
> > Alright, well cdh-user@ is publicly viewable I believe but here is the
> most
> > important bit of what I told Alex:
> >
> > You need to set HADOOP_HOME variable or else Flume will not be able to
> find
> > the HDFS jars if you have put then in some non-standard place such as
> your
> > home directory.
> >
> > Also it turns out that he is running Hadoop on Mac which is not something
> > I'm really familiar with but CDH doesn't target it from a QA perspective.
> >
> > HTH,
> > Mike
> >
> > On May 11, 2012, at 2:09 PM, Ralph Goers wrote:
> >
> > A word of caution here.  If these issues are also relevant to the Apache
> > distribution then some form of the discussion should come back here so
> that
> > it benefits users of the vanilla distribution.
> >
> > Ralph
> >
> > On May 11, 2012, at 1:17 PM, Mike Percy wrote:
> >
> > bcc: flume-user@incubator.apache.org
> >
> > This issue involves a version of Flume distributed with CDH, so moving to
> > cdh-user@cloudera.org which is the proper support channel.
> >
> > Thanks,
> > Mike
> >
> > On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:
> >
> > Hello,
> >
> > I am new to Flume. I am trying to use hdfs as a sink. My first
> > problem is that flume hangs during initialization.
> > Same problem as:
> >
> > https://issues.apache.org/jira/browse/FLUME-1077
> >
> > and
> >
> > https://issues.apache.org/jira/browse/FLUME-1093
> >
> > My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
> > "flume-ng-1.1.0-cdh3u4"
> > I noticed that Hadoop and Flume have different versions of Guava.
> >
> > If I remove Flume's Guava jar and use Hadoop's:
> >
> > <code>
> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > --classpath
> >
> ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
> > </code>
> >
> > I get this error:
> >
> > <code>
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/google/common/util/concurrent/ThreadFactoryBuilder
> > at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
> > at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
> > at org.apache.flume.node.Application.run(Application.java:135)
> > at org.apache.flume.node.Application.main(Application.java:70)
> > Caused by: java.lang.ClassNotFoundException:
> > com.google.common.util.concurrent.ThreadFactoryBuilder
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > ... 4 more
> > </code>
> >
> > If I put back Flume's Guava jar:
> >
> > <code>
> > bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> > --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
> > </code>
> >
> > I get this error:
> >
> > <code>
> > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
> > at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> > at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
> > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
> > at
> >
> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
> > at
> >
> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
> > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
> > at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
> > at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
> > at
> org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:680)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > ... 21 more
> > </code>
> >
> > Has anybody been able to successfully run Flume with a HDFS sink?
> > What's the magic combination?
> >
> > --
> > Alex Soto
> > lexsoto@gmail.com
> >
> > -
> > Amicus Plato, sed magis amica veritas.
> >
> >
> >
> >
>
>
>
> --
> Alex Soto
> lexsoto@gmail.com
>
> -
> Amicus Plato, sed magis amica veritas.
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

Re: Unable to use HDFS as sink

Posted by "lexsoto@gmail.com" <le...@gmail.com>.
Thanks for the help.  This is working now after setting the
HADOOP_HOME variable.
Now the problem is testing it.

In a different terminal I am typing:

  export HADOOP_HOME=~/hadoop-0.20.2-cdh3u4
  bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F server.log

but I am getting the exception:

Exception in thread "main" java.lang.NoClassDefFoundError:
com/google/common/collect/Lists
	at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:170)
	at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Lists
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 2 more

Any idea?

Thanks

On Fri, May 11, 2012 at 9:25 PM, Mike Percy <mp...@cloudera.com> wrote:
> Alright, well cdh-user@ is publicly viewable I believe but here is the most
> important bit of what I told Alex:
>
> You need to set HADOOP_HOME variable or else Flume will not be able to find
> the HDFS jars if you have put then in some non-standard place such as your
> home directory.
>
> Also it turns out that he is running Hadoop on Mac which is not something
> I'm really familiar with but CDH doesn't target it from a QA perspective.
>
> HTH,
> Mike
>
> On May 11, 2012, at 2:09 PM, Ralph Goers wrote:
>
> A word of caution here.  If these issues are also relevant to the Apache
> distribution then some form of the discussion should come back here so that
> it benefits users of the vanilla distribution.
>
> Ralph
>
> On May 11, 2012, at 1:17 PM, Mike Percy wrote:
>
> bcc: flume-user@incubator.apache.org
>
> This issue involves a version of Flume distributed with CDH, so moving to
> cdh-user@cloudera.org which is the proper support channel.
>
> Thanks,
> Mike
>
> On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:
>
> Hello,
>
> I am new to Flume. I am trying to use hdfs as a sink. My first
> problem is that flume hangs during initialization.
> Same problem as:
>
> https://issues.apache.org/jira/browse/FLUME-1077
>
> and
>
> https://issues.apache.org/jira/browse/FLUME-1093
>
> My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
> "flume-ng-1.1.0-cdh3u4"
> I noticed that Hadoop and Flume have different versions of Guava.
>
> If I remove Flume's Guava jar and use Hadoop's:
>
> <code>
> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> --classpath
> ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
> </code>
>
> I get this error:
>
> <code>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/util/concurrent/ThreadFactoryBuilder
> at
> org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
> at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
> at org.apache.flume.node.Application.run(Application.java:135)
> at org.apache.flume.node.Application.main(Application.java:70)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.util.concurrent.ThreadFactoryBuilder
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 4 more
> </code>
>
> If I put back Flume's Guava jar:
>
> <code>
> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
> </code>
>
> I get this error:
>
> <code>
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
> at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> </code>
>
> Has anybody been able to successfully run Flume with a HDFS sink?
> What's the magic combination?
>
> --
> Alex Soto
> lexsoto@gmail.com
>
> -
> Amicus Plato, sed magis amica veritas.
>
>
>
>



-- 
Alex Soto
lexsoto@gmail.com

-
Amicus Plato, sed magis amica veritas.

Re: Unable to use HDFS as sink

Posted by Mike Percy <mp...@cloudera.com>.
Alright, well cdh-user@ is publicly viewable I believe but here is the most important bit of what I told Alex:

You need to set HADOOP_HOME variable or else Flume will not be able to find the HDFS jars if you have put then in some non-standard place such as your home directory.

Also it turns out that he is running Hadoop on Mac which is not something I'm really familiar with but CDH doesn't target it from a QA perspective.

HTH,
Mike

On May 11, 2012, at 2:09 PM, Ralph Goers wrote:

> A word of caution here.  If these issues are also relevant to the Apache distribution then some form of the discussion should come back here so that it benefits users of the vanilla distribution.
> 
> Ralph
> 
> On May 11, 2012, at 1:17 PM, Mike Percy wrote:
> 
>> bcc: flume-user@incubator.apache.org
>> 
>> This issue involves a version of Flume distributed with CDH, so moving to cdh-user@cloudera.org which is the proper support channel.
>> 
>> Thanks,
>> Mike
>> On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:
>> 
>>> Hello,
>>> 
>>> I am new to Flume. I am trying to use hdfs as a sink. My first
>>> problem is that flume hangs during initialization.
>>> Same problem as:
>>> 
>>> https://issues.apache.org/jira/browse/FLUME-1077
>>> 
>>> and
>>> 
>>> https://issues.apache.org/jira/browse/FLUME-1093
>>> 
>>> My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
>>> "flume-ng-1.1.0-cdh3u4"
>>> I noticed that Hadoop and Flume have different versions of Guava.
>>> 
>>> If I remove Flume's Guava jar and use Hadoop's:
>>> 
>>> <code>
>>> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>>> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
>>> </code>
>>> 
>>> I get this error:
>>> 
>>> <code>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/common/util/concurrent/ThreadFactoryBuilder
>>> at org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
>>> at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
>>> at org.apache.flume.node.Application.run(Application.java:135)
>>> at org.apache.flume.node.Application.main(Application.java:70)
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.google.common.util.concurrent.ThreadFactoryBuilder
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> ... 4 more
>>> </code>
>>> 
>>> If I put back Flume's Guava jar:
>>> 
>>> <code>
>>> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>>> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
>>> </code>
>>> 
>>> I get this error:
>>> 
>>> <code>
>>> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
>>> java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
>>> at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
>>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
>>> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
>>> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>>> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
>>> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
>>> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
>>> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
>>> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
>>> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:680)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>> ... 21 more
>>> </code>
>>> 
>>> Has anybody been able to successfully run Flume with a HDFS sink?
>>> What's the magic combination?
>>> 
>>> --
>>> Alex Soto
>>> lexsoto@gmail.com
>>> 
>>> -
>>> Amicus Plato, sed magis amica veritas.
>> 
>

Re: Unable to use HDFS as sink

Posted by Ralph Goers <ra...@dslextreme.com>.
A word of caution here.  If these issues are also relevant to the Apache distribution then some form of the discussion should come back here so that it benefits users of the vanilla distribution.

Ralph

On May 11, 2012, at 1:17 PM, Mike Percy wrote:

> bcc: flume-user@incubator.apache.org
> 
> This issue involves a version of Flume distributed with CDH, so moving to cdh-user@cloudera.org which is the proper support channel.
> 
> Thanks,
> Mike
> On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:
> 
>> Hello,
>> 
>> I am new to Flume. I am trying to use hdfs as a sink. My first
>> problem is that flume hangs during initialization.
>> Same problem as:
>> 
>> https://issues.apache.org/jira/browse/FLUME-1077
>> 
>> and
>> 
>> https://issues.apache.org/jira/browse/FLUME-1093
>> 
>> My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
>> "flume-ng-1.1.0-cdh3u4"
>> I noticed that Hadoop and Flume have different versions of Guava.
>> 
>> If I remove Flume's Guava jar and use Hadoop's:
>> 
>> <code>
>> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
>> </code>
>> 
>> I get this error:
>> 
>> <code>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> com/google/common/util/concurrent/ThreadFactoryBuilder
>> at org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
>> at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
>> at org.apache.flume.node.Application.run(Application.java:135)
>> at org.apache.flume.node.Application.main(Application.java:70)
>> Caused by: java.lang.ClassNotFoundException:
>> com.google.common.util.concurrent.ThreadFactoryBuilder
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> ... 4 more
>> </code>
>> 
>> If I put back Flume's Guava jar:
>> 
>> <code>
>> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
>> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
>> </code>
>> 
>> I get this error:
>> 
>> <code>
>> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
>> at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
>> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
>> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
>> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
>> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
>> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
>> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
>> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
>> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> ... 21 more
>> </code>
>> 
>> Has anybody been able to successfully run Flume with a HDFS sink?
>> What's the magic combination?
>> 
>> --
>> Alex Soto
>> lexsoto@gmail.com
>> 
>> -
>> Amicus Plato, sed magis amica veritas.
>

Re: Unable to use HDFS as sink

Posted by Mike Percy <mp...@cloudera.com>.
bcc: flume-user@incubator.apache.org 

This issue involves a version of Flume distributed with CDH, so moving to cdh-user@cloudera.org which is the proper support channel.

Thanks,
Mike


On Friday, May 11, 2012 at 1:02 PM, lexsoto@gmail.com wrote:

> Hello,
> 
> I am new to Flume. I am trying to use hdfs as a sink. My first
> problem is that flume hangs during initialization.
> Same problem as:
> 
> https://issues.apache.org/jira/browse/FLUME-1077
> 
> and
> 
> https://issues.apache.org/jira/browse/FLUME-1093
> 
> My version of Hadoop is Cloudera's "cdh3u4", and Flume version is
> "flume-ng-1.1.0-cdh3u4"
> I noticed that Hadoop and Flume have different versions of Guava.
> 
> If I remove Flume's Guava jar and use Hadoop's:
> 
> <code>
> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar:~/hadoop-0.20.2-cdh3u4/lib/guava-r09-jarjar.jar
> </code>
> 
> I get this error:
> 
> <code>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/util/concurrent/ThreadFactoryBuilder
> at org.apache.flume.lifecycle.LifecycleSupervisor.<init>(LifecycleSupervisor.java:54)
> at org.apache.flume.node.FlumeNode.<init>(FlumeNode.java:42)
> at org.apache.flume.node.Application.run(Application.java:135)
> at org.apache.flume.node.Application.main(Application.java:70)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.util.concurrent.ThreadFactoryBuilder
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 4 more
> </code>
> 
> If I put back Flume's Guava jar:
> 
> <code>
> bin/flume-ng agent --conf ./conf/ -f ~/tail2hdfs.conf -n agent1
> --classpath ~/hadoop-0.20.2-cdh3u4/hadoop-core-0.20.2-cdh3u4.jar
> </code>
> 
> I get this error:
> 
> <code>
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/thirdparty/guava/common/collect/LinkedListMultimap
> at org.apache.hadoop.hdfs.SocketCache.<init>(SocketCache.java:48)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:249)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:217)
> at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1563)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1597)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1579)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:183)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:57)
> at org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:49)
> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:95)
> at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:117)
> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:361)
> at org.apache.flume.sink.hdfs.HDFSEventSink$1.call(HDFSEventSink.java:357)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.thirdparty.guava.common.collect.LinkedListMultimap
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> </code>
> 
> Has anybody been able to successfully run Flume with a HDFS sink?
> What's the magic combination?
> 
> -- 
> Alex Soto
> lexsoto@gmail.com (mailto:lexsoto@gmail.com)
> 
> -
> Amicus Plato, sed magis amica veritas.
> 
>