You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ayon Sinha <ay...@yahoo.com> on 2011/06/15 05:31:30 UTC
problem with streaming and libjars
None of my options are working. Any help is appreciated.
[ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
-outputformat : class not found : MultiFileOutput
Streaming Command Failed!
[ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
-outputformat : class not found : MultiFileOutput
Streaming Command Failed!
[ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-archives /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
-outputformat : class not found : MultiFileOutput
Streaming Command Failed!
[ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
-outputformat : class not found : MultiFileOutput
Streaming Command Failed!
[ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -files /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
-outputformat : class not found : MultiFileOutput
Streaming Command Failed!
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.
Re: problem with streaming and libjars
Posted by Ayon Sinha <ay...@yahoo.com>.
Thanks Joe. You are right about the package name. Initially I did not have the class in a package. What made it to work is putting the class in a package and not the default package. For some reason it does not like default packaged classes.
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.
________________________________
From: Joey Echeverria <jo...@cloudera.com>
To: common-user@hadoop.apache.org; Ayon Sinha <ay...@yahoo.com>
Sent: Thursday, June 16, 2011 6:06 AM
Subject: Re: problem with streaming and libjars
I would try the following:
hadoop -libjars /home/ayon/jars/MultiOutput.jar jar
/usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar
-libjars /home/ayon/jars/MultiOutput.jar -input
/user/ayon/streaming_test_input -output
/user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat
-outputformat MultiFileOutput
Another thing to note, I'm pretty sure you need to specify the full
package with the name of the output format class (e.g.
my.package.MultiFileOutput).
-Joey
On Tue, Jun 14, 2011 at 11:31 PM, Ayon Sinha <ay...@yahoo.com> wrote:
> None of my options are working. Any help is appreciated.
>
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-archives /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -files /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
>
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: problem with streaming and libjars
Posted by Joey Echeverria <jo...@cloudera.com>.
I would try the following:
hadoop -libjars /home/ayon/jars/MultiOutput.jar jar
/usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar
-libjars /home/ayon/jars/MultiOutput.jar -input
/user/ayon/streaming_test_input -output
/user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat
-outputformat MultiFileOutput
Another thing to note, I'm pretty sure you need to specify the full
package with the name of the output format class (e.g.
my.package.MultiFileOutput).
-Joey
On Tue, Jun 14, 2011 at 11:31 PM, Ayon Sinha <ay...@yahoo.com> wrote:
> None of my options are working. Any help is appreciated.
>
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-archives /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar-libjars /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
> [ayon@dwdb002 ~]$ hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2-cdh3u0.jar -libjars /home/ayon/jars/MultiOutput.jar -files /home/ayon/jars/MultiOutput.jar -input /user/ayon/streaming_test_input -output /user/ayon/streaming_test_output -mapper /bin/cat -reducer /bin/cat -outputformat MultiFileOutput -file /home/ayon/jars/MultiOutput.jar
> -outputformat : class not found : MultiFileOutput
> Streaming Command Failed!
>
>
> -Ayon
> See My Photos on Flickr
> Also check out my Blog for answers to commonly asked questions.
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434