You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Andrew McNabb <am...@mcnabbs.org> on 2006/10/17 22:12:06 UTC
HadoopStreaming
HadoopStreaming looks really cool, and I'm trying it for the first time.
I'm obviously doing something wrong, but I have no clue what.
I made a goofy little wordcount mapper and reducer in Python, and I'm
running HadoopStreaming with the following alias:
alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar /home/amcnabb/hadoop/build/hadoop-streaming.jar'
Here is the job I ran, after doing "hadoop dfs -put kjv kjv". From the output,
it is clear that: "Job not Successful!" However, I have no idea of
what's causing the problem. Am I messing up something obvious? Where
should I look to see what's really happening? Thanks.
amcnabb@prodigy:~/svn/mrpso/python% hadoop-streaming -mapper "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py" -reducer "/usr/bin/python /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
06/10/17 14:09:11 INFO ipc.Client: org.apache.hadoop.io.ObjectWritable ConnectionCuller maxidletime=1000ms: starting
packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-amcnabb/mapred/local]
06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013
06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run:
06/10/17 14:09:11 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=prodigy:50006 -kill job_0013
06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
06/10/17 14:09:12 INFO streaming.StreamJob: map 100% reduce 100%
06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run:
06/10/17 14:09:12 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=prodigy:50006 -kill job_0013
06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
06/10/17 14:09:12 INFO streaming.StreamJob: killJob...
Exception in thread "main" java.io.IOException: Job not Successful!
at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:558)
at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63)
at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.util.RunJar.main(RunJar.java:137)
amcnabb@prodigy:~/svn/mrpso/python%
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
RE: HadoopStreaming
Posted by Yoram Arnon <ya...@yahoo-inc.com>.
You may specify multiple -input "<wildcard>" statements.
Take care to quote the wildcard part to prevent your local shell from
parsing it.
You may specify any property you like using -jobconf. Common uses are
mapred.map.tasks and mapred.reduce.tasks to override the defaults for number
of maps and reduces, but anything is allowed.
Another useful argument is -cmdenv <key>=<value> to override environment
variables. A common use is to ship a dynamic library and set LD_LIBRARY_PATH
to '.', but override any variable your program expects.
Yoram
> -----Original Message-----
> From: Hairong Kuang [mailto:hairong@yahoo-inc.com]
> Sent: Tuesday, October 17, 2006 7:11 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: HadoopStreaming
>
> Hadoop streaming assumes that inputs are files. If kjv is a
> directory, you
> may use the option "-input kjv/*".
>
> Hairong
>
> -----Original Message-----
> From: Andrew McNabb [mailto:amcnabb@mcnabbs.org]
> Sent: Tuesday, October 17, 2006 6:22 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: HadoopStreaming
>
> On Tue, Oct 17, 2006 at 03:52:59PM -0700, Yoram Arnon wrote:
> > Try changing your command to read
> >
> > hadoop-streaming \
> > -mapper "/usr/bin/python mapper.py" \
> > -file "/home/amcnabb/svn/mrpso/python/mapper.py" \ -reducer
> > "/usr/bin/python reducer.py" \ -file
> > "/home/amcnabb/svn/mrpso/python/reducer.py" \ -input kjv \ -output
> > kjvout
>
> I'll try this first thing in the morning.
>
> > I assume kjv is a file and kjvout is a directory - they should be.
>
> Actually, I was doing it the same way as other Hadoop stuff I've done:
> kjv is a directory in DFS. Does HadoopStreaming do it in a
> different way
> from most other Hadoop stuff?
>
> In any case, how do I make it take a directory as input if
> that's what I
> need?
>
> > I also assume /usr/bin/python is the path to python *on the cluster
> > machines*. Otherwise, you can do -mapper "python mapper.py" -file
> > /usr/bin/python -file /home/amcnabb/svn/mrpso/python/mapper.py
>
> > I recommend adding -jobconf mapred.job.name="kjv", to make the
> > jobtracker history more readable.
> >
>
> I didn't know about that option. I'll do that.
>
> Thanks for all of the tips.
>
> --
> Andrew McNabb
> http://www.mcnabbs.org/andrew/
> PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
>
>
RE: HadoopStreaming
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
Hadoop streaming assumes that inputs are files. If kjv is a directory, you
may use the option "-input kjv/*".
Hairong
-----Original Message-----
From: Andrew McNabb [mailto:amcnabb@mcnabbs.org]
Sent: Tuesday, October 17, 2006 6:22 PM
To: hadoop-user@lucene.apache.org
Subject: Re: HadoopStreaming
On Tue, Oct 17, 2006 at 03:52:59PM -0700, Yoram Arnon wrote:
> Try changing your command to read
>
> hadoop-streaming \
> -mapper "/usr/bin/python mapper.py" \
> -file "/home/amcnabb/svn/mrpso/python/mapper.py" \ -reducer
> "/usr/bin/python reducer.py" \ -file
> "/home/amcnabb/svn/mrpso/python/reducer.py" \ -input kjv \ -output
> kjvout
I'll try this first thing in the morning.
> I assume kjv is a file and kjvout is a directory - they should be.
Actually, I was doing it the same way as other Hadoop stuff I've done:
kjv is a directory in DFS. Does HadoopStreaming do it in a different way
from most other Hadoop stuff?
In any case, how do I make it take a directory as input if that's what I
need?
> I also assume /usr/bin/python is the path to python *on the cluster
> machines*. Otherwise, you can do -mapper "python mapper.py" -file
> /usr/bin/python -file /home/amcnabb/svn/mrpso/python/mapper.py
> I recommend adding -jobconf mapred.job.name="kjv", to make the
> jobtracker history more readable.
>
I didn't know about that option. I'll do that.
Thanks for all of the tips.
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
Re: HadoopStreaming
Posted by Andrew McNabb <am...@mcnabbs.org>.
On Tue, Oct 17, 2006 at 03:52:59PM -0700, Yoram Arnon wrote:
> Try changing your command to read
>
> hadoop-streaming \
> -mapper "/usr/bin/python mapper.py" \
> -file "/home/amcnabb/svn/mrpso/python/mapper.py" \
> -reducer "/usr/bin/python reducer.py" \
> -file "/home/amcnabb/svn/mrpso/python/reducer.py" \
> -input kjv \
> -output kjvout
I'll try this first thing in the morning.
> I assume kjv is a file and kjvout is a directory - they should be.
Actually, I was doing it the same way as other Hadoop stuff I've done:
kjv is a directory in DFS. Does HadoopStreaming do it in a different
way from most other Hadoop stuff?
In any case, how do I make it take a directory as input if that's what I
need?
> I also assume /usr/bin/python is the path to python *on the cluster
> machines*. Otherwise, you can do
> -mapper "python mapper.py" -file /usr/bin/python -file
> /home/amcnabb/svn/mrpso/python/mapper.py
> I recommend adding -jobconf mapred.job.name="kjv", to make the jobtracker
> history more readable.
>
I didn't know about that option. I'll do that.
Thanks for all of the tips.
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
RE: HadoopStreaming
Posted by Yoram Arnon <ya...@yahoo-inc.com>.
Try changing your command to read
hadoop-streaming \
-mapper "/usr/bin/python mapper.py" \
-file "/home/amcnabb/svn/mrpso/python/mapper.py" \
-reducer "/usr/bin/python reducer.py" \
-file "/home/amcnabb/svn/mrpso/python/reducer.py" \
-input kjv \
-output kjvout
I assume kjv is a file and kjvout is a directory - they should be.
I also assume /usr/bin/python is the path to python *on the cluster
machines*. Otherwise, you can do
-mapper "python mapper.py" -file /usr/bin/python -file
/home/amcnabb/svn/mrpso/python/mapper.py
I recommend adding -jobconf mapred.job.name="kjv", to make the jobtracker
history more readable.
Yoram
> -----Original Message-----
> From: Andrew McNabb [mailto:amcnabb@mcnabbs.org]
> Sent: Tuesday, October 17, 2006 1:12 PM
> To: hadoop-user@lucene.apache.org
> Subject: HadoopStreaming
>
> HadoopStreaming looks really cool, and I'm trying it for the
> first time.
> I'm obviously doing something wrong, but I have no clue what.
>
> I made a goofy little wordcount mapper and reducer in Python, and I'm
> running HadoopStreaming with the following alias:
>
> alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar
> /home/amcnabb/hadoop/build/hadoop-streaming.jar'
>
> Here is the job I ran, after doing "hadoop dfs -put kjv kjv".
> From the output,
> it is clear that: "Job not Successful!" However, I have no idea of
> what's causing the problem. Am I messing up something obvious? Where
> should I look to see what's really happening? Thanks.
>
> amcnabb@prodigy:~/svn/mrpso/python% hadoop-streaming -mapper
> "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py"
> -reducer "/usr/bin/python
> /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout
> 06/10/17 14:09:11 INFO conf.Configuration: parsing
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing
> file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
> 06/10/17 14:09:11 INFO ipc.Client:
> org.apache.hadoop.io.ObjectWritable ConnectionCuller
> maxidletime=1000ms: starting
> packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar
> 06/10/17 14:09:11 INFO conf.Configuration: parsing
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml
> 06/10/17 14:09:11 INFO conf.Configuration: parsing
> file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml
> 06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs():
> [/tmp/hadoop-amcnabb/mapred/local]
> 06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013
> 06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run:
> 06/10/17 14:09:11 INFO streaming.StreamJob:
> /home/amcnabb/hadoop/bin/../bin/hadoop job
> -Dmapred.job.tracker=prodigy:50006 -kill job_0013
> 06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL:
> http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob: map 100% reduce 100%
> 06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run:
> 06/10/17 14:09:12 INFO streaming.StreamJob:
> /home/amcnabb/hadoop/bin/../bin/hadoop job
> -Dmapred.job.tracker=prodigy:50006 -kill job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL:
> http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013
> 06/10/17 14:09:12 INFO streaming.StreamJob: killJob...
> Exception in thread "main" java.io.IOException: Job not Successful!
> at
> org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(Stre
> amJob.java:558)
> at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63)
> at
> org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreami
> ng.java:29)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess
> orImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> odAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:137)
> amcnabb@prodigy:~/svn/mrpso/python%
>
> --
> Andrew McNabb
> http://www.mcnabbs.org/andrew/
> PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
>