You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jim the Standing Bear <st...@gmail.com> on 2007/11/01 20:35:33 UTC
clarification on running a job locally
Hello,
I am in need of some clarifications on how to run a hadoop job locally.
The cluster was originally set up to have two nodes, where one of them
also acts as the master node and job tracker.
According to the wiki, I can run a job locally by altering
"mapred.job.tracker" and "fs.default.name" properties to "local" in
hadoop-site.xml. But when I start the server, it stack dumped:
localhost: starting secondarynamenode, logging to /home/blahblahblah
localhost: Exception in thread "main" java.lang.RuntimeException: Not
a host:port pair: local
Apparently it didn't like the value "local"?
Also, the wiki noted that all these XML configuration files should be
included somewhere in the class path to the job, does it mean I need
to include the XMLs as I do jars?
Thank
-- Jim
Re: clarification on running a job locally
Posted by Chris K Wensel <ch...@wensel.net>.
hmm.. maybe we are talking about different things. i'll politely pass
this one on.
On Nov 1, 2007, at 1:05 PM, Jim the Standing Bear wrote:
> Hi Chris,
>
> I meant "Local Runner".
>
> From the sounds of your email, it seems that modifying the two
> fore-mentioned properties is not enough to get a cluster node to run
> as a local runner?
>
> Also, what about "including .xml to the class path"? Do I include
> them like jar files? My ant script has included all the external jars
> in the lib/ directory - do I need to do the same for the xml files?
>
> thanks
>
> -- Jim
>
> On 11/1/07, Chris K Wensel <ch...@wensel.net> wrote:
>> Hey Jim
>>
>> what do you mean by local?
>>
>> This wiki page defines two 'local' modes.
>> http://wiki.apache.org/lucene-hadoop/QuickStart
>>
>> One with a local cluster, and one with all jobs running standalone.
>>
>> The conf dir that ships with Hadoop is setup for standalone.
>>
>> I find it useful to have a copy of 'conf' named 'conf-localhost' that
>> has the local cluster version of 'hadoop-site.xml'. then use
>>
>>> ./bin/hadoop --config ./conf-localhost ....
>>
>> to start her up.. (after following the other wiki steps)
>>
>> cheers
>> ckw
>>
>> On Nov 1, 2007, at 12:35 PM, Jim the Standing Bear wrote:
>>
>>> Hello,
>>>
>>> I am in need of some clarifications on how to run a hadoop job
>>> locally.
>>>
>>> The cluster was originally set up to have two nodes, where one of
>>> them
>>> also acts as the master node and job tracker.
>>>
>>> According to the wiki, I can run a job locally by altering
>>> "mapred.job.tracker" and "fs.default.name" properties to "local" in
>>> hadoop-site.xml. But when I start the server, it stack dumped:
>>>
>>> localhost: starting secondarynamenode, logging to /home/blahblahblah
>>> localhost: Exception in thread "main" java.lang.RuntimeException:
>>> Not
>>> a host:port pair: local
>>>
>>> Apparently it didn't like the value "local"?
>>>
>>> Also, the wiki noted that all these XML configuration files
>>> should be
>>> included somewhere in the class path to the job, does it mean I need
>>> to include the XMLs as I do jars?
>>>
>>> Thank
>>>
>>> -- Jim
>>
>> Chris K Wensel
>> chris@wensel.net
>> http://chris.wensel.net/
>>
>>
>>
>>
>
>
> --
> --------------------------------------
> Standing Bear Has Spoken
> --------------------------------------
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
Re: clarification on running a job locally
Posted by Jim the Standing Bear <st...@gmail.com>.
Hi Chris,
I meant "Local Runner".
>From the sounds of your email, it seems that modifying the two
fore-mentioned properties is not enough to get a cluster node to run
as a local runner?
Also, what about "including .xml to the class path"? Do I include
them like jar files? My ant script has included all the external jars
in the lib/ directory - do I need to do the same for the xml files?
thanks
-- Jim
On 11/1/07, Chris K Wensel <ch...@wensel.net> wrote:
> Hey Jim
>
> what do you mean by local?
>
> This wiki page defines two 'local' modes.
> http://wiki.apache.org/lucene-hadoop/QuickStart
>
> One with a local cluster, and one with all jobs running standalone.
>
> The conf dir that ships with Hadoop is setup for standalone.
>
> I find it useful to have a copy of 'conf' named 'conf-localhost' that
> has the local cluster version of 'hadoop-site.xml'. then use
>
> > ./bin/hadoop --config ./conf-localhost ....
>
> to start her up.. (after following the other wiki steps)
>
> cheers
> ckw
>
> On Nov 1, 2007, at 12:35 PM, Jim the Standing Bear wrote:
>
> > Hello,
> >
> > I am in need of some clarifications on how to run a hadoop job
> > locally.
> >
> > The cluster was originally set up to have two nodes, where one of them
> > also acts as the master node and job tracker.
> >
> > According to the wiki, I can run a job locally by altering
> > "mapred.job.tracker" and "fs.default.name" properties to "local" in
> > hadoop-site.xml. But when I start the server, it stack dumped:
> >
> > localhost: starting secondarynamenode, logging to /home/blahblahblah
> > localhost: Exception in thread "main" java.lang.RuntimeException: Not
> > a host:port pair: local
> >
> > Apparently it didn't like the value "local"?
> >
> > Also, the wiki noted that all these XML configuration files should be
> > included somewhere in the class path to the job, does it mean I need
> > to include the XMLs as I do jars?
> >
> > Thank
> >
> > -- Jim
>
> Chris K Wensel
> chris@wensel.net
> http://chris.wensel.net/
>
>
>
>
--
--------------------------------------
Standing Bear Has Spoken
--------------------------------------
Re: clarification on running a job locally
Posted by Chris K Wensel <ch...@wensel.net>.
Hey Jim
what do you mean by local?
This wiki page defines two 'local' modes.
http://wiki.apache.org/lucene-hadoop/QuickStart
One with a local cluster, and one with all jobs running standalone.
The conf dir that ships with Hadoop is setup for standalone.
I find it useful to have a copy of 'conf' named 'conf-localhost' that
has the local cluster version of 'hadoop-site.xml'. then use
> ./bin/hadoop --config ./conf-localhost ....
to start her up.. (after following the other wiki steps)
cheers
ckw
On Nov 1, 2007, at 12:35 PM, Jim the Standing Bear wrote:
> Hello,
>
> I am in need of some clarifications on how to run a hadoop job
> locally.
>
> The cluster was originally set up to have two nodes, where one of them
> also acts as the master node and job tracker.
>
> According to the wiki, I can run a job locally by altering
> "mapred.job.tracker" and "fs.default.name" properties to "local" in
> hadoop-site.xml. But when I start the server, it stack dumped:
>
> localhost: starting secondarynamenode, logging to /home/blahblahblah
> localhost: Exception in thread "main" java.lang.RuntimeException: Not
> a host:port pair: local
>
> Apparently it didn't like the value "local"?
>
> Also, the wiki noted that all these XML configuration files should be
> included somewhere in the class path to the job, does it mean I need
> to include the XMLs as I do jars?
>
> Thank
>
> -- Jim
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
Re: clarification on running a job locally
Posted by Jim the Standing Bear <st...@gmail.com>.
Thanks Ted. just as I thought.
On 11/1/07, Ted Dunning <td...@veoh.com> wrote:
>
> It is definitely easier to build a jar and use the hadoop script. You can
> do it yourself, though. Just duplicate the line in bin/hadoop that runs
> java and prefix it with "echo" to see what is happening.
>
>
> On 11/1/07 1:37 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
>
> > Hi Ted,
> >
> > It is funny how I am having so much difficulties with hadoop... since
> > i am on this subject, let me ask another stupid question:
> >
> > In order to run a hadoop job, I must zip it into a jar, and then use
> > $HADOOP_HOME/bin/hadoop to launch it, correct? I cannot simply use
> > "java" to directly launch it, right?
> >
> > best wishes
> >
> > Yiming
> >
> >
> > On 11/1/07, Ted Dunning <td...@veoh.com> wrote:
> >>
> >> When running a job standalone (typically for debugging), it suffices for me
> >> to simply add -Dmapred.job.tracker=local to the program command line.
> >>
> >> If you simply want to run the program on a single node, then you can just
> >> stop the other node. Using local storage with distributed programs is not
> >> recommended.
> >>
> >>
> >> On 11/1/07 12:35 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> I am in need of some clarifications on how to run a hadoop job locally.
> >>>
> >>> The cluster was originally set up to have two nodes, where one of them
> >>> also acts as the master node and job tracker.
> >>>
> >>> According to the wiki, I can run a job locally by altering
> >>> "mapred.job.tracker" and "fs.default.name" properties to "local" in
> >>> hadoop-site.xml. But when I start the server, it stack dumped:
> >>>
> >>> localhost: starting secondarynamenode, logging to /home/blahblahblah
> >>> localhost: Exception in thread "main" java.lang.RuntimeException: Not
> >>> a host:port pair: local
> >>>
> >>> Apparently it didn't like the value "local"?
> >>>
> >>> Also, the wiki noted that all these XML configuration files should be
> >>> included somewhere in the class path to the job, does it mean I need
> >>> to include the XMLs as I do jars?
> >>>
> >>> Thank
> >>>
> >>> -- Jim
> >>
> >>
> >
>
>
--
--------------------------------------
Standing Bear Has Spoken
--------------------------------------
Re: clarification on running a job locally
Posted by Ted Dunning <td...@veoh.com>.
It is definitely easier to build a jar and use the hadoop script. You can
do it yourself, though. Just duplicate the line in bin/hadoop that runs
java and prefix it with "echo" to see what is happening.
On 11/1/07 1:37 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
> Hi Ted,
>
> It is funny how I am having so much difficulties with hadoop... since
> i am on this subject, let me ask another stupid question:
>
> In order to run a hadoop job, I must zip it into a jar, and then use
> $HADOOP_HOME/bin/hadoop to launch it, correct? I cannot simply use
> "java" to directly launch it, right?
>
> best wishes
>
> Yiming
>
>
> On 11/1/07, Ted Dunning <td...@veoh.com> wrote:
>>
>> When running a job standalone (typically for debugging), it suffices for me
>> to simply add -Dmapred.job.tracker=local to the program command line.
>>
>> If you simply want to run the program on a single node, then you can just
>> stop the other node. Using local storage with distributed programs is not
>> recommended.
>>
>>
>> On 11/1/07 12:35 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am in need of some clarifications on how to run a hadoop job locally.
>>>
>>> The cluster was originally set up to have two nodes, where one of them
>>> also acts as the master node and job tracker.
>>>
>>> According to the wiki, I can run a job locally by altering
>>> "mapred.job.tracker" and "fs.default.name" properties to "local" in
>>> hadoop-site.xml. But when I start the server, it stack dumped:
>>>
>>> localhost: starting secondarynamenode, logging to /home/blahblahblah
>>> localhost: Exception in thread "main" java.lang.RuntimeException: Not
>>> a host:port pair: local
>>>
>>> Apparently it didn't like the value "local"?
>>>
>>> Also, the wiki noted that all these XML configuration files should be
>>> included somewhere in the class path to the job, does it mean I need
>>> to include the XMLs as I do jars?
>>>
>>> Thank
>>>
>>> -- Jim
>>
>>
>
Re: clarification on running a job locally
Posted by Jim the Standing Bear <st...@gmail.com>.
Hi Ted,
It is funny how I am having so much difficulties with hadoop... since
i am on this subject, let me ask another stupid question:
In order to run a hadoop job, I must zip it into a jar, and then use
$HADOOP_HOME/bin/hadoop to launch it, correct? I cannot simply use
"java" to directly launch it, right?
best wishes
Yiming
On 11/1/07, Ted Dunning <td...@veoh.com> wrote:
>
> When running a job standalone (typically for debugging), it suffices for me
> to simply add -Dmapred.job.tracker=local to the program command line.
>
> If you simply want to run the program on a single node, then you can just
> stop the other node. Using local storage with distributed programs is not
> recommended.
>
>
> On 11/1/07 12:35 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
>
> > Hello,
> >
> > I am in need of some clarifications on how to run a hadoop job locally.
> >
> > The cluster was originally set up to have two nodes, where one of them
> > also acts as the master node and job tracker.
> >
> > According to the wiki, I can run a job locally by altering
> > "mapred.job.tracker" and "fs.default.name" properties to "local" in
> > hadoop-site.xml. But when I start the server, it stack dumped:
> >
> > localhost: starting secondarynamenode, logging to /home/blahblahblah
> > localhost: Exception in thread "main" java.lang.RuntimeException: Not
> > a host:port pair: local
> >
> > Apparently it didn't like the value "local"?
> >
> > Also, the wiki noted that all these XML configuration files should be
> > included somewhere in the class path to the job, does it mean I need
> > to include the XMLs as I do jars?
> >
> > Thank
> >
> > -- Jim
>
>
--
--------------------------------------
Standing Bear Has Spoken
--------------------------------------
Re: clarification on running a job locally
Posted by Ted Dunning <td...@veoh.com>.
When running a job standalone (typically for debugging), it suffices for me
to simply add -Dmapred.job.tracker=local to the program command line.
If you simply want to run the program on a single node, then you can just
stop the other node. Using local storage with distributed programs is not
recommended.
On 11/1/07 12:35 PM, "Jim the Standing Bear" <st...@gmail.com> wrote:
> Hello,
>
> I am in need of some clarifications on how to run a hadoop job locally.
>
> The cluster was originally set up to have two nodes, where one of them
> also acts as the master node and job tracker.
>
> According to the wiki, I can run a job locally by altering
> "mapred.job.tracker" and "fs.default.name" properties to "local" in
> hadoop-site.xml. But when I start the server, it stack dumped:
>
> localhost: starting secondarynamenode, logging to /home/blahblahblah
> localhost: Exception in thread "main" java.lang.RuntimeException: Not
> a host:port pair: local
>
> Apparently it didn't like the value "local"?
>
> Also, the wiki noted that all these XML configuration files should be
> included somewhere in the class path to the job, does it mean I need
> to include the XMLs as I do jars?
>
> Thank
>
> -- Jim