You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Meng Mao <me...@gmail.com> on 2008/08/04 20:10:25 UTC

having different HADOOP_HOME for master and slaves?

I'm trying to set up 2 Hadoop installations on my master node, one of which
will have permissions that allow more users to run Hadoop.
But I don't really need anything different on the datanodes, so I'd like to
keep those as-is. With that switch, the HADOOP_HOME on the master will be
different from that on the datanodes.

After shutting down the old hadoop, I tried to start-all the new one, and
encountered this:
$ bin/stop-all.sh
no jobtracker to stop
node2: bash: line 0: cd: /new/dir/hadoop/bin/..: No such file or directory
node2: bash: /new/dir/hadoop/bin/hadoop-daemon.sh: No such file or directory

I consulted the documentation at:
http://hadoop.apache.org/core/docs/current/cluster_setup.html#Installation
which only has 2 bits of info on this --
1) "The root of the distribution is referred to as HADOOP_HOME. All machines
in the cluster usually have the same HADOOP_HOME path."
and
2) "Once all the necessary configuration is complete, distribute the files
to the HADOOP_CONF_DIR directory on all the machines, typically
${HADOOP_HOME}/conf."

So I forgot to do anything about the second instruction. After doing so, I
got:
$ bin/stop-all.sh
no jobtracker to stop
node2: bash: /new/dir/hadoop/bin/hadoop-daemon.sh: No such file or directory

Ok, it found the config dir, but now it expects the binary to be located at
the same HADOOP_HOME that the master uses?

I suppose I could, for each datanode, symlink things to point to the actual
Hadoop installation. But really, I would like the setup that is hinted as
possible by statement 1). Is there a way I could do it, or should that bit
of documentation read, "All machines in the cluster _must_ have the same
HADOOP_HOME?"

Thanks!

Re: having different HADOOP_HOME for master and slaves?

Posted by Meng Mao <me...@gmail.com>.
Is there any way for me to log and find out why the NameNode process is not
launching on the master?

On Mon, Aug 4, 2008 at 8:19 PM, Meng Mao <me...@gmail.com> wrote:

> assumption -- if I run stop-all.sh _successfully_ on a Hadoop deployment
> (which means every node in the grid is using the same path to Hadoop), then
> that Hadoop installation becomes invisible, and then any other Hadoop
> deployment could start up and take its place on the grid. Let me know if
> this assumption is wrong.
>
> I was having a lot of grief trying to do a parallel, better permissioned
> Hadoop install the easy way, so I just went ahead and make copies on each
> node into the /new/dir location, and pointed hdfs.tmp.dir appropriately.
>
> So in a normal start-all.sh sequence, we have the following processes
> spawned:
> - master has NameNode, 2ndyNameNode, and JobTracker
> - worker has DataNode and TaskTracker
>
> After I powered down the normal Hadoop installation. I tried to
> start-all.sh mine. Again, everything with this Hadoop should point its home
> to /new/dir/hadoop, unless there's some deep hidden param I didn't know
> about. The processes I got were only
> - master: 2ndyNameNode, JobTracker
> - worker: TaskTracker
>
> Another hint is the error that calling the hadoop shell gives:
> $ bin/hadoop dfs -ls /
> 08/08/04 19:25:32 INFO ipc.Client: Retrying connect to server:
> master/ip:50001. Already tried 1 time(s).
> 08/08/04 19:25:33 INFO ipc.Client: Retrying connect to server: master
> /ip:50001. Already tried 2 time(s).
> 08/08/04 19:25:34 INFO ipc.Client: Retrying connect to server:
> master/ip:50001. Already tried 3 time(s).
>
> I can't for the life of me reason why the others are missing.
>
> On Mon, Aug 4, 2008 at 4:17 PM, Meng Mao <me...@gmail.com> wrote:
>
>> I see. I think I could also modify the hadoop-env.sh in the new conf/
>> folders per datanode to point
>> to the right place for HADOOP_HOME.
>>
>>
>> On Mon, Aug 4, 2008 at 3:21 PM, Allen Wittenauer <aw...@yahoo-inc.com>wrote:
>>
>>>
>>>
>>>
>>> On 8/4/08 11:10 AM, "Meng Mao" <me...@gmail.com> wrote:
>>> > I suppose I could, for each datanode, symlink things to point to the
>>> actual
>>> > Hadoop installation. But really, I would like the setup that is hinted
>>> as
>>> > possible by statement 1). Is there a way I could do it, or should that
>>> bit
>>> > of documentation read, "All machines in the cluster _must_ have the
>>> same
>>> > HADOOP_HOME?"
>>>
>>>     If you run the -all scripts, they assume the location is the same.
>>> AFAIK, there is nothing preventing you from building your own -all
>>> scripts
>>> that point to the different location to start/stop the data nodes.
>>>
>>>
>>>
>>
>>
>> --
>> hustlin, hustlin, everyday I'm hustlin
>>
>
>
>
> --
> hustlin, hustlin, everyday I'm hustlin
>



-- 
hustlin, hustlin, everyday I'm hustlin

Re: having different HADOOP_HOME for master and slaves?

Posted by Meng Mao <me...@gmail.com>.
assumption -- if I run stop-all.sh _successfully_ on a Hadoop deployment
(which means every node in the grid is using the same path to Hadoop), then
that Hadoop installation becomes invisible, and then any other Hadoop
deployment could start up and take its place on the grid. Let me know if
this assumption is wrong.

I was having a lot of grief trying to do a parallel, better permissioned
Hadoop install the easy way, so I just went ahead and make copies on each
node into the /new/dir location, and pointed hdfs.tmp.dir appropriately.

So in a normal start-all.sh sequence, we have the following processes
spawned:
- master has NameNode, 2ndyNameNode, and JobTracker
- worker has DataNode and TaskTracker

After I powered down the normal Hadoop installation. I tried to start-all.sh
mine. Again, everything with this Hadoop should point its home to
/new/dir/hadoop, unless there's some deep hidden param I didn't know about.
The processes I got were only
- master: 2ndyNameNode, JobTracker
- worker: TaskTracker

Another hint is the error that calling the hadoop shell gives:
$ bin/hadoop dfs -ls /
08/08/04 19:25:32 INFO ipc.Client: Retrying connect to server:
master/ip:50001. Already tried 1 time(s).
08/08/04 19:25:33 INFO ipc.Client: Retrying connect to server: master
/ip:50001. Already tried 2 time(s).
08/08/04 19:25:34 INFO ipc.Client: Retrying connect to server:
master/ip:50001. Already tried 3 time(s).

I can't for the life of me reason why the others are missing.

On Mon, Aug 4, 2008 at 4:17 PM, Meng Mao <me...@gmail.com> wrote:

> I see. I think I could also modify the hadoop-env.sh in the new conf/
> folders per datanode to point
> to the right place for HADOOP_HOME.
>
>
> On Mon, Aug 4, 2008 at 3:21 PM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:
>
>>
>>
>>
>> On 8/4/08 11:10 AM, "Meng Mao" <me...@gmail.com> wrote:
>> > I suppose I could, for each datanode, symlink things to point to the
>> actual
>> > Hadoop installation. But really, I would like the setup that is hinted
>> as
>> > possible by statement 1). Is there a way I could do it, or should that
>> bit
>> > of documentation read, "All machines in the cluster _must_ have the same
>> > HADOOP_HOME?"
>>
>>     If you run the -all scripts, they assume the location is the same.
>> AFAIK, there is nothing preventing you from building your own -all scripts
>> that point to the different location to start/stop the data nodes.
>>
>>
>>
>
>
> --
> hustlin, hustlin, everyday I'm hustlin
>



-- 
hustlin, hustlin, everyday I'm hustlin

Re: having different HADOOP_HOME for master and slaves?

Posted by Meng Mao <me...@gmail.com>.
I see. I think I could also modify the hadoop-env.sh in the new conf/
folders per datanode to point
to the right place for HADOOP_HOME.

On Mon, Aug 4, 2008 at 3:21 PM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:

>
>
>
> On 8/4/08 11:10 AM, "Meng Mao" <me...@gmail.com> wrote:
> > I suppose I could, for each datanode, symlink things to point to the
> actual
> > Hadoop installation. But really, I would like the setup that is hinted as
> > possible by statement 1). Is there a way I could do it, or should that
> bit
> > of documentation read, "All machines in the cluster _must_ have the same
> > HADOOP_HOME?"
>
>     If you run the -all scripts, they assume the location is the same.
> AFAIK, there is nothing preventing you from building your own -all scripts
> that point to the different location to start/stop the data nodes.
>
>
>


-- 
hustlin, hustlin, everyday I'm hustlin

Re: having different HADOOP_HOME for master and slaves?

Posted by Allen Wittenauer <aw...@yahoo-inc.com>.


On 8/4/08 11:10 AM, "Meng Mao" <me...@gmail.com> wrote:
> I suppose I could, for each datanode, symlink things to point to the actual
> Hadoop installation. But really, I would like the setup that is hinted as
> possible by statement 1). Is there a way I could do it, or should that bit
> of documentation read, "All machines in the cluster _must_ have the same
> HADOOP_HOME?"

    If you run the -all scripts, they assume the location is the same.
AFAIK, there is nothing preventing you from building your own -all scripts
that point to the different location to start/stop the data nodes.