You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ben Kucinich <be...@gmail.com> on 2008/02/07 19:51:59 UTC
Starting up a larger cluster
In the Nutch wiki, I was reading this
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
I have problems understanding this section:
== Starting up a larger cluster ==
Ensure that the Hadoop package is accessible from the same path on
all nodes that are to be included in the cluster. If you have
separated configuration from the install then ensure that the config
directory is also accessible the same way.
Populate the slaves file with the nodes to be included in the
cluster. One node per line.
1) Does the first line mean, that I have to place the hadoop folder in
exactly the same location on every slave node? For example, if I put
hadoop home directory in my /usr/local/ in master node it should be
present in /user/local/ in all the slave nodes as well?
2) I ran start-all.sh in one node (192.168.1.2) with fs.default.name
as 192.168.1.2:9000 and mapred.job.tracker as 192.168.1.2:9001. So, I
believe this will play the role of master node. I did not populate the
slaves file with any slave nodes. But in many other systems,
192.168.1.3, 192.168.1.4, etc. I made the same settings in
hadoop-site.xml. So I believe these are slave nodes. Now in the slave
nodes I ran commands like bin/hadoop -dfs put dir newdir and the
newdir was created in the DFS. I wonder how the master node allowed
the slave nodes to put the files even though I did not populate the
slaves file.
Please help me with these queries since I am new to Hadoop.
Re: Starting up a larger cluster
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Feb 12, 2008, at 7:08 AM, Marco Nicosia wrote:
> DFS should place one replica per rack:
> http://issues.apache.org/jira/browse/HADOOP-2559
No, that would hurt the aggregate write throughput. Read the comment
on 2559:
http://issues.apache.org/jira/browse/HADOOP-2559?
focusedCommentId=12567129#action_12567129
Re: Starting up a larger cluster
Posted by Marco Nicosia <ma...@yahoo-inc.com>.
DFS should place one replica per rack:
http://issues.apache.org/jira/browse/HADOOP-2559
On 2/9/08 22:53, "Owen O'Malley" <oo...@yahoo-inc.com> wrote:
>
> On Feb 8, 2008, at 9:32 AM, Jeff Eastman wrote:
>
>> I noticed that phenomena right off the bat. Is that a designed
>> "feature"
>> or just an unhappy consequence of how blocks are allocated?
>
> It was driven by a desire to maximize HDFS write throughput, which
> has unfortunate effects in the case of a small set of nodes uploading
> data.
>
> We are going to be exploring different block placements and looking
> at what happens to performance as we do so. The current block
> allocations look like:
>
> local node -> rack local -> off rack
>
> My personal inclination is for something like:
>
> rack local -> off rack -> other node on second rack
>
> That will dramatically cut down on both the node and rack hotspots
> without killing your write performance, because you still only have
> one write through the network core.
>
> -- Owen
>
--
Marco Nicosia - Grid Services Ops
Systems, Tools, and Services Group
Re: Starting up a larger cluster
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Feb 8, 2008, at 9:32 AM, Jeff Eastman wrote:
> I noticed that phenomena right off the bat. Is that a designed
> "feature"
> or just an unhappy consequence of how blocks are allocated?
It was driven by a desire to maximize HDFS write throughput, which
has unfortunate effects in the case of a small set of nodes uploading
data.
We are going to be exploring different block placements and looking
at what happens to performance as we do so. The current block
allocations look like:
local node -> rack local -> off rack
My personal inclination is for something like:
rack local -> off rack -> other node on second rack
That will dramatically cut down on both the node and rack hotspots
without killing your write performance, because you still only have
one write through the network core.
-- Owen
Re: Starting up a larger cluster
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 2/8/08 9:32 AM, "Jeff Eastman" <je...@collab.net> wrote:
> I noticed that phenomena right off the bat. Is that a designed "feature"
> or just an unhappy consequence of how blocks are allocated?
My understanding is that this is by design--when you are running a MR
job, you want the output, temp files, etc, to be local.
> Ted
> compensates for this by aggressively rebalancing his cluster often by
> adjusting the replication up and down, but I wonder if an improvement in
> the allocation strategy would improve this.
IIRC, we're getting a block re-balancer in 0.16 so this particular
annoyance should mostly go away soon.
> I've also used Ted's trick, with less than marvelous results. I'd hate
> to pull my biggest machine (where I store all the backup files) out of
> the cluster just to get more even block distribution but I may have to.
Been there, done that. (At one time, we were decomm'ing entire racks to
force redistribution. I seem recall that we hit a bug so we then slowed
down to doing 10 at a time.)
RE: Starting up a larger cluster
Posted by Jeff Eastman <je...@collab.net>.
I noticed that phenomena right off the bat. Is that a designed "feature"
or just an unhappy consequence of how blocks are allocated? Ted
compensates for this by aggressively rebalancing his cluster often by
adjusting the replication up and down, but I wonder if an improvement in
the allocation strategy would improve this.
I've also used Ted's trick, with less than marvelous results. I'd hate
to pull my biggest machine (where I store all the backup files) out of
the cluster just to get more even block distribution but I may have to.
Jeff
-----Original Message-----
From: Allen Wittenauer [mailto:aw@yahoo-inc.com]
Sent: Friday, February 08, 2008 9:15 AM
To: core-user@hadoop.apache.org
Subject: Re: Starting up a larger cluster
On 2/7/08 11:01 PM, "Tim Wintle" <ti...@teamrubber.com> wrote:
> it's
> useful to be able to connect from nodes that aren't in the slaves file
> so that you can put in input data direct from another machine that's
not
> part of the cluster,
I'd actually recommend this as a best practice. We've been bit
over...
and over... and over... with users loading data into HDFS from a data
node
only to discover that the block distribution is pretty horrid.... which
in
turn means that MR performance is equally horrid. [Remember: all writes
will
go the local node if it is a data node!]
We're now down to the point that we've got one (relatively smaller)
grid
that is used for data loading/creation/extraction which then distcp's
its
contents to another grid.
Less than ideal, but definitely helps the performance of the entire
'real' grid.
Re: Starting up a larger cluster
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 2/7/08 11:01 PM, "Tim Wintle" <ti...@teamrubber.com> wrote:
> it's
> useful to be able to connect from nodes that aren't in the slaves file
> so that you can put in input data direct from another machine that's not
> part of the cluster,
I'd actually recommend this as a best practice. We've been bit over...
and over... and over... with users loading data into HDFS from a data node
only to discover that the block distribution is pretty horrid.... which in
turn means that MR performance is equally horrid. [Remember: all writes will
go the local node if it is a data node!]
We're now down to the point that we've got one (relatively smaller) grid
that is used for data loading/creation/extraction which then distcp's its
contents to another grid.
Less than ideal, but definitely helps the performance of the entire
'real' grid.
RE: Starting up a larger cluster
Posted by Tim Wintle <ti...@teamrubber.com>.
You can set which nodes are allowed to connect in hadoop-site.xml - it's
useful to be able to connect from nodes that aren't in the slaves file
so that you can put in input data direct from another machine that's not
part of the cluster, or add extra machines on the fly (just make sure
they're routing correctly first!). You can also run jobs direct from
your workstation (without having to scp your code, ssh etc)
If you look through the shell scripts you should see exactly what the
slaves file is used for. It's fairly easy to modify the scripts to start
a single rack, so you should be able to bring up machines when you need
them.
Tim
On Thu, 2008-02-07 at 12:24 -0800, Jeff Eastman wrote:
> Hi Ben,
>
> I've been down this same path recently and I think I understand your
> issues:
>
> 1) Yes, you need the hadoop folder to be in the same location on each
> node. Only the master node actually uses the slaves file, to start up
> DataNode and JobTracker daemons on those nodes.
> 2) If you did not specify any slave nodes on your master node then the
> start-all did not create these processes on any nodes other than master.
> This node can be accessed and the dfs written to from other machines as
> you can do but there is no replication since there is only one DataNode.
>
> Try running jps on your other nodes to verify this, and access the
> NameNode web page to see what slaves you actually have running. By
> adding your slave nodes to the slaves file on your master and bouncing
> hadoop you should see a big difference in the size of your cluster.
>
> Good luck, it's an adventure,
> Jeff
>
> -----Original Message-----
> From: Ben Kucinich [mailto:benkucinich@gmail.com]
> Sent: Thursday, February 07, 2008 10:52 AM
> To: core-user@hadoop.apache.org
> Subject: Starting up a larger cluster
>
> In the Nutch wiki, I was reading this
> http://wiki.apache.org/hadoop/GettingStartedWithHadoop
>
> I have problems understanding this section:
>
> == Starting up a larger cluster ==
>
> Ensure that the Hadoop package is accessible from the same path on
> all nodes that are to be included in the cluster. If you have
> separated configuration from the install then ensure that the config
> directory is also accessible the same way.
> Populate the slaves file with the nodes to be included in the
> cluster. One node per line.
>
> 1) Does the first line mean, that I have to place the hadoop folder in
> exactly the same location on every slave node? For example, if I put
> hadoop home directory in my /usr/local/ in master node it should be
> present in /user/local/ in all the slave nodes as well?
>
> 2) I ran start-all.sh in one node (192.168.1.2) with fs.default.name
> as 192.168.1.2:9000 and mapred.job.tracker as 192.168.1.2:9001. So, I
> believe this will play the role of master node. I did not populate the
> slaves file with any slave nodes. But in many other systems,
> 192.168.1.3, 192.168.1.4, etc. I made the same settings in
> hadoop-site.xml. So I believe these are slave nodes. Now in the slave
> nodes I ran commands like bin/hadoop -dfs put dir newdir and the
> newdir was created in the DFS. I wonder how the master node allowed
> the slave nodes to put the files even though I did not populate the
> slaves file.
>
> Please help me with these queries since I am new to Hadoop.
>
RE: Starting up a larger cluster
Posted by Jeff Eastman <je...@collab.net>.
Oops, should be TaskTracker.
-----Original Message-----
From: Jeff Eastman [mailto:jeastman@collab.net]
Sent: Thursday, February 07, 2008 12:24 PM
To: core-user@hadoop.apache.org
Subject: RE: Starting up a larger cluster
Hi Ben,
I've been down this same path recently and I think I understand your
issues:
1) Yes, you need the hadoop folder to be in the same location on each
node. Only the master node actually uses the slaves file, to start up
DataNode and JobTracker daemons on those nodes.
2) If you did not specify any slave nodes on your master node then the
start-all did not create these processes on any nodes other than master.
This node can be accessed and the dfs written to from other machines as
you can do but there is no replication since there is only one DataNode.
Try running jps on your other nodes to verify this, and access the
NameNode web page to see what slaves you actually have running. By
adding your slave nodes to the slaves file on your master and bouncing
hadoop you should see a big difference in the size of your cluster.
Good luck, it's an adventure,
Jeff
-----Original Message-----
From: Ben Kucinich [mailto:benkucinich@gmail.com]
Sent: Thursday, February 07, 2008 10:52 AM
To: core-user@hadoop.apache.org
Subject: Starting up a larger cluster
In the Nutch wiki, I was reading this
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
I have problems understanding this section:
== Starting up a larger cluster ==
Ensure that the Hadoop package is accessible from the same path on
all nodes that are to be included in the cluster. If you have
separated configuration from the install then ensure that the config
directory is also accessible the same way.
Populate the slaves file with the nodes to be included in the
cluster. One node per line.
1) Does the first line mean, that I have to place the hadoop folder in
exactly the same location on every slave node? For example, if I put
hadoop home directory in my /usr/local/ in master node it should be
present in /user/local/ in all the slave nodes as well?
2) I ran start-all.sh in one node (192.168.1.2) with fs.default.name
as 192.168.1.2:9000 and mapred.job.tracker as 192.168.1.2:9001. So, I
believe this will play the role of master node. I did not populate the
slaves file with any slave nodes. But in many other systems,
192.168.1.3, 192.168.1.4, etc. I made the same settings in
hadoop-site.xml. So I believe these are slave nodes. Now in the slave
nodes I ran commands like bin/hadoop -dfs put dir newdir and the
newdir was created in the DFS. I wonder how the master node allowed
the slave nodes to put the files even though I did not populate the
slaves file.
Please help me with these queries since I am new to Hadoop.
RE: Starting up a larger cluster
Posted by Jeff Eastman <je...@collab.net>.
Hi Ben,
I've been down this same path recently and I think I understand your
issues:
1) Yes, you need the hadoop folder to be in the same location on each
node. Only the master node actually uses the slaves file, to start up
DataNode and JobTracker daemons on those nodes.
2) If you did not specify any slave nodes on your master node then the
start-all did not create these processes on any nodes other than master.
This node can be accessed and the dfs written to from other machines as
you can do but there is no replication since there is only one DataNode.
Try running jps on your other nodes to verify this, and access the
NameNode web page to see what slaves you actually have running. By
adding your slave nodes to the slaves file on your master and bouncing
hadoop you should see a big difference in the size of your cluster.
Good luck, it's an adventure,
Jeff
-----Original Message-----
From: Ben Kucinich [mailto:benkucinich@gmail.com]
Sent: Thursday, February 07, 2008 10:52 AM
To: core-user@hadoop.apache.org
Subject: Starting up a larger cluster
In the Nutch wiki, I was reading this
http://wiki.apache.org/hadoop/GettingStartedWithHadoop
I have problems understanding this section:
== Starting up a larger cluster ==
Ensure that the Hadoop package is accessible from the same path on
all nodes that are to be included in the cluster. If you have
separated configuration from the install then ensure that the config
directory is also accessible the same way.
Populate the slaves file with the nodes to be included in the
cluster. One node per line.
1) Does the first line mean, that I have to place the hadoop folder in
exactly the same location on every slave node? For example, if I put
hadoop home directory in my /usr/local/ in master node it should be
present in /user/local/ in all the slave nodes as well?
2) I ran start-all.sh in one node (192.168.1.2) with fs.default.name
as 192.168.1.2:9000 and mapred.job.tracker as 192.168.1.2:9001. So, I
believe this will play the role of master node. I did not populate the
slaves file with any slave nodes. But in many other systems,
192.168.1.3, 192.168.1.4, etc. I made the same settings in
hadoop-site.xml. So I believe these are slave nodes. Now in the slave
nodes I ran commands like bin/hadoop -dfs put dir newdir and the
newdir was created in the DFS. I wonder how the master node allowed
the slave nodes to put the files even though I did not populate the
slaves file.
Please help me with these queries since I am new to Hadoop.