You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Zhanlei Ma <zm...@vmware.com> on 2010/04/01 05:12:56 UTC
How to Recommission?
How to Recommission or decommission DataNode(s) in hadoop???
Decommission(Del some Datanodes):
On a large cluster removing one or two data-nodes will not lead to any data loss, because name-node will replicate their blocks as long as it
will detect that the nodes are dead. With a large number of nodes getting removed or dying the probability of losing data is higher.
Hadoop offers the decommission feature to retire a set of existing data-nodes. The nodes to be retired should be included into the exclude file,
and the exclude file name should be specified as a configuration parameter dfs.hosts.exclude. This file should have been specified during
namenode startup. It could be a zero length file. You must use the full hostname, ip or ip:port format in this file. Then the shell command
bin/hadoop dfsadmin -refreshNodes
should be called, which forces the name-node to re-read the exclude file and start the decommission process.
Decommission does not happen momentarily since it requires replication of potentially a large number of blocks and we do not want the cluster to
be overwhelmed with just this one job. The decommission progress can be monitored on the name-node Web UI. Until all blocks are replicated the
node will be in "Decommission In Progress" state. When decommission is done the state will change to "Decommissioned". The nodes can be removed
whenever decommission is finished.
But how to Recommission? Wish your help.
Thanks.
RE: How to Recommission?
Posted by Zhanlei Ma <zm...@vmware.com>.
I got it,Thanks
-----Original Message-----
From: Michael Thomas [mailto:thomas@hep.caltech.edu]
Sent: 2010年4月1日 11:17
To: general@hadoop.apache.org
Subject: Re: How to Recommission?
On 03/31/2010 08:12 PM, Zhanlei Ma wrote:
> How to Recommission or decommission DataNode(s) in hadoop???
> Decommission(Del some Datanodes):
> On a large cluster removing one or two data-nodes will not lead to any data loss, because name-node will replicate their blocks as long as it
>
[...]
>
>
>
> But how to Recommission? Wish your help.
> Thanks.
>
Remove the hostname from your dfs.hosts.exclude file and run 'hadoop
dfsadmin -refreshNodes'. Then start the datanode process in the
'recommissioned' datanode again.
--Mike
Re: How to Recommission?
Posted by Michael Thomas <th...@hep.caltech.edu>.
On 03/31/2010 08:12 PM, Zhanlei Ma wrote:
> How to Recommission or decommission DataNode(s) in hadoop???
> Decommission(Del some Datanodes):
> On a large cluster removing one or two data-nodes will not lead to any data loss, because name-node will replicate their blocks as long as it
>
[...]
>
>
>
> But how to Recommission? Wish your help.
> Thanks.
>
Remove the hostname from your dfs.hosts.exclude file and run 'hadoop
dfsadmin -refreshNodes'. Then start the datanode process in the
'recommissioned' datanode again.
--Mike
Re: How to Recommission?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/31/10 8:12 PM, "Zhanlei Ma" <zm...@vmware.com> wrote:
> But how to Recommission? Wish your help.
Take them out of dfs.exclude and refreshnodes again.
Re: How to Recommission?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/31/10 8:12 PM, "Zhanlei Ma" <zm...@vmware.com> wrote:
> But how to Recommission? Wish your help.
Take them out of dfs.exclude and refreshnodes again.
Re: How to Recommission?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On 3/31/10 8:12 PM, "Zhanlei Ma" <zm...@vmware.com> wrote:
> But how to Recommission? Wish your help.
Take them out of dfs.exclude and refreshnodes again.