You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Arthur Chan <ar...@gmail.com> on 2015/04/04 16:25:05 UTC
Hadoop 2.6.0, How to add/remove node to/from running cluster
Hi,
Need help, I have few questions:
My Hadoop version: 2.6.0 (3 nodes)
My OS: CentOS 7
Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
Q2) How to rebalance the cluster after the new node is added?
Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
I am new to Hadoop 2.6.0. please help.
Regards
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
Just one file.
Step1: edit hdfs-site.xml in the NameNode.
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode,
add the full name of the node to be decommissioned
Step3: run the following command:
$ hdfs dfsadmin -refreshNodes
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
distribute-exclude.sh is to distribute the exclude file
to all the NameNodes. If you are running only one NameNode
(i.e. you are not using HDFS Federation), you don't need to
use this script.
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
This command is wrong.
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
For decommissioning NodeManager, just stopping NodeManager is fine.
If you want to decommission NodeManager gracefully, you need to do
the following steps.
1. edit yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/myhadoop/etc/hadoop/nodemanager-excludes</value>
</property>
2. edit /myhadoop/etc/hadoop/nodemanager-excludes
in the ResourceManager, add the full name of the node
to be decommissioned
3. execute the following command
$ yarn rmadmin -refreshNodes
Regards,
Akira
On 4/5/15 09:58, Arthur Chan wrote:
> Hi, Thanks.
>
>
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
>
>
> Could you advise if my following steps are correct?
>
>
> === My Hadoop is 2.6.0 ===
> Step 1:
> edit mapred-site.xml (ADD)
> <property>
>
<name>dfs.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>dfs.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 2:
>
> edit hdfs-site.xml (ADD)
>
> <property>
>
<name>mapred.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>mapred.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 3:
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
> nodes to it
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the
> node to be decommissioned
>
> step 4: run the following command
>
> *hdfs dfsadmin –refreshNodes*
>
> *hdfs mradmin –refreshNodes *
>
>
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
>
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
>
>
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
> Regards
> Arthur
>
>
> On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp <ma...@oss.nttdata.co.jp>> wrote:
>
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer>
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands>
>
> Regards,
> Akira
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
Just one file.
Step1: edit hdfs-site.xml in the NameNode.
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode,
add the full name of the node to be decommissioned
Step3: run the following command:
$ hdfs dfsadmin -refreshNodes
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
distribute-exclude.sh is to distribute the exclude file
to all the NameNodes. If you are running only one NameNode
(i.e. you are not using HDFS Federation), you don't need to
use this script.
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
This command is wrong.
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
For decommissioning NodeManager, just stopping NodeManager is fine.
If you want to decommission NodeManager gracefully, you need to do
the following steps.
1. edit yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/myhadoop/etc/hadoop/nodemanager-excludes</value>
</property>
2. edit /myhadoop/etc/hadoop/nodemanager-excludes
in the ResourceManager, add the full name of the node
to be decommissioned
3. execute the following command
$ yarn rmadmin -refreshNodes
Regards,
Akira
On 4/5/15 09:58, Arthur Chan wrote:
> Hi, Thanks.
>
>
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
>
>
> Could you advise if my following steps are correct?
>
>
> === My Hadoop is 2.6.0 ===
> Step 1:
> edit mapred-site.xml (ADD)
> <property>
>
<name>dfs.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>dfs.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 2:
>
> edit hdfs-site.xml (ADD)
>
> <property>
>
<name>mapred.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>mapred.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 3:
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
> nodes to it
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the
> node to be decommissioned
>
> step 4: run the following command
>
> *hdfs dfsadmin –refreshNodes*
>
> *hdfs mradmin –refreshNodes *
>
>
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
>
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
>
>
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
> Regards
> Arthur
>
>
> On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp <ma...@oss.nttdata.co.jp>> wrote:
>
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer>
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands>
>
> Regards,
> Akira
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
Just one file.
Step1: edit hdfs-site.xml in the NameNode.
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode,
add the full name of the node to be decommissioned
Step3: run the following command:
$ hdfs dfsadmin -refreshNodes
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
distribute-exclude.sh is to distribute the exclude file
to all the NameNodes. If you are running only one NameNode
(i.e. you are not using HDFS Federation), you don't need to
use this script.
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
This command is wrong.
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
For decommissioning NodeManager, just stopping NodeManager is fine.
If you want to decommission NodeManager gracefully, you need to do
the following steps.
1. edit yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/myhadoop/etc/hadoop/nodemanager-excludes</value>
</property>
2. edit /myhadoop/etc/hadoop/nodemanager-excludes
in the ResourceManager, add the full name of the node
to be decommissioned
3. execute the following command
$ yarn rmadmin -refreshNodes
Regards,
Akira
On 4/5/15 09:58, Arthur Chan wrote:
> Hi, Thanks.
>
>
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
>
>
> Could you advise if my following steps are correct?
>
>
> === My Hadoop is 2.6.0 ===
> Step 1:
> edit mapred-site.xml (ADD)
> <property>
>
<name>dfs.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>dfs.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 2:
>
> edit hdfs-site.xml (ADD)
>
> <property>
>
<name>mapred.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>mapred.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 3:
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
> nodes to it
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the
> node to be decommissioned
>
> step 4: run the following command
>
> *hdfs dfsadmin –refreshNodes*
>
> *hdfs mradmin –refreshNodes *
>
>
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
>
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
>
>
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
> Regards
> Arthur
>
>
> On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp <ma...@oss.nttdata.co.jp>> wrote:
>
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer>
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands>
>
> Regards,
> Akira
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
Just one file.
Step1: edit hdfs-site.xml in the NameNode.
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
Step2: edit /myhadoop/etc/hadoop/datanode-excludes in the NameNode,
add the full name of the node to be decommissioned
Step3: run the following command:
$ hdfs dfsadmin -refreshNodes
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
distribute-exclude.sh is to distribute the exclude file
to all the NameNodes. If you are running only one NameNode
(i.e. you are not using HDFS Federation), you don't need to
use this script.
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
This command is wrong.
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
For decommissioning NodeManager, just stopping NodeManager is fine.
If you want to decommission NodeManager gracefully, you need to do
the following steps.
1. edit yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/myhadoop/etc/hadoop/nodemanager-excludes</value>
</property>
2. edit /myhadoop/etc/hadoop/nodemanager-excludes
in the ResourceManager, add the full name of the node
to be decommissioned
3. execute the following command
$ yarn rmadmin -refreshNodes
Regards,
Akira
On 4/5/15 09:58, Arthur Chan wrote:
> Hi, Thanks.
>
>
> For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
> files should be modified?
>
>
> Could you advise if my following steps are correct?
>
>
> === My Hadoop is 2.6.0 ===
> Step 1:
> edit mapred-site.xml (ADD)
> <property>
>
<name>dfs.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>dfs.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 2:
>
> edit hdfs-site.xml (ADD)
>
> <property>
>
<name>mapred.hosts</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
> <property>
>
<name>mapred.hosts.exclude</name>
> <value>/myhadoop/etc/hadoop/datanode-excludes</value>
> </property>
>
>
> step 3:
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
> nodes to it
> edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the
> node to be decommissioned
>
> step 4: run the following command
>
> *hdfs dfsadmin –refreshNodes*
>
> *hdfs mradmin –refreshNodes *
>
>
> Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
> use "*hdfs dfsadmin –refreshNodes", *
> instead, should use "sbin/*distribute-exclude.sh"*
>
> Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
> load main class mradmin"
>
>
> Please advise the correct way to decommission a datanode completely (for
> both DFS and Nodemanager) in Hadoop 2.x
> Regards
> Arthur
>
>
> On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA
> <ajisakaa@oss.nttdata.co.jp <ma...@oss.nttdata.co.jp>> wrote:
>
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Balancer
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer>
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/__r2.6.0/hadoop-project-dist/__hadoop-hdfs/HdfsUserGuide.__html#Shell_Commands
> <http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands>
>
> Regards,
> Akira
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, Thanks.
For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
files should be modified?
Could you advise if my following steps are correct?
=== My Hadoop is 2.6.0 ===
Step 1:
edit mapred-site.xml (ADD)
<property>
<name>dfs.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 2:
edit hdfs-site.xml (ADD)
<property>
<name>mapred.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>mapred.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 3:
edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
nodes to it
edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the node
to be decommissioned
step 4: run the following command
*hdfs dfsadmin –refreshNodes*
*hdfs mradmin –refreshNodes *
Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
use "*hdfs
dfsadmin –refreshNodes", *
instead, should use "sbin/*distribute-exclude.sh"*
Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
load main class mradmin"
Please advise the correct way to decommission a datanode completely (for
both DFS and Nodemanager) in Hadoop 2.x
Regards
Arthur
On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA <aj...@oss.nttdata.co.jp>
wrote:
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Balancer
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
>
> Regards,
> Akira
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, Thanks.
For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
files should be modified?
Could you advise if my following steps are correct?
=== My Hadoop is 2.6.0 ===
Step 1:
edit mapred-site.xml (ADD)
<property>
<name>dfs.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 2:
edit hdfs-site.xml (ADD)
<property>
<name>mapred.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>mapred.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 3:
edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
nodes to it
edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the node
to be decommissioned
step 4: run the following command
*hdfs dfsadmin –refreshNodes*
*hdfs mradmin –refreshNodes *
Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
use "*hdfs
dfsadmin –refreshNodes", *
instead, should use "sbin/*distribute-exclude.sh"*
Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
load main class mradmin"
Please advise the correct way to decommission a datanode completely (for
both DFS and Nodemanager) in Hadoop 2.x
Regards
Arthur
On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA <aj...@oss.nttdata.co.jp>
wrote:
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Balancer
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
>
> Regards,
> Akira
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, Thanks.
For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
files should be modified?
Could you advise if my following steps are correct?
=== My Hadoop is 2.6.0 ===
Step 1:
edit mapred-site.xml (ADD)
<property>
<name>dfs.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 2:
edit hdfs-site.xml (ADD)
<property>
<name>mapred.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>mapred.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 3:
edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
nodes to it
edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the node
to be decommissioned
step 4: run the following command
*hdfs dfsadmin –refreshNodes*
*hdfs mradmin –refreshNodes *
Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
use "*hdfs
dfsadmin –refreshNodes", *
instead, should use "sbin/*distribute-exclude.sh"*
Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
load main class mradmin"
Please advise the correct way to decommission a datanode completely (for
both DFS and Nodemanager) in Hadoop 2.x
Regards
Arthur
On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA <aj...@oss.nttdata.co.jp>
wrote:
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Balancer
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
>
> Regards,
> Akira
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, Thanks.
For 3), about "dfs.hosts, dfs.hosts.exclude", how many configuration XML
files should be modified?
Could you advise if my following steps are correct?
=== My Hadoop is 2.6.0 ===
Step 1:
edit mapred-site.xml (ADD)
<property>
<name>dfs.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 2:
edit hdfs-site.xml (ADD)
<property>
<name>mapred.hosts</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
<property>
<name>mapred.hosts.exclude</name>
<value>/myhadoop/etc/hadoop/datanode-excludes</value>
</property>
step 3:
edit /myhadoop/etc/hadoop/datanode-excludes, add the full names of all
nodes to it
edit /myhadoop/etc/hadoop/datanode-excludes, add the full name of the node
to be decommissioned
step 4: run the following command
*hdfs dfsadmin –refreshNodes*
*hdfs mradmin –refreshNodes *
Q1) It seems that for 2.6.0 (or even 2.4.1) the correct way should not
use "*hdfs
dfsadmin –refreshNodes", *
instead, should use "sbin/*distribute-exclude.sh"*
Q2) I tried *hdfs mradmin –refreshNodes, got * "Error: Could not find or
load main class mradmin"
Please advise the correct way to decommission a datanode completely (for
both DFS and Nodemanager) in Hadoop 2.x
Regards
Arthur
On Sun, Apr 5, 2015 at 12:53 AM, Akira AJISAKA <aj...@oss.nttdata.co.jp>
wrote:
> Hi Arthur,
>
> > 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>
> Just starting a datanode is fine.
> The datanode will be added to the cluster automatically.
>
> > 2) How to rebalance the cluster after the new node is added?
>
> Please see http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Balancer
>
> > 3) I could not find the section about how to decommission a datanode
> > from a running cluster
>
> Please see -refreshNodes section of this document.
> http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/
> hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
>
> Regards,
> Akira
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
Just starting a datanode is fine.
The datanode will be added to the cluster automatically.
> 2) How to rebalance the cluster after the new node is added?
Please see
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
Please see -refreshNodes section of this document.
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
Regards,
Akira
On 4/4/15 23:56, Arthur Chan wrote:
> Hi, thanks
>
>
> But, this link should be outdated and is not good for Hadoop 2.6.0 (or
> Hadoop 2.x I think)
> Problems here if use the steps in
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
> 1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin
> now if the Hadoop version is 2.6.0)
>
> 2) ./sbin/hadoop-daemon.sh start tasktracker
>
> starting tasktracker, logging to
> /edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
>
> DEPRECATED: Use of this script to execute mapred command is deprecated.
>
> Instead use the mapred command for it.
>
> Sorry, the tasktracker command is no longer supported.
>
> You may find similar functionality with the "yarn" shell command.
>
> Usage: mapred [--config confdir] COMMAND
>
> where COMMAND is one of:
>
> pipes run a Pipes job
>
> job manipulate MapReduce jobs
>
> queue get information regarding JobQueues
>
>
>
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
>
>
> Please help ! (especially for Q3 how to decommission a datanode from a
> running cluster)
>
>
> Regards
>
>
> On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <ashutosh.k78@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan
> <arthur.hk.chan@gmail.com <ma...@gmail.com>> wrote:
>
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0
> cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
Just starting a datanode is fine.
The datanode will be added to the cluster automatically.
> 2) How to rebalance the cluster after the new node is added?
Please see
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
Please see -refreshNodes section of this document.
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
Regards,
Akira
On 4/4/15 23:56, Arthur Chan wrote:
> Hi, thanks
>
>
> But, this link should be outdated and is not good for Hadoop 2.6.0 (or
> Hadoop 2.x I think)
> Problems here if use the steps in
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
> 1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin
> now if the Hadoop version is 2.6.0)
>
> 2) ./sbin/hadoop-daemon.sh start tasktracker
>
> starting tasktracker, logging to
> /edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
>
> DEPRECATED: Use of this script to execute mapred command is deprecated.
>
> Instead use the mapred command for it.
>
> Sorry, the tasktracker command is no longer supported.
>
> You may find similar functionality with the "yarn" shell command.
>
> Usage: mapred [--config confdir] COMMAND
>
> where COMMAND is one of:
>
> pipes run a Pipes job
>
> job manipulate MapReduce jobs
>
> queue get information regarding JobQueues
>
>
>
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
>
>
> Please help ! (especially for Q3 how to decommission a datanode from a
> running cluster)
>
>
> Regards
>
>
> On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <ashutosh.k78@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan
> <arthur.hk.chan@gmail.com <ma...@gmail.com>> wrote:
>
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0
> cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
Just starting a datanode is fine.
The datanode will be added to the cluster automatically.
> 2) How to rebalance the cluster after the new node is added?
Please see
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
Please see -refreshNodes section of this document.
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
Regards,
Akira
On 4/4/15 23:56, Arthur Chan wrote:
> Hi, thanks
>
>
> But, this link should be outdated and is not good for Hadoop 2.6.0 (or
> Hadoop 2.x I think)
> Problems here if use the steps in
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
> 1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin
> now if the Hadoop version is 2.6.0)
>
> 2) ./sbin/hadoop-daemon.sh start tasktracker
>
> starting tasktracker, logging to
> /edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
>
> DEPRECATED: Use of this script to execute mapred command is deprecated.
>
> Instead use the mapred command for it.
>
> Sorry, the tasktracker command is no longer supported.
>
> You may find similar functionality with the "yarn" shell command.
>
> Usage: mapred [--config confdir] COMMAND
>
> where COMMAND is one of:
>
> pipes run a Pipes job
>
> job manipulate MapReduce jobs
>
> queue get information regarding JobQueues
>
>
>
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
>
>
> Please help ! (especially for Q3 how to decommission a datanode from a
> running cluster)
>
>
> Regards
>
>
> On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <ashutosh.k78@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan
> <arthur.hk.chan@gmail.com <ma...@gmail.com>> wrote:
>
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0
> cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
Hi Arthur,
> 1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
Just starting a datanode is fine.
The datanode will be added to the cluster automatically.
> 2) How to rebalance the cluster after the new node is added?
Please see
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
Please see -refreshNodes section of this document.
http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Shell_Commands
Regards,
Akira
On 4/4/15 23:56, Arthur Chan wrote:
> Hi, thanks
>
>
> But, this link should be outdated and is not good for Hadoop 2.6.0 (or
> Hadoop 2.x I think)
> Problems here if use the steps in
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
> 1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin
> now if the Hadoop version is 2.6.0)
>
> 2) ./sbin/hadoop-daemon.sh start tasktracker
>
> starting tasktracker, logging to
> /edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
>
> DEPRECATED: Use of this script to execute mapred command is deprecated.
>
> Instead use the mapred command for it.
>
> Sorry, the tasktracker command is no longer supported.
>
> You may find similar functionality with the "yarn" shell command.
>
> Usage: mapred [--config confdir] COMMAND
>
> where COMMAND is one of:
>
> pipes run a Pipes job
>
> job manipulate MapReduce jobs
>
> queue get information regarding JobQueues
>
>
>
> 3) I could not find the section about how to decommission a datanode
> from a running cluster
>
>
> Please help ! (especially for Q3 how to decommission a datanode from a
> running cluster)
>
>
> Regards
>
>
> On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <ashutosh.k78@gmail.com
> <ma...@gmail.com>> wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan
> <arthur.hk.chan@gmail.com <ma...@gmail.com>> wrote:
>
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0
> cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, thanks
But, this link should be outdated and is not good for Hadoop 2.6.0 (or
Hadoop 2.x I think)
Problems here if use the steps in
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin now
if the Hadoop version is 2.6.0)
2) ./sbin/hadoop-daemon.sh start tasktracker
starting tasktracker, logging to
/edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
Sorry, the tasktracker command is no longer supported.
You may find similar functionality with the "yarn" shell command.
Usage: mapred [--config confdir] COMMAND
where COMMAND is one of:
pipes run a Pipes job
job manipulate MapReduce jobs
queue get information regarding JobQueues
3) I could not find the section about how to decommission a datanode from a
running cluster
Please help ! (especially for Q3 how to decommission a datanode from a
running cluster)
Regards
On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <as...@gmail.com>
wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Need help, I have few questions:
>> My Hadoop version: 2.6.0 (3 nodes)
>> My OS: CentOS 7
>>
>> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>> Q2) How to rebalance the cluster after the new node is added?
>> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>>
>> I am new to Hadoop 2.6.0. please help.
>>
>> Regards
>>
>>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, thanks
But, this link should be outdated and is not good for Hadoop 2.6.0 (or
Hadoop 2.x I think)
Problems here if use the steps in
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin now
if the Hadoop version is 2.6.0)
2) ./sbin/hadoop-daemon.sh start tasktracker
starting tasktracker, logging to
/edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
Sorry, the tasktracker command is no longer supported.
You may find similar functionality with the "yarn" shell command.
Usage: mapred [--config confdir] COMMAND
where COMMAND is one of:
pipes run a Pipes job
job manipulate MapReduce jobs
queue get information regarding JobQueues
3) I could not find the section about how to decommission a datanode from a
running cluster
Please help ! (especially for Q3 how to decommission a datanode from a
running cluster)
Regards
On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <as...@gmail.com>
wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Need help, I have few questions:
>> My Hadoop version: 2.6.0 (3 nodes)
>> My OS: CentOS 7
>>
>> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>> Q2) How to rebalance the cluster after the new node is added?
>> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>>
>> I am new to Hadoop 2.6.0. please help.
>>
>> Regards
>>
>>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, thanks
But, this link should be outdated and is not good for Hadoop 2.6.0 (or
Hadoop 2.x I think)
Problems here if use the steps in
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin now
if the Hadoop version is 2.6.0)
2) ./sbin/hadoop-daemon.sh start tasktracker
starting tasktracker, logging to
/edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
Sorry, the tasktracker command is no longer supported.
You may find similar functionality with the "yarn" shell command.
Usage: mapred [--config confdir] COMMAND
where COMMAND is one of:
pipes run a Pipes job
job manipulate MapReduce jobs
queue get information regarding JobQueues
3) I could not find the section about how to decommission a datanode from a
running cluster
Please help ! (especially for Q3 how to decommission a datanode from a
running cluster)
Regards
On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <as...@gmail.com>
wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Need help, I have few questions:
>> My Hadoop version: 2.6.0 (3 nodes)
>> My OS: CentOS 7
>>
>> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>> Q2) How to rebalance the cluster after the new node is added?
>> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>>
>> I am new to Hadoop 2.6.0. please help.
>>
>> Regards
>>
>>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Arthur Chan <ar...@gmail.com>.
Hi, thanks
But, this link should be outdated and is not good for Hadoop 2.6.0 (or
Hadoop 2.x I think)
Problems here if use the steps in
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
1) the hadoop-daemon.sh is no longer in bin folder (should be in sbin now
if the Hadoop version is 2.6.0)
2) ./sbin/hadoop-daemon.sh start tasktracker
starting tasktracker, logging to
/edh/edh_logs/hadoop/edhuser/hadoop-edhuser-tasktracker-m1036.emblocsoft.net.out
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
Sorry, the tasktracker command is no longer supported.
You may find similar functionality with the "yarn" shell command.
Usage: mapred [--config confdir] COMMAND
where COMMAND is one of:
pipes run a Pipes job
job manipulate MapReduce jobs
queue get information regarding JobQueues
3) I could not find the section about how to decommission a datanode from a
running cluster
Please help ! (especially for Q3 how to decommission a datanode from a
running cluster)
Regards
On Sat, Apr 4, 2015 at 10:34 PM, Ashutosh Kumar <as...@gmail.com>
wrote:
>
> http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
>
>
> On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Need help, I have few questions:
>> My Hadoop version: 2.6.0 (3 nodes)
>> My OS: CentOS 7
>>
>> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
>> Q2) How to rebalance the cluster after the new node is added?
>> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>>
>> I am new to Hadoop 2.6.0. please help.
>>
>> Regards
>>
>>
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Ashutosh Kumar <as...@gmail.com>.
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
>
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Ashutosh Kumar <as...@gmail.com>.
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
>
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Ashutosh Kumar <as...@gmail.com>.
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
>
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>
Re: Hadoop 2.6.0, How to add/remove node to/from running cluster
Posted by Ashutosh Kumar <as...@gmail.com>.
http://wiki.apache.org/hadoop/FAQ#I_have_a_new_node_I_want_to_add_to_a_running_Hadoop_cluster.3B_how_do_I_start_services_on_just_one_node.3F
On Sat, Apr 4, 2015 at 7:55 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> Need help, I have few questions:
> My Hadoop version: 2.6.0 (3 nodes)
> My OS: CentOS 7
>
> Q1) How to add a new datanode to a running Hadoop 2.6.0 cluster?
> Q2) How to rebalance the cluster after the new node is added?
> Q3) How to decommission a datanode from the running Hadoop 2.6.0 cluster?
>
> I am new to Hadoop 2.6.0. please help.
>
> Regards
>
>