You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by jo...@accenture.com on 2011/08/12 17:58:19 UTC
Speed up node under replicated block during decomission
Hi All,
I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
<property>
<!-- 100Mbit/s -->
<name>dfs.balance.bandwidthPerSec</name>
<value>131072000</value>
</property>
Speed didn't seem to pick up. Do you know what may be happening?
Thanks!
Jonathan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
RE: Speed up node under replicated block during decomission
Posted by jo...@accenture.com.
I did have these settings on all the hdfs-site.xml nodes:
<property>
<!-- 100Mbit/s -->
<name>dfs.balance.bandwidthPerSec</name>
<value>131072000</value>
</property>
<property>
<name>dfs.max-repl-streams</name>
<value>50</value>
</property>
It is still taking over 1 day or longer for 1TB of under replicated blocks to replicate.
Thanks!
Jonathan
-----Original Message-----
From: Joey Echeverria [mailto:joey@cloudera.com]
Sent: Friday, August 12, 2011 9:14 AM
To: common-user@hadoop.apache.org
Subject: Re: Speed up node under replicated block during decomission
You can configure the undocumented variable dfs.max-repl-streams to
increase the number of replications a data-node is allowed to handle
at one time. The default value is 2. [1]
-Joey
[1] https://issues.apache.org/jira/browse/HADOOP-2606?focusedCommentId=12578700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12578700
On Fri, Aug 12, 2011 at 12:09 PM, Charles Wimmer <cw...@yahoo-inc.com> wrote:
> The balancer bandwidth setting does not affect decommissioning nodes. Decommisssioning nodes replicate as fast as the cluster is capable.
>
> The replication pace has many variables.
> Number nodes that are participating in the replication.
> The amount of network bandwidth each has.
> The amount of other HDFS activity at the time.
> Total blocks being replicated.
> Total data being replicated.
> Many others.
>
>
> On 8/12/11 8:58 AM, "jonathan.hwang@accenture.com" <jo...@accenture.com> wrote:
>
> Hi All,
>
> I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
>
> Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
> <property>
> <!-- 100Mbit/s -->
> <name>dfs.balance.bandwidthPerSec</name>
> <value>131072000</value>
> </property>
>
> Speed didn't seem to pick up. Do you know what may be happening?
>
> Thanks!
> Jonathan
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: Speed up node under replicated block during decomission
Posted by Joey Echeverria <jo...@cloudera.com>.
You can configure the undocumented variable dfs.max-repl-streams to
increase the number of replications a data-node is allowed to handle
at one time. The default value is 2. [1]
-Joey
[1] https://issues.apache.org/jira/browse/HADOOP-2606?focusedCommentId=12578700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12578700
On Fri, Aug 12, 2011 at 12:09 PM, Charles Wimmer <cw...@yahoo-inc.com> wrote:
> The balancer bandwidth setting does not affect decommissioning nodes. Decommisssioning nodes replicate as fast as the cluster is capable.
>
> The replication pace has many variables.
> Number nodes that are participating in the replication.
> The amount of network bandwidth each has.
> The amount of other HDFS activity at the time.
> Total blocks being replicated.
> Total data being replicated.
> Many others.
>
>
> On 8/12/11 8:58 AM, "jonathan.hwang@accenture.com" <jo...@accenture.com> wrote:
>
> Hi All,
>
> I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
>
> Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
> <property>
> <!-- 100Mbit/s -->
> <name>dfs.balance.bandwidthPerSec</name>
> <value>131072000</value>
> </property>
>
> Speed didn't seem to pick up. Do you know what may be happening?
>
> Thanks!
> Jonathan
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
RE: Speed up node under replicated block during decomission
Posted by Michael Segel <mi...@hotmail.com>.
Just a thought...
Really quick and dirty thing to do is to turn off the node.
Within 10 minutes the node looks down to the JT and NN so it gets marked as down.
Run an fsck and it will show the files as under replicated and then will do the replication at the faster speed to rebalance the cluster.
(100MB/sec should be ok on a 1GBe link)
Then you can drop the next node... much faster than trying to decomission the node.
Its not the best way to do it, but it works.
> From: harsh@cloudera.com
> Date: Fri, 12 Aug 2011 22:38:08 +0530
> Subject: Re: Speed up node under replicated block during decomission
> To: common-user@hadoop.apache.org
>
> It could be that your process has hung cause a particular resident
> block (file) requires a very large replication factor, and your
> remaining # of nodes is less than that value. This is a genuine reason
> for hang (but must be fixed). The process usually waits until there
> are no under-replicated blocks, so I'd use fsck to check if any such
> ones are present and setrep them to a lower value.
>
> On Fri, Aug 12, 2011 at 9:28 PM, <jo...@accenture.com> wrote:
> > Hi All,
> >
> > I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
> >
> > Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
> > <property>
> > <!-- 100Mbit/s -->
> > <name>dfs.balance.bandwidthPerSec</name>
> > <value>131072000</value>
> > </property>
> >
> > Speed didn't seem to pick up. Do you know what may be happening?
> >
> > Thanks!
> > Jonathan
> >
> > This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
> >
>
>
>
> --
> Harsh J
Re: Speed up node under replicated block during decomission
Posted by Harsh J <ha...@cloudera.com>.
It could be that your process has hung cause a particular resident
block (file) requires a very large replication factor, and your
remaining # of nodes is less than that value. This is a genuine reason
for hang (but must be fixed). The process usually waits until there
are no under-replicated blocks, so I'd use fsck to check if any such
ones are present and setrep them to a lower value.
On Fri, Aug 12, 2011 at 9:28 PM, <jo...@accenture.com> wrote:
> Hi All,
>
> I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
>
> Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
> <property>
> <!-- 100Mbit/s -->
> <name>dfs.balance.bandwidthPerSec</name>
> <value>131072000</value>
> </property>
>
> Speed didn't seem to pick up. Do you know what may be happening?
>
> Thanks!
> Jonathan
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.
>
--
Harsh J
Re: Speed up node under replicated block during decomission
Posted by sridhar basam <sr...@basam.org>.
On Fri, Aug 12, 2011 at 11:58 AM, <jo...@accenture.com> wrote:
> Hi All,
>
> I'm trying to decommission data node from my cluster. I put the data node
> in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name
> nodes. The under-replicated blocks are starting to replicate, but it's
> going down in a very slow pace. For 1 TB of data it takes over 1 day to
> complete. We change the settings as below and try to increase the
> replication rate.
>
> Added this to hdfs-site.xml on all the nodes on the cluster and restarted
> the data nodes and name node processes.
> <property>
> <!-- 100Mbit/s -->
> <name>dfs.balance.bandwidthPerSec</name>
> <value>131072000</value>
> </property>
>
> Speed didn't seem to pick up. Do you know what may be happening?
>
>
Are you seeing any sort of resource starvation on your data nodes? I/O,
network or CPU?
Sridhar
> Thanks!
> Jonathan
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the email by you is prohibited.
>
Re: Speed up node under replicated block during decomission
Posted by Charles Wimmer <cw...@yahoo-inc.com>.
The balancer bandwidth setting does not affect decommissioning nodes. Decommisssioning nodes replicate as fast as the cluster is capable.
The replication pace has many variables.
Number nodes that are participating in the replication.
The amount of network bandwidth each has.
The amount of other HDFS activity at the time.
Total blocks being replicated.
Total data being replicated.
Many others.
On 8/12/11 8:58 AM, "jonathan.hwang@accenture.com" <jo...@accenture.com> wrote:
Hi All,
I'm trying to decommission data node from my cluster. I put the data node in the /usr/lib/hadoop/conf/dfs.hosts.exclude list and restarted the name nodes. The under-replicated blocks are starting to replicate, but it's going down in a very slow pace. For 1 TB of data it takes over 1 day to complete. We change the settings as below and try to increase the replication rate.
Added this to hdfs-site.xml on all the nodes on the cluster and restarted the data nodes and name node processes.
<property>
<!-- 100Mbit/s -->
<name>dfs.balance.bandwidthPerSec</name>
<value>131072000</value>
</property>
Speed didn't seem to pick up. Do you know what may be happening?
Thanks!
Jonathan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the email by you is prohibited.