You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Mathias Herberts (JIRA)" <ji...@apache.org> on 2011/01/21 16:59:48 UTC
[jira] Created: (HDFS-1590) Decommissioning never ends when node to
decommission has blocks that are under-replicated and cannot be replicated
to the expected level of replication
Decommissioning never ends when node to decommission has blocks that are under-replicated and cannot be replicated to the expected level of replication
-------------------------------------------------------------------------------------------------------------------------------------------------------
Key: HDFS-1590
URL: https://issues.apache.org/jira/browse/HDFS-1590
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.2
Environment: Linux
Reporter: Mathias Herberts
Priority: Minor
On a test cluster with 4 DNs and a default repl level of 3, I recently attempted to decommission one of the DNs. Right after the modification of the dfs.hosts.exclude file and the 'dfsadmin -refreshNodes', I could see the blocks being replicated to other nodes.
After a while, the replication stopped but the node was not marked as decommissioned.
When running an 'fsck -files -blocks -locations' I saw that all files had a replication of 4 (which is logical given there are 4 DNs), but some of the files had an expected replication set to 10 (those were job.jar files from M/R jobs).
I ran 'fs -setrep 3' on those files and shortly after the namenode reported the DN as decommissioned.
Shouldn't this case be checked by the NameNode when decommissioning a node? I.e considere a node decommissioned if either one of the following is true for each block on the node being decommissioned:
1. It is replicated more than the expected replication level.
2. It is replicated as much as possible given the available nodes, even though it is less replicated than expected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-1590) Decommissioning never ends when node
to decommission has blocks that are under-replicated and cannot be
replicated to the expected level of replication
Posted by "Jim Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458987#comment-13458987 ]
Jim Huang commented on HDFS-1590:
---------------------------------
If the NN is able to complete the replication of all the blocks of the decommissioning DN to other participating DNs, then that should satisfy the decommission operation. NameNode itself already keeps track of the underplicated blocks, so it should not hold back this node from finishing decommissioning and putting it to dead node list. If I follow this logic, the most extrem case will be when there are only 1 DN remaining and you want to decommission that. That doesn't make sense to me, because it will render HDFS unusable, might as well, put the cluster into safemode and shut it down.
This scenario is mostly encountered in development and QA environments where the cluster footprint is really small and tiny. So the proposed patch should not be so drastic that it undermines the stability of the existing HDFS code base.
> Decommissioning never ends when node to decommission has blocks that are under-replicated and cannot be replicated to the expected level of replication
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1590
> URL: https://issues.apache.org/jira/browse/HDFS-1590
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.2
> Environment: Linux
> Reporter: Mathias Herberts
> Assignee: Harsh J
> Priority: Minor
>
> On a test cluster with 4 DNs and a default repl level of 3, I recently attempted to decommission one of the DNs. Right after the modification of the dfs.hosts.exclude file and the 'dfsadmin -refreshNodes', I could see the blocks being replicated to other nodes.
> After a while, the replication stopped but the node was not marked as decommissioned.
> When running an 'fsck -files -blocks -locations' I saw that all files had a replication of 4 (which is logical given there are 4 DNs), but some of the files had an expected replication set to 10 (those were job.jar files from M/R jobs).
> I ran 'fs -setrep 3' on those files and shortly after the namenode reported the DN as decommissioned.
> Shouldn't this case be checked by the NameNode when decommissioning a node? I.e considere a node decommissioned if either one of the following is true for each block on the node being decommissioned:
> 1. It is replicated more than the expected replication level.
> 2. It is replicated as much as possible given the available nodes, even though it is less replicated than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1590) Decommissioning never ends when node
to decommission has blocks that are under-replicated and cannot be
replicated to the expected level of replication
Posted by "Harsh J Chouraria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J Chouraria reassigned HDFS-1590:
---------------------------------------
Assignee: Harsh J Chouraria
> Decommissioning never ends when node to decommission has blocks that are under-replicated and cannot be replicated to the expected level of replication
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1590
> URL: https://issues.apache.org/jira/browse/HDFS-1590
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.2
> Environment: Linux
> Reporter: Mathias Herberts
> Assignee: Harsh J Chouraria
> Priority: Minor
>
> On a test cluster with 4 DNs and a default repl level of 3, I recently attempted to decommission one of the DNs. Right after the modification of the dfs.hosts.exclude file and the 'dfsadmin -refreshNodes', I could see the blocks being replicated to other nodes.
> After a while, the replication stopped but the node was not marked as decommissioned.
> When running an 'fsck -files -blocks -locations' I saw that all files had a replication of 4 (which is logical given there are 4 DNs), but some of the files had an expected replication set to 10 (those were job.jar files from M/R jobs).
> I ran 'fs -setrep 3' on those files and shortly after the namenode reported the DN as decommissioned.
> Shouldn't this case be checked by the NameNode when decommissioning a node? I.e considere a node decommissioned if either one of the following is true for each block on the node being decommissioned:
> 1. It is replicated more than the expected replication level.
> 2. It is replicated as much as possible given the available nodes, even though it is less replicated than expected.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1590) Decommissioning never ends when node
to decommission has blocks that are under-replicated and cannot be
replicated to the expected level of replication
Posted by "Harsh J Chouraria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030373#comment-13030373 ]
Harsh J Chouraria commented on HDFS-1590:
-----------------------------------------
I hit this as well a couple of weeks ago. Would be nice to have it fixed.
What could be considered as a "safe replica #s available" level? A min. of 3? A formula based on rounded-off average replication factor? Your ideas please! :)
> Decommissioning never ends when node to decommission has blocks that are under-replicated and cannot be replicated to the expected level of replication
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1590
> URL: https://issues.apache.org/jira/browse/HDFS-1590
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.2
> Environment: Linux
> Reporter: Mathias Herberts
> Priority: Minor
>
> On a test cluster with 4 DNs and a default repl level of 3, I recently attempted to decommission one of the DNs. Right after the modification of the dfs.hosts.exclude file and the 'dfsadmin -refreshNodes', I could see the blocks being replicated to other nodes.
> After a while, the replication stopped but the node was not marked as decommissioned.
> When running an 'fsck -files -blocks -locations' I saw that all files had a replication of 4 (which is logical given there are 4 DNs), but some of the files had an expected replication set to 10 (those were job.jar files from M/R jobs).
> I ran 'fs -setrep 3' on those files and shortly after the namenode reported the DN as decommissioned.
> Shouldn't this case be checked by the NameNode when decommissioning a node? I.e considere a node decommissioned if either one of the following is true for each block on the node being decommissioned:
> 1. It is replicated more than the expected replication level.
> 2. It is replicated as much as possible given the available nodes, even though it is less replicated than expected.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1590) Decommissioning never ends when node
to decommission has blocks that are under-replicated and cannot be
replicated to the expected level of replication
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458926#comment-13458926 ]
Luke Lu commented on HDFS-1590:
-------------------------------
This has become a FEI (Frequently Encountered Issue) for new dev/QAs :)
Maybe we should introduce a dfsadmin -decommission command that handles all these logic with a --force option to automatically setrep #remaining-nodes on the files with replications > #remaining-nodes and a --force-data-loss option to immediately remove nodes (mostly for testing purpose.)
> Decommissioning never ends when node to decommission has blocks that are under-replicated and cannot be replicated to the expected level of replication
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1590
> URL: https://issues.apache.org/jira/browse/HDFS-1590
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.2
> Environment: Linux
> Reporter: Mathias Herberts
> Assignee: Harsh J
> Priority: Minor
>
> On a test cluster with 4 DNs and a default repl level of 3, I recently attempted to decommission one of the DNs. Right after the modification of the dfs.hosts.exclude file and the 'dfsadmin -refreshNodes', I could see the blocks being replicated to other nodes.
> After a while, the replication stopped but the node was not marked as decommissioned.
> When running an 'fsck -files -blocks -locations' I saw that all files had a replication of 4 (which is logical given there are 4 DNs), but some of the files had an expected replication set to 10 (those were job.jar files from M/R jobs).
> I ran 'fs -setrep 3' on those files and shortly after the namenode reported the DN as decommissioned.
> Shouldn't this case be checked by the NameNode when decommissioning a node? I.e considere a node decommissioned if either one of the following is true for each block on the node being decommissioned:
> 1. It is replicated more than the expected replication level.
> 2. It is replicated as much as possible given the available nodes, even though it is less replicated than expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira