You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Colin Patrick McCabe (JIRA)" <ji...@apache.org> on 2015/03/20 00:24:38 UTC

[jira] [Updated] (HDFS-7960) The full block report should contain a "numStoragesOnDn" field that allows the NameNode to prune old storages

     [ https://issues.apache.org/jira/browse/HDFS-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-7960:
---------------------------------------
         Description: 
The full block report should contain a "numStoragesOnDn" field that allows the NameNode to prune old storages.  Currently, the NameNode can't do this pruning because it doesn't know how many storages are on the DataNode, and the full block report can be split into multiple RPCs.

While it is true that storages will be removed by HDFS-7596 when they are empty, this mechanism doesn't work for datanode hotplug.  In the case of datanode hotplug, an entire storage simply goes away because of reconfiguration.  Currently, the NN may never perceive this "zombie storage" as being empty, unless the NN itself is restarted.

  was:NameNode should be able to remove storages that is not represented on DataNode. For example, hot swapped on that DataNode or DN restarted with less data dirs.  It seems that once a datanode storage is removed from a datanode, those blocks on the storage will not be reconciled as gone from the Namenode until the namenode has been restarted.

            Priority: Critical  (was: Major)
    Target Version/s: 2.7.0  (was: 3.0.0)
             Summary: The full block report should contain a "numStoragesOnDn" field that allows the NameNode to prune old storages  (was: NameNode should prune storages that are no longer existed on DataNode)

> The full block report should contain a "numStoragesOnDn" field that allows the NameNode to prune old storages
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7960
>                 URL: https://issues.apache.org/jira/browse/HDFS-7960
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Colin Patrick McCabe
>            Priority: Critical
>
> The full block report should contain a "numStoragesOnDn" field that allows the NameNode to prune old storages.  Currently, the NameNode can't do this pruning because it doesn't know how many storages are on the DataNode, and the full block report can be split into multiple RPCs.
> While it is true that storages will be removed by HDFS-7596 when they are empty, this mechanism doesn't work for datanode hotplug.  In the case of datanode hotplug, an entire storage simply goes away because of reconfiguration.  Currently, the NN may never perceive this "zombie storage" as being empty, unless the NN itself is restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)