[jira] Issue Comment Edited: (HADOOP-1762) Namenode does not need to store storageID and datanodeID persistently

["Konstantin Shvachko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526648 ] 

shv edited comment on HADOOP-1762 at 9/11/07 6:15 PM:
----------------------------------------------------------------------

Mostly cosmetic comments:
# Datanode.setNewStorageID():
{code}    } catch (UnknownHostException ignored) {} {code}
Should not be ignored but rather logged.
# We should  deprecate  FSEdits.OP_DATANODE_ADD and  FSEdits.OP_DATANODE_REMOVE
so that we could eventually remove them sometimes.
# FSNamesystem.registerDatanode():
{code}
        NameNode.stateChangeLog.info( "BLOCK* NameSystem.registerDatanode: "
                                      + "node " + nodeS.getName()
                                      + " is replaced by " + nodeReg.getName() +
                                      " since they have the same storageID " +
                                      nodeReg.getStorageID() +
                                      ". If this message repeats, both nodes " +
                                      "might have same storageID by random " +
                                      "chance. You need to restart one of the " +
                                      "nodes with its data cleared. ");
{code}
This message may cause confusion in the regular case.
The possibility of storage id collision is so negligible that we should not hint on it every time one node *_legally_* replaces another with the same storage id.
Let's retain the original redaction.

I also reviewed the first variant of the patch. They are very similar. The main difference is who generates the new storage id 
for the new data-node. And I prefer the case when data-nodes are able to independently generate unique storage ids for themselves.

The ids here are unique since:
- data-nodes started on different nodes have different ips;
- data-nodes started on the same node (but in different storage directories) at the same time will have different ports;
- data-nodes started on the same node on the same port but at different times will have different timestamps;
- in the latter case even somebody turns time back before starting the second node the random number is likely to prevent the collision.

B.t.w. something like that should be in the comments for Datanode.setNewStorageID().

      was (Author: shv):
    Mostly cosmetic comments:
# Datanode.setNewStorageID():
{code}    } catch (UnknownHostException ignored) {} {code}
Should not be ignored but rather logged.
# We should  deprecate  FSEdits.OP_DATANODE_ADD and  FSEdits.OP_DATANODE_REMOVE
so that we could eventually remove them sometimes.
# FSNamesystem.registerDatanode():
{code}
        NameNode.stateChangeLog.info( "BLOCK* NameSystem.registerDatanode: "
                                      + "node " + nodeS.getName()
                                      + " is replaced by " + nodeReg.getName() +
                                      " since they have the same storageID " +
                                      nodeReg.getStorageID() +
                                      ". If this message repeats, both nodes " +
                                      "might have same storageID by random " +
                                      "chance. You need to restart one of the " +
                                      "nodes with its data cleared. ");
{code}
This message may cause confusion in the regular case.
The possibility of storage id collision is so negligible that we should not hint on it every time one node *_legally_* replaces another with the same storage id.
Let's retain the original redaction.


  
> Namenode does not need to store storageID and datanodeID persistently
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-1762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1762
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>         Attachments: HADOOP-1762-Opt2.patch, HADOOP-1762-Opt2.patch, HADOOP-1762-Opt2.patch, HADOOP-1762-Opt2.patch, HADOOP-1762.patch
>
>
> Currently Namenode stores all the storage-ids it generates since the beginning (since last format). It allocates a new storageID everytime a new datanode comes online. It also stores all the known datanode ids since the beginning. 
> It would be better if Namenode did not have to keep track of these. I will describe a proposal in the next comment. 
> This has implecations regd how Namenode helps administrators identify 'dead datanodes' etc. These issues are addressed in HADOOP-1138.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.