[jira] [Created] (HDFS-5505) DataStorage.format() should ignore NamespaceID

["Eric Sirianni (JIRA)" <ji...@apache.org>]

Eric Sirianni created HDFS-5505:
-----------------------------------

             Summary: DataStorage.format() should ignore NamespaceID
                 Key: HDFS-5505
                 URL: https://issues.apache.org/jira/browse/HDFS-5505
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, federation
    Affects Versions: 2.2.0
            Reporter: Eric Sirianni
            Priority: Minor


I'm observing inconsistencies in the {{DataStorage.namespaceID}} value passed in to my {{FsDatasetSpi}} plugin:
* Upon _initial_ startup (when the DataNode is formatted), the storage {{DataStorage.namespaceID}} value is populated with the namespace ID from the NameNode (or an arbitrary namenode in a federated setup).
* Upon subsequent startups the {{DataStorage.namespaceID}} value is {{NULL}}.

The issue appears to be different handling of the {{DataStorage.namespaceID}} value in {{DataStorage.format()}} vs. {{DataStorage.setFieldsFromProperties()}}:

{code:title=DataStorage.java}
  void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid) throws IOException {
    sd.clearDirectory(); // create directory
    this.layoutVersion = HdfsConstants.LAYOUT_VERSION;
    this.clusterID = nsInfo.getClusterID();
    this.namespaceID = nsInfo.getNamespaceID();
  }
....

  protected void setFieldsFromProperties(Properties props, StorageDirectory sd)
    // Read NamespaceID in version before federation
    if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
      setNamespaceID(props, sd);
    }
{code}

If I understand correctly, the proper fix is to leave namespaceID *unset* in {{DataStorage.format()}}.  The namespaceID here is invalid anyway since it is susceptible to race conditions in federated NameNode setups -- it will happen to match the namespaceID of the first {{BPOfferService}} thread that successfully handshakes.




--
This message was sent by Atlassian JIRA
(v6.1#6144)