[jira] [Created] (HDFS-11684) Potential scalability bug on datanode recommission

["Riza Suminto (JIRA)" <ji...@apache.org>]

Riza Suminto created HDFS-11684:
-----------------------------------

             Summary: Potential scalability bug on datanode recommission
                 Key: HDFS-11684
                 URL: https://issues.apache.org/jira/browse/HDFS-11684
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 2.7.3
            Reporter: Riza Suminto


We are academic researcher working on scalability bug detection.

Upon analyzing Hadoop code, our static analysis program detect
potential scalability problem along this function calls:

DatanodeManager.refreshNodes() -> DatanodeManager.refreshDatanodes() -> DecommissionManager.stopDecommission() -> BlockManager.processExtraRedundancyBlocksOnInService()

DatanodeManager.refreshNodes() call namesystem.writeLock(),
which will lock FSNamesystem. While locking FSNamesystem, it does
nested loop over datanodes in DatanodeManager.refreshDatanodes() and
over datanode blocks in BlockManager.processExtraRedundancyBlocksOnInService().

Looking only at number of executed lines, refreshDatanodes will
execute N*(136*B+137) lines of codes in total, where N is number of
datanodes and B is number of blocks per node.

This is a potential scalability bug that may render FSNamesystem
unresponsive for long time if refreshNodes trigger recommissioning of
many fat datanodes, having large number of over replicated datablocks.

This bug also discussed in HDFS-10477, but still left unresolved.

In older hadoop version, our program also detect similar scalability
problem in decommissioning datanodes, which has been fixed on
HADOOP-4061. However, recommissioning still have scalabillity problem
in current version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org