[jira] [Created] (HDFS-15636) NameNode computes load by group when choosing datanodes.

["Jinglun (Jira)" <ji...@apache.org>]

Jinglun created HDFS-15636:
------------------------------

             Summary: NameNode computes load by group when choosing datanodes.
                 Key: HDFS-15636
                 URL: https://issues.apache.org/jira/browse/HDFS-15636
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Jinglun


We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd datanodes. The HOT files are stored with ALL_SSD and cold ones are stored with HOT. There is a big chance the NameNode couldn't choose enough nodes for writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A temporary solution is  to increase the 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the unbalance of load of all the datanodes.
We should let the NameNode compute load by group. The ssd nodes and hdd nodes are computed separately and each group has its own average load. When the NameNode chooses a hdd node it only compares the node's load with
 the average load of the hdd group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org