You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Jinglun (Jira)" <ji...@apache.org> on 2020/10/16 07:50:00 UTC
[jira] [Resolved] (HDFS-15636) NameNode computes load by group when
choosing datanodes.
[ https://issues.apache.org/jira/browse/HDFS-15636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jinglun resolved HDFS-15636.
----------------------------
Resolution: Duplicate
Duplicate with HDFS-14383
> NameNode computes load by group when choosing datanodes.
> --------------------------------------------------------
>
> Key: HDFS-15636
> URL: https://issues.apache.org/jira/browse/HDFS-15636
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
>
> We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd datanodes. The HOT files are stored with ALL_SSD and cold ones are stored with HOT. There is a big chance the NameNode couldn't choose enough nodes for writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A temporary solution is to increase the 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the unbalance of load of all the datanodes.
> We should let the NameNode compute load by group. The ssd nodes and hdd nodes are computed separately and each group has its own average load. When the NameNode chooses a hdd node it only compares the node's load with
> the average load of the hdd group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org