[jira] [Assigned] (HDFS-14383) Compute datanode load based on StoragePolicy

["Ayush Saxena (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ayush Saxena reassigned HDFS-14383:
-----------------------------------

    Assignee: Ayush Saxena

> Compute datanode load based on StoragePolicy
> --------------------------------------------
>
>                 Key: HDFS-14383
>                 URL: https://issues.apache.org/jira/browse/HDFS-14383
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs, namenode
>    Affects Versions: 2.7.3, 3.1.2
>            Reporter: Karthik Palanisamy
>            Assignee: Ayush Saxena
>            Priority: Major
>         Attachments: HDFS-14383-01.patch, HDFS-14383-02.patch
>
>
> Datanode load check logic needs to be changed because existing computation will not consider StoragePolicy.
> DatanodeManager#getInServiceXceiverAverage
> {code}
> public double getInServiceXceiverAverage() {
>  double avgLoad = 0;
>  final int nodes = getNumDatanodesInService();
>  if (nodes != 0) {
>  final int xceivers = heartbeatManager
>  .getInServiceXceiverCount();
>  avgLoad = (double)xceivers/nodes;
>  }
>  return avgLoad;
> }
> {code}
>  
> For example: with 10 nodes (HOT), average 50 xceivers and 90 nodes (COLD) with average 10 xceivers the calculated threshold by the NN is 28 (((500 + 900)/100)*2), which means those 10 nodes (the whole HOT tier) becomes unavailable when the COLD tier nodes are barely in use. Turning this check off helps to mitigate this issue, however the dfs.namenode.replication.considerLoad helps to "balance" the load of the DNs, upon turning it off can lead to situations where specific DNs are "overloaded".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org