[jira] [Created] (HDFS-16613) EC: Improve performance of decommissioning dn with many ec blocks

["caozhiqiang (Jira)" <ji...@apache.org>]

caozhiqiang created HDFS-16613:
----------------------------------

             Summary: EC: Improve performance of decommissioning dn with many ec blocks
                 Key: HDFS-16613
                 URL: https://issues.apache.org/jira/browse/HDFS-16613
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: ec, erasure-coding, namenode
    Affects Versions: 3.4.0
            Reporter: caozhiqiang
            Assignee: caozhiqiang


In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. The reason is unlike replication blocks can be replicated from any dn which has the same block replication, the ec block have to be replicated from the decommissioning dn. The configurations dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit will limit the replication speed, but increase these configurations will create risk to the whole cluster's network. So it should add a new configuration to limit the decommissioning dn, distinguished from the cluster wide max-streams limit.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org