[jira] [Resolved] (HDFS-17538) Add tranfer priority queue for decommissioning datanode

["Yuanbo Liu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-17538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuanbo Liu resolved HDFS-17538.
-------------------------------
    Resolution: Duplicate

> Add tranfer priority queue for decommissioning datanode
> -------------------------------------------------------
>
>                 Key: HDFS-17538
>                 URL: https://issues.apache.org/jira/browse/HDFS-17538
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Yuanbo Liu
>            Priority: Major
>         Attachments: image-2024-05-29-16-24-45-601.png, image-2024-05-29-16-26-58-359.png, image-2024-05-29-16-27-35-886.png
>
>
> When decommissioning datanode, blocks will be checked one by one disk, then blocks will be sent to trigger tranfer works in DN. This will make one disk of decommissioning dn very busy and cpus stuck in io-wait with high loads, and sometime even lead to OOM as below:
> !image-2024-05-29-16-24-45-601.png|width=909,height=170!
> !image-2024-05-29-16-26-58-359.png|width=909,height=228!
> !image-2024-05-29-16-27-35-886.png|width=930,height=218!
> Proposal to add priority queue for transfering blocks when decommisioning datanode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org