You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-issues@hadoop.apache.org by "Wei Song (Jira)" <ji...@apache.org> on 2025/02/28 01:10:00 UTC

[jira] [Created] (HDFS-17743) Support random datanode ordering in getBlockLocations()

Wei Song created HDFS-17743:
-------------------------------

             Summary: Support random datanode ordering in getBlockLocations()
                 Key: HDFS-17743
                 URL: https://issues.apache.org/jira/browse/HDFS-17743
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
    Affects Versions: 3.4.1, 2.10.0
            Reporter: Wei Song


At LinkedIn, we aren't able to rely on data locality due to various reasons, overall the current implementation of sorting by network topology didn't improve overall performance, it also caused unnecessary concentration of load on specific datanodes. We would like to request for a random policy that randomly order datanodes, we expect this policy to improve load distribution. 

 

The new policy should be configurable, and it is disabled. When enabled, it replaces the current network topology based datanode ordering.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org