[jira] [Updated] (HDFS-11104) BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance

["Doris Gu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doris Gu updated HDFS-11104:
----------------------------
    Affects Version/s: 3.0.0-alpha1

> BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-11104
>                 URL: https://issues.apache.org/jira/browse/HDFS-11104
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Doris Gu
>            Assignee: Ajith S
>
> if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault prior chooseTarget in turn:
> {quote}
>     DatanodeStorageInfo[] chooseTarget(String src,
>       int numOfReplicas,
>       Node writer,
>       Set<Node> excludedNodes,
>       long blocksize,
>       List<DatanodeDescriptor> favoredNodes,
>       BlockStoragePolicy storagePolicy) {
>     try {
>     ...
>    *for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas; i++)* {
>         DatanodeDescriptor favoredNode = favoredNodes.get(i);
>         // Choose a single node which is local to favoredNode.
>         // 'results' is updated within chooseLocalNode
>         final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
>             favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
>             results, avoidStaleNodes, storageTypes, false);
>       ...
> {quote}
> why not shuffle it here? Make block more balanced, save the cost balancer will pay and make cluster more stable.
> {quote}
> for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()]))) 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org