[jira] [Commented] (HDDS-4260) Balance datanode selection for pipeline creation

["Xiaoyu Yao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198594#comment-17198594 ] 

Xiaoyu Yao commented on HDDS-4260:
----------------------------------

Looks deeper into this. This is more of a rebalancing pipeline issue when the new nodes are added to the cluster. The Scrubber can be a candidate to remove the existing pipelines so that the creator can create new ones on the new nodes. 

The existing PipelinePlacementPolicy#chooseDatanodes() already has logic to consider the existing load on the node(pipeline count). The reason of imbalance is cause by the time the new nodes are added, each node in the cluster already reaches it maximum count node allowed. Pipeline creator can't create new pipeline any more.



> Balance datanode selection for pipeline creation
> ------------------------------------------------
>
>                 Key: HDDS-4260
>                 URL: https://issues.apache.org/jira/browse/HDDS-4260
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>    Affects Versions: 1.0.0
>            Reporter: Glen Geng
>            Assignee: Xiaoyu Yao
>            Priority: Major
>
> Recently [~glengeng] reported an unbalanced node selection when muti-raft is enabled in there cluster deployment. After checking the pipeline placement policy I found that  we chooses node from healthy node one by one if topology aware is not configured. Even if topology aware is configured, the anchor node and the third nodes are supposed to be on the same rack. Without shuffle, it is likely to always get the same set of nodes with the last one being unchosen.
> This ticket is opened to add a  shuffle of the halthyNodes before getResultSet() in
> PipelinePlacementPolicy#chooseDatanodes() to balance the node selection when it is invoked multiple times in the case of multi-raft.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org