You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Baolong Mao (Jira)" <ji...@apache.org> on 2021/01/22 05:32:02 UTC
[jira] [Comment Edited] (HDDS-4656) Add a container balancer tool
or service for HDDS
[ https://issues.apache.org/jira/browse/HDDS-4656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269801#comment-17269801 ]
Baolong Mao edited comment on HDDS-4656 at 1/22/21, 5:32 AM:
-------------------------------------------------------------
[~linyiqun] [~kennethlnnn] [~ljain] Thanks for your review and reply, I'm glad you are interested in this proposal.
> `balancer rate limit is also very important`
Yeah, definitely, will finish that part.
> I think it might also be good to look at pros and cons of having a separate service.
Yeah, there would be lots of pros if we put the balancer into the SCM.
- It would be easier to use and sync with ReplicationManager for balancing the containers.
- If needed the balancing commands can be provided as part of admin command line.
- It can use the rack aware placement policy in SCM to provide default balancing feature.
But it have some cons too.
- It would affect the SCM core logic
- Hard to implement and update the balancer related code.
- Make SCM fat if we put lots features into it while some of these features can be an independent tool.
> let container balancer as an independent program tool like HDFS balancer +1
Thanks for your +1, hope you can try this feature while we finish it.
was (Author: maobaolong):
[~linyiqun] [~kennethlnnn] [~ljain] Thanks for your review and reply, I'm glad you are interested in this proposal.
> `balancer rate limit is also very important`
Yeah, definitely, will finish that part.
> I think it might also be good to look at pros and cons of having a separate service.
Yeah, there would be lots of pros if we put the balancer into the SCM.
- It would be easier to use and sync with ReplicationManager for balancing the containers.
- If needed the balancing commands can be provided as part of admin command line.
- It can use the rack aware placement policy in SCM to provide default balancing feature.
But it have some cons too.
- It would affect the SCM core logic
- Hard to implement and update the balancer related code.
- Make SCM fat if we put lots features into it while some of these features can be an independent tool.
> Add a container balancer tool or service for HDDS
> -------------------------------------------------
>
> Key: HDDS-4656
> URL: https://issues.apache.org/jira/browse/HDDS-4656
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: Ozone Datanode, SCM, Tools
> Affects Versions: 1.1.0
> Reporter: Baolong Mao
> Assignee: Baolong Mao
> Priority: Major
>
> When an existing Ozone cluster is nearly full, we have to add more datanodes into the Ozone cluster, but there are two issue we must face.
> - When new allocate container request coming, SCM should better to choose the datanodes in low usage, if not, the performance will getting pool.
> - For read request, the existing datanodes stored lots of blocks, so they are responsible for serving the read request and supply the data stream service, meanwhile, the new coming datanodes can help nothing.
> If we have a balancer tool just like hdfs balancer, we can move the block or container from some high usage datanodes to low, I think this is one of necessary tools for Ozone.
> container balancer design doc https://docs.google.com/document/d/15PdYaP6aLB18ptbcOK3XWlL4Y1PLfIP-ll30KjNPZ2g/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org