[GitHub] [ozone] JacksonYao287 commented on pull request #3963: HDDS-6572. EC: ReplicationManager - add move manager for container move

[GitBox <gi...@apache.org>]

JacksonYao287 commented on PR #3963:
URL: https://github.com/apache/ozone/pull/3963#issuecomment-1318032210

   @siddhantsangwan thanks for the comments.
   IMHO, container balancer, as a tool of optimizing the data distribution of the whole cluster, should be run when the cluster is not busy with workload. although we can run it anytime, but running a balancer in a busy cluster is not recommended. if we have to run it at busy time, we can set some parameters(for example, size.moved.max.per.iteration) to limit the influence it will bring to the busy cluster. so in majority of cases, for a good cluster administrator, running the balancer + replication manager together will not be a big problem.
   
   >What limits to we have on the balancer to stop too many commands getting scheduled on an under-used datanode? 
   
   seems we do not have any limit about this for now. we can add a configure item, maybe `max.replication.command.sent.to.one.target.per.iteration`, to limit the count of replicaiton command sent to a single target datanode in one iteration.  
   
   >I understand that is needed, but I wonder if we could re-use the replication manager over-replicated-processor to do this instead. 
   
   i get your point here that we`d better let all the command sent from a single place, and this will also share the traffic control of RM.
   there are two problems i can think up for now:
   1 RM is a background service and run periodically , but movemanger will take action as soon as it is notified some op is commpleted. i am not quite sure if RM is sleeping, is it correct to call `ReplicationManager.processOverReplicated(...)` 
   
   2 in majority cases, if a replication command is completed, the container is overreplicated and movemanager will delete the replica on source datanode. but for RM, it does not konw whether the container is being moved. when RM found this container is overreplicated, it may delete the replica in target datanode, which is not we want.
   
   
   >I also had a quick scan of the balancer config - is it not topology aware by default?
   for now, we have two `FindTargetStrategy`, `FindTargetStrategyByUsageInfo` and `FindTargetStrategyByNetworkTopology`. in some case , when choosing a target , the two strategy may conflict. for example , a datanode may have a lower disk usage, but have a farer distance from a given source datanode. for now , we use
    `FindTargetStrategyByUsageInfo` by default. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org