[jira] [Updated] (HDDS-5413) Limit num of containers to process per round for ReplicationManager.

["Mark Gui (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Gui updated HDDS-5413:
---------------------------
        Parent: HDDS-5657
    Issue Type: Sub-task  (was: Improvement)

> Limit num of containers to process per round for ReplicationManager.
> --------------------------------------------------------------------
>
>                 Key: HDDS-5413
>                 URL: https://issues.apache.org/jira/browse/HDDS-5413
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Mark Gui
>            Assignee: Mark Gui
>            Priority: Major
>              Labels: pull-request-available
>
> For now, ReplicationManager process all containers at once, this will potentially bring healthy load to datanodes if there are a lot of containers to be replicated/deleted/closed.
> So it is nice to have a bound for each round, HDFS has similar settings, and this issue tries to implement sth like 'dfs.block.misreplication.processing.limit: 10000' in HDFS.
>  
> This is just a limit on the number of containers to be processed, note that ReplicationManager count each container as processed no mater it is under replicated or over replicated or good. And all the cmds are queued and will be sent with heartbeats.
> So this limit does not directly limit the data to be replicated, just to have a basic throttling for ReplicationManager in-memory processing, so 10000 should be good for ozone as well.
> And we should add more throttling limits in the future.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org