[GitHub] [ozone] sodonnel commented on pull request #3482: HDDS-6829. Limit the no of inflight replication tasks in SCM.

[GitBox <gi...@apache.org>]

sodonnel commented on PR #3482:
URL: https://github.com/apache/ozone/pull/3482#issuecomment-1148487383

   I have a couple of thoughts here:
   
   1. Our main concern is pending replications. The replication manager also processes over-replication, as well as handling containers which need to be closed, quasi-closed, unhealthy etc. Putting the limiter where it is in this change, delays all those things from happening too. It will also impact EC containers which are going to go through a new code path and hence will not face the same problem. 
   
   2. There is a replication Manager Report in place, and if we only process some of the containers, the report, which is only updated after each full replication manager run will not have the correct numbers.
   
   3. This solution is a temporary measure until we get the new replication manager ready, and it only really affects things inside the LegacyReplicationManager. I think it would be better if we confined this solution to within the LegacyReplicationManager, and will allow us to develop the new version freely.
   
   I think we would be better placing a limit on the pending in-flight replication tasks, rather than a limit on the number of containers processed. That way the replication manager will still process over replication and all the other health check tasks, but we can skip scheduling a replication for under-replication if there are too many pending already. The report can also be populated fully, with over / under replicated counts, even if all the under replication tasks are not scheduled.
   
   It would also be good to count up how many were skipped on each iteration if possible so it can be logged to give some insight into what is happening. It might be slightly tricky to do this the way things are currently structured, so that would be nice to have.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org