[jira] [Created] (HDDS-5327) EC: WritableEcContainerProvider should dynamically adjust the open container groups

["Stephen O'Donnell (Jira)" <ji...@apache.org>]

Stephen O'Donnell created HDDS-5327:
---------------------------------------

             Summary: EC: WritableEcContainerProvider should dynamically adjust the open container groups
                 Key: HDDS-5327
                 URL: https://issues.apache.org/jira/browse/HDDS-5327
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: SCM
            Reporter: Stephen O'Donnell


After some discussion we concluded that for any given EC policy, a minimum number of pipelines should be allocated so writes can be directed to different containers.

The absolute maximum number of pipelines can be calculated as some function of the cluster nodes and disks, but we are still unsure about how to calculate that limit.

The number of pipelines should be able to grow from the minimum toward the maximum, depending on the write load on the cluster, or perhaps the age of the oldest open container (to prevent too many containers remaining open for too long).

The goal is to allow a sufficient number of open containers so the writes are spread across different disks, without having to maintain too many open containers on the cluster.

If the write load is very high, there should be more open containers than if the write load is very low.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org