[jira] [Created] (HDDS-4062) Non rack aware pipelines should not be created if multiple racks are alive

["Stephen O'Donnell (Jira)" <ji...@apache.org>]

Stephen O'Donnell created HDDS-4062:
---------------------------------------

             Summary: Non rack aware pipelines should not be created if multiple racks are alive
                 Key: HDDS-4062
                 URL: https://issues.apache.org/jira/browse/HDDS-4062
             Project: Hadoop Distributed Data Store
          Issue Type: Improvement
          Components: SCM
    Affects Versions: 0.7.0
            Reporter: Stephen O'Donnell
            Assignee: Stephen O'Donnell


If we have a scenario where one rack has more nodes that others, it is possible for all hosts in the cluster to have reached their pipeline limit, while 3 nodes on the larger rack have not. 

The current fallback logic will then allow a pipeline to be created which uses only the 3 nodes on the same rack, violating the rack placement policy.

There may be other ways this could happen with cluster load too, were the pipeline capacity has reached its limit on some nodes but not others.

The proposal here, is that if the cluster has multiple racks AND there are healthy nodes covering at least 2 racks, where healthy is defined as a node which is registered and not stale or dead, then we should not allow "fallback" (pipelines which span only 1 rack) pipelines to be created.

This means if you have a badly configured cluster - eg Rack 1 = 10 nodes; Rack 2 = 1 node, the pipeline limit will be constrained by the capacity of that 1 node on rack 2. Even a setup like Rack 1 = 10 nodes, Rack 2 = 5 would be constrained by this.

This constraint is better than creating non rack aware pipelines, and the rule above will handle the case when the cluster degrades to 1 rack, as the healthy node definition will notice only 1 rack is alive.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org