[jira] [Created] (YARN-10168) FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct

["Wangda Tan (Jira)" <ji...@apache.org>]

Wangda Tan created YARN-10168:
---------------------------------

             Summary: FS-CS Convert: Converter tool doesn't handle min/max resource conversion correct
                 Key: YARN-10168
                 URL: https://issues.apache.org/jira/browse/YARN-10168
             Project: Hadoop YARN
          Issue Type: Sub-task
            Reporter: Wangda Tan


Trying to understand logics of convert min and max resource from FS to CS, and found some issues:

1)

In FSQueueConverter#emitMaximumCapacity

Existing logic in FS is to either specify a maximum percentage for queues against cluster resources. Or, specify an absolute valued maximum resource.

In the existing FS2CS converter, when a percentage-based maximum resource is specified, the converter takes a global resource from fs2cs CLI, and applies percentages to that. It is not correct since the percentage-based value will get lost, and in the future when cluster resources go up and down, the maximum resource cannot be changed.

2)

The logic to deal with min/weight resource is also questionable:

The existing fs2cs tool, it takes precedence of percentage over absoluteResource, and could set both to a queue config. See FSQueueConverter.Capacity#toString

However, in CS, comparing to FS, the weights/min resource is quite different:

CS use the same queue.capacity to specify both percentage-based or absolute-resource-based configs (Similar to how FS deal with maximum Resource).
 The capacity defines guaranteed resource, which also impact fairshare of the queue. (The more guaranteed resource a queue has, the larger "pie" the queue can get if there's any additional available resource).
 In FS, minResource defined the guaranteed resource, and weight defined how much the pie can grow to.

So to me, in FS, we should pick and choose either weight or minResource to generate CS.

3)

In FS, mix-use of absolute-resource configs (like min/maxResource), and percentage-based (like weight) is allowed. But in CS, it is not allowed. The reason is discussed on YARN-5881, and find [a]Should we support specifying a mix of percentage ...

The existing fs2cs doesn't handle the issue, which could set mixed absolute resource and percentage-based resources.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org