You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yu-Lin Chen (Jira)" <ji...@apache.org> on 2023/12/19 09:00:00 UTC
[jira] [Comment Edited] (YUNIKORN-2257) Add rest API to retrieve node utilization for multiple resource types
[ https://issues.apache.org/jira/browse/YUNIKORN-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798496#comment-17798496 ]
Yu-Lin Chen edited comment on YUNIKORN-2257 at 12/19/23 8:59 AM:
-----------------------------------------------------------------
Update Jira description based on the Slack/PR discussion result.
Also copy the suggestion from Wilfred Spiegelenburg in Slack:
{code:java}
Wilfred Spiegelenburg 10 hours ago:
We do need to be careful with changing the response on an existing API. This REST call had been deprecated once before already.
If someone has started using the current version based on the YuniKorn 1.4 release we have to go through the cycle of add a new one.
The implementation for this new multi resource type will end up being completely different from what we have now. We will also break the current WEB UI if we change the REST response until we get the new graphs in.
When I looked yesterday I saw that the current reused implementation for the new form leads to a really inefficient implementation of the new form. I asked Mani to revert the commit until that implementation is fixed:
It triggers a storm of calls to CalculateAbsUsedCapacity Nodes have around 10 resource types in normal cases. It gets called for each resource type separately while it should only be called once per node. The function calculates the absolute used capacity for all resource types in one go. As an example for a 5000 node cluster with 10 resource types per node the overhead is an unneeded 45000 calls to CalculateAbsUsedCapacity . Even if they are fast that is not acceptable.
Combine all these points and it is better to add a new end point: node-utilizations (plural form) {code}
was (Author: yu-lin chen):
Update Jira description based on the Slack/PR discussion result.
Also copy the suggestion from Wilfred Spiegelenburg :
{code:java}
Wilfred Spiegelenburg 10 hours ago:
We do need to be careful with changing the response on an existing API. This REST call had been deprecated once before already.
If someone has started using the current version based on the YuniKorn 1.4 release we have to go through the cycle of add a new one.
The implementation for this new multi resource type will end up being completely different from what we have now. We will also break the current WEB UI if we change the REST response until we get the new graphs in.
When I looked yesterday I saw that the current reused implementation for the new form leads to a really inefficient implementation of the new form. I asked Mani to revert the commit until that implementation is fixed:
It triggers a storm of calls to CalculateAbsUsedCapacity Nodes have around 10 resource types in normal cases. It gets called for each resource type separately while it should only be called once per node. The function calculates the absolute used capacity for all resource types in one go. As an example for a 5000 node cluster with 10 resource types per node the overhead is an unneeded 45000 calls to CalculateAbsUsedCapacity . Even if they are fast that is not acceptable.
Combine all these points and it is better to add a new end point: node-utilizations (plural form) {code}
> Add rest API to retrieve node utilization for multiple resource types
> ---------------------------------------------------------------------
>
> Key: YUNIKORN-2257
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2257
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: core - common
> Reporter: Yu-Lin Chen
> Assignee: Yu-Lin Chen
> Priority: Major
> Labels: pull-request-available, release-notes
>
> The current 'Nodes Resource Utilization' bar chart in the UI only shows domaint resource. To allow user to identify the top overloaded resources type. We need a new api that returns node utlization for multiple partititions and multiple resource types.
> The previous API introduced in YUNIKORN-2087 will be marked as deprecated, and we will calculate domaint reosurces from the new api's result.
> {code:java}
> [
> {
> "partition": "default",
> "nodeUtilizationList" : []*dao.NodesUtilDAOInfo
> },
> {
> "partition": "other-partition",
> "nodeUtilizationList" : []*dao.NodesUtilDAOInfo
> }
> ] {code}
> {*}API to be d{*}{*}eprecated{*}{*}{{*}}:
> * /ws/v1/scheduler/node-utilization
> *New API:* (Return all partition and all type of resources)
> * /ws/v1/scheduler/node-utilizations (plural form)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org