[jira] (YUNIKORN-829) Produce metrics on queue-level resource utilization

["Chenya Zhang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YUNIKORN-829 ]


    Chenya Zhang deleted comment on YUNIKORN-829:
    ---------------------------------------

was (Author: chenya_zhang):
+1

Queue level resource "utilization" is also critical for budget approval on queue capacity for the bigger goal of cost saving.

Admins may not want to give users more resources if they are wasting much. 

It is not practical for admins to check all application-level resource utilization in a queue to make a decision.

 

> Produce metrics on queue-level resource utilization
> ---------------------------------------------------
>
>                 Key: YUNIKORN-829
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-829
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: core - scheduler, shim - kubernetes
>            Reporter: Chaoran Yu
>            Priority: Major
>
> YuniKorn already has metrics on the resources requested/allocated for each queue. But we have no visibility into how much of the allocated resources are actually being used. Take Spark as an example, an under-optimized job may request 1 TB of total executor memory but the actual processing logic only uses 100 GB. This has the consequence that other jobs might not be able to fit in the queue. Having a metric that shows the real utilization will help members of a queue better understand their job characteristics and optimize the jobs.
> K8s metrics server has metrics on real utilization. YK may be able to perform some aggregations to arrive at the stats at the queue level. This is a k8s-specific solution though.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org