You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yongjun Zhang (Jira)" <ji...@apache.org> on 2023/09/25 21:37:00 UTC

[jira] [Closed] (YUNIKORN-1996) Change a log about queue update failure due to max capacity reached from Warn to Debug

     [ https://issues.apache.org/jira/browse/YUNIKORN-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yongjun Zhang closed YUNIKORN-1996.
-----------------------------------
    Resolution: Invalid

> Change a log about queue update failure due to max capacity reached from Warn to Debug
> --------------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-1996
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-1996
>             Project: Apache YuniKorn
>          Issue Type: Improvement
>          Components: core - scheduler
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>            Priority: Major
>              Labels: pull-request-available
>
> We are seeing similar issue as in YUNIKORN-1985:
> Tons of logs  (62k of them in 3 seconds for the same request) because the max capacity of a queue has reached,
> {code:java}
>        log.Log(log.SchedApplication).Warn("queue update failed unexpectedly",
>          zap.Error(err))
>  {code}
> in
> {code:java}
>  func (sa *Application) tryNode(node *Node, ask *AllocationAsk) *Allocation {
> ...
> // everything OK really allocate
> alloc := NewAllocation(common.GetNewUUID(), node.NodeID, ask)
> if node.AddAllocation(alloc) {
>    if err := sa.queue.IncAllocatedResource(alloc.GetAllocatedResource(), false); err != nil {
>       log.Log(log.SchedApplication).Warn("queue update failed unexpectedly",
>          zap.Error(err))
>       // revert the node update
>       node.RemoveAllocation(alloc.GetUUID())
>       return nil
>    }{code}
> I strongly suspect it’s simply because Yunikorn is trying a lot of nodes again and again, without being aware that the queue capacity exceeded, thus doing unnecessary work (because each try at that time is going to fail due to max capacity reached)
> This certainly would impact Yunikorn’s performance.
> I guess we need to introduce a categories of exceptions (MaxQueueCapReached, RequiredNodeUnavailable etc) that require delay before retry, and let the upper stack to catch the exception, put the allocation into a queue or something similar, and wait for certain period of time before retrying.
> But as a first step, we can just change the log to Debug level. Since the UI provide a way to check how much resource a given queue is used, and whether it's at its max capacity reached, we don't lose too much diagnosis capability after changing the log to Debug.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org