[jira] [Commented] (YUNIKORN-2030) Check Headroom checking doesn't prevent failure to allocate resource due to max resource limit exceeded

["Yongjun Zhang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YUNIKORN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774733#comment-17774733 ] 

Yongjun Zhang commented on YUNIKORN-2030:
-----------------------------------------

It looks to me the problem might be, when scheduler does allocation for an application, it only acquires application lock, not the queue lock:
{code:java}
// tryAllocate will perform a regular allocation of a pending request, includes placeholders.
func (sa *Application) tryAllocate(headRoom *resources.Resource, preemptionDelay time.Duration, preemptAttemptsRemaining *int, nodeIterator func() NodeIterator, fullNodeIterator func() NodeIterator, getNodeFn func(string) *Node) *Allocation {
  sa.Lock()
  defer sa.Unlock()
  ...{code}
 

If scheduler concurrently schedule for multiple applications, then it could update the queue capacity when doing allocation for one app, and the allocation for the other app may fail with the symptom reported in this jira.

The question is, does the scheduler do allocation for multiple allocation concurrently?

> Check Headroom checking doesn't prevent failure to allocate resource due to max resource limit exceeded
> -------------------------------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-2030
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-2030
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>            Priority: Major
>
> As reported in YUNIKORN-1996, we are seeing many messages like below from time to time:
> {code:java}
>  WARN    objects/application.go:1504     queue update failed unexpectedly        {“error”: “allocation (map[memory:37580963840 pods:1 vcore:2000]) puts queue ‘root.test-queue’ over maximum allocation (map[memory:3300011278336 vcore:390584]), current usage (map[memory:3291983380480 pods:91 vcore:186000])“}{code}
> Restarting Yunikorn helps stoppinging it. Creating this Jira to investigate why it happened, because it's not supposed to happen as we check if there is enough resource headroom before calling 
>  
> {code:java}
> func (sa *Application) tryNode(node *Node, ask *AllocationAsk) *Allocation 
> {code}
> which printed the above message, and only call it when there is enough headroom.
> There maybe a bug in headroom checking?
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org