You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yongjun Zhang (Jira)" <ji...@apache.org> on 2023/09/15 04:02:00 UTC
[jira] [Commented] (YUNIKORN-1985) possible log spew in application object in tryAllocate()
[ https://issues.apache.org/jira/browse/YUNIKORN-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765435#comment-17765435 ]
Yongjun Zhang commented on YUNIKORN-1985:
-----------------------------------------
We are seeing similar issue:
Tons of logs (62k of them in 3 seconds for the same request) because the max capacity of a queue has reached,
{code:java}
log.Log(log.SchedApplication).Warn("queue update failed unexpectedly",
zap.Error(err))
{code}
in
{code:java}
func (sa *Application) tryNode(node *Node, ask *AllocationAsk) *Allocation {
...
// everything OK really allocate
alloc := NewAllocation(common.GetNewUUID(), node.NodeID, ask)
if node.AddAllocation(alloc) {
if err := sa.queue.IncAllocatedResource(alloc.GetAllocatedResource(), false); err != nil {
log.Log(log.SchedApplication).Warn("queue update failed unexpectedly",
zap.Error(err))
// revert the node update
node.RemoveAllocation(alloc.GetUUID())
return nil
}{code}
I strongly suspect it’s simply because Yunikorn is trying a lot of nodes again and again, without being aware that the queue capacity exceeded, thus doing unnecessary work (because each try at that time is going to fail due to max capacity reached)
This certainly would impact Yunikorn’s performance.
I guess we need o introduce a categories of exceptions (MaxQueueCapReached, RequiredNodeUnavailable etc) that require delay before retry, and let the upper stack to catch the exception, put the allocation into a queue or something similar, and wait for certain period of time before retrying.
Thanks.
> possible log spew in application object in tryAllocate()
> --------------------------------------------------------
>
> Key: YUNIKORN-1985
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1985
> Project: Apache YuniKorn
> Issue Type: Improvement
> Components: core - scheduler
> Reporter: Wilfred Spiegelenburg
> Assignee: Kuan-Po Tseng
> Priority: Minor
> Labels: newbie
>
> If a pod has a required node and cannot be scheduled we keep on attempting the pod placement and cause a log spew:
> {code:java}
> 2023-06-15T10:42:43.546Z WARN objects/application.go:993 required node is not found (could be transient) {"application ID": "yunikorn-ag-ecp12s-1-autogen", "allocationKey": "b51b0a36-214a-4a12-a285-d4319f4b5254", "required node": "worker-31.XXXX"} {code}
> This can happen if a daemonset pod is created and the node is not in the right state.
> We should move this to debug or have a rate limit on how often we log this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org