You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yongjun Zhang (Jira)" <ji...@apache.org> on 2024/02/07 23:39:00 UTC
[jira] [Commented] (YUNIKORN-2030) Need to check headroom when trying other nodes for reserved allocations
[ https://issues.apache.org/jira/browse/YUNIKORN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815456#comment-17815456 ]
Yongjun Zhang commented on YUNIKORN-2030:
-----------------------------------------
I added a bunch of diag logs and finally got a reproduction of the issue, and the logs helped to identify a missing headRoom check which I'm providing a PR.
With the two stack traces attached below, I did the following analysis to identify the bug and fix:
# stackTraceA and stackTraceB, based on the code, they should from different scheduling cycle, with time gap is 0.001 second, for different ask
# The passed headRoom to stacktraceA is 27511488512, the allocation request to stacckTraceA is 23394779136, thus after this allocation, the resulted headRoom is 4116709376, which is exactly the headRoom passed to and seen by stackTraceB. This means these two staclTraces are neighboring scheduling cycles.
# with headRoom 27511488512, the allocation request of stackTraceA of 23394779136 can be satisfied, however, with the resulted headRoom 4116709376 from stackTraceA, stackTraceB failed with the reproduction symptom.
# I suspect there may be a headRoom checking bug, because the resource request of stackTraceB is 23394779136, which is larger than the resulted headRoom from stackTraceA ({*}4116709376{*} reported by stackTraceB): given *23394779136* is larger than *4116709376, how the earlier headRoom checking can pass, so that this illegal state is entered.* But the headRoom checking code seems straightforward.
# This made me to suspect that headRoom checking was not done for this allocation, and I found that place. The PR shows where the headRoom checking is missing.
stackTraceA:
{code:java}
3845 2024-02-03T04:48:58.833Z WARN objects/queue.go:1070 Potential allocation causing inconsistent state A {"allockey": "d622ab3f-d9ed-44ec-892d-329e8a6f6963", "application": "a28b08df-galaxypinj-runner-pin-manual--2 0-1", "headRoom": {"attachable-volumes-aws-ebs":23425,"ephemeral-storage":55532238435958,"hugepages-1Gi":0,"hugepages-2Mi":0,"hugepages-32Mi":0,"hugepages-64Ki":0,"memory":27511488512,"pods":48682,"vcore":109976}, "leafQueue": "r oot.org-core.homefeed-infra-workflow.tier2", "allocationRequest": "map[memory:23394779136 pods:1 vcore:2000]", "queueUsage": "map[memory:3301213995008 pods:142 vcore:284000]", "maxResource": "map[memory:3328725483520 vcore:393976 ]", "memHeadroom": 4116709376, "vcoreHeadroom": 107976, "rootQueueUsage": "map[memory:304276817575936 pods:9015 vcore:21488000]", "rootMaxResource": "map[attachable-volumes-aws-ebs:23425 ephemeral-storage:55532238435958 hugepages -1Gi:0 hugepages-2Mi:0 hugepages-32Mi:0 hugepages-64Ki:0 memory:315878420541440 pods:57696 vcore:38325770]"}
3846 goroutine 96 [running]:
3847 runtime/debug.Stack()
3848 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/.go/src/runtime/debug/stack.go:24 +0x64
3849 runtime/debug.PrintStack()
3850 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/.go/src/runtime/debug/stack.go:16 +0x1c
3851 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).IncAllocatedResourceDebug(0x402ce5be00, 0x40d960ab40, 0x40a1f3cee0, 0x40d960ab50, 0x0)
3852 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1084 +0xec0
3853 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Application).tryNodeDebug(0x40265a2f00, 0x40d8756ba0?, 0x4043258f20, 0x40a1f3cee0)
3854 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/application.go:1677 +0x124
3855 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Application).tryReservedAllocate(0x40265a2f00, 0x40d960ab40, 0x4033155d78)
3856 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/application.go:1295 +0x1f4
3857 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5be00, 0x4016db2fc0?)
3858 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1471 +0x304
3859 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5bc80, 0x4016db2fc0?)
3860 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3861 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5a300, 0x24820?)
3862 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3863 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402a903980, 0x0?)
3864 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3865 github.com/apache/yunikorn-core/pkg/scheduler.(*PartitionContext).tryReservedAllocate(0x402cc0a9c0)
3866 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/partition.go:814 +0x90
3867 github.com/apache/yunikorn-core/pkg/scheduler.(*ClusterContext).schedule(0x4033155f80?)
3868 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/context.go:131 +0xf4
3869 github.com/apache/yunikorn-core/pkg/scheduler.(*Scheduler).internalSchedule(0x40003b5020)
3870 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/scheduler.go:88 +0xd4
3871 created by github.com/apache/yunikorn-core/pkg/scheduler.(*Scheduler).StartService
3872 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/scheduler.go:67 +0x21c
3873 2024-02-03T04:48:58.834Z INFO objects/application.go:865 reservation removed successfully {"node": "ip-10-181-118-197.ec2.internal", "app": "test-app", "ask": "d622ab3f-d9e d-44ec-892d-329e8a6f6963"}
3874 2024-02-03T04:48:58.834Z INFO scheduler/partition.go:945 allocation ask is unreserved {"appID": "test-app", "queue": "test-queue", "allocationK ey": "d622ab3f-d9ed-44ec-892d-329e8a6f6963", "node": "ip-10-181-118-197.ec2.internal", "reservationsRemoved": 1}
3875 2024-02-03T04:48:58.834Z INFO scheduler/partition.go:891 scheduler allocation processed {"appID": "test-app", "allocationKey": "d622ab3f-d9ed-44ec-892d-329e8a6f6963", "uuid": "59 7101e7-dfd7-44f7-86ed-5e694c6244b7", "allocatedResource": "map[memory:23394779136 pods:1 vcore:2000]", "placeholder": false, "targetNode": "ip-10-181-118-197.ec2.internal"}
{code}
Notice the “{*}scheduler allocation processed”{*} at the end
stackTraceB:
{code:java}
3879 2024-02-03T04:48:58.835Z WARN objects/application.go:1679 queue update failed unexpectedly {"application": "test-app", "ask-app": "a28b08df-galaxypinj-runner-pin-manual--20- 1", "ask-id": "b604da5a-7d54-418b-8fff-b0682f9fb209", "error": "allocation (map[memory:23394779136 pods:1 vcore:2000]) puts queue 'test-queue' over maximum allocation (map[memory:3328725483520 vco re:393976]), current usage (map[memory:3324608774144 pods:143 vcore:286000]) passed headRoom (map[attachable-volumes-aws-ebs:23425 ephemeral-storage:55532238435958 hugepages-1Gi:0 hugepages-2Mi:0 hugepages-32Mi:0 hugepages-64Ki:0 memory:4116709376 pods:48681 vcore:107976])"}
3880 goroutine 96 [running]:
3881 runtime/debug.Stack()
3882 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/.go/src/runtime/debug/stack.go:24 +0x64
3883 runtime/debug.PrintStack()
3884 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/.go/src/runtime/debug/stack.go:16 +0x1c
3885 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Application).tryNodeDebug(0x40265a2f00, 0x14fd9e0?, 0x404374d080, 0x40c57c4000)
3886 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/application.go:1684 +0x3e4
3887 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Application).tryNodesNoReserveDebug(0x4033155060?, 0x40925ff050?, 0x40c57c4000, {0x19c8370, 0x40842a6c40}, {0x4003ca7f20, 0x1e})
3888 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/application.go:1421 +0xe0
3889 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Application).tryReservedAllocate(0x40265a2f00, 0x40d960adf0, 0x4033155d78)
3890 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/application.go:1312 +0x3a4
3891 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5be00, 0x4016db2fc0?)
3892 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1471 +0x304
3893 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5bc80, 0x4016db2fc0?)
3894 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3895 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402ce5a300, 0x24820?)
3896 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3897 github.com/apache/yunikorn-core/pkg/scheduler/objects.(*Queue).TryReservedAllocate(0x402a903980, 0x0?)
3898 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/objects/queue.go:1490 +0xd0
3899 github.com/apache/yunikorn-core/pkg/scheduler.(*PartitionContext).tryReservedAllocate(0x402cc0a9c0)
3900 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/partition.go:814 +0x90
3901 github.com/apache/yunikorn-core/pkg/scheduler.(*ClusterContext).schedule(0x4033155f80?)
3902 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/context.go:131 +0xf4
3903 github.com/apache/yunikorn-core/pkg/scheduler.(*Scheduler).internalSchedule(0x40003b5020)
3904 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/scheduler.go:88 +0xd4
3905 created by github.com/apache/yunikorn-core/pkg/scheduler.(*Scheduler).StartService
3906 /mnt/jenkins-data/workspace/build-yunikorn-docker-image-worker/yunikorn-core/pkg/scheduler/scheduler.go:67 +0x21c
{code}
> Need to check headroom when trying other nodes for reserved allocations
> -----------------------------------------------------------------------
>
> Key: YUNIKORN-2030
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2030
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Priority: Major
>
> As reported in YUNIKORN-1996, we are seeing many messages like below from time to time:
> {code:java}
> WARN objects/application.go:1504 queue update failed unexpectedly {“error”: “allocation (map[memory:37580963840 pods:1 vcore:2000]) puts queue ‘root.test-queue’ over maximum allocation (map[memory:3300011278336 vcore:390584]), current usage (map[memory:3291983380480 pods:91 vcore:186000])“}{code}
> Restarting Yunikorn helps stoppinging it. Creating this Jira to investigate why it happened, because it's not supposed to happen as we check if there is enough resource headroom before calling
>
> {code:java}
> func (sa *Application) tryNode(node *Node, ask *AllocationAsk) *Allocation
> {code}
> which printed the above message, and only call it when there is enough headroom.
> There maybe a bug in headroom checking?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org