You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2022/03/29 09:27:00 UTC
[jira] [Updated] (YUNIKORN-1161) Pods not linked to placeholders are stuck in Running state if YK is restarted
[ https://issues.apache.org/jira/browse/YUNIKORN-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Bacsko updated YUNIKORN-1161:
-----------------------------------
Summary: Pods not linked to placeholders are stuck in Running state if YK is restarted (was: Pods not linked to placeholders are stuck in Running state)
> Pods not linked to placeholders are stuck in Running state if YK is restarted
> -----------------------------------------------------------------------------
>
> Key: YUNIKORN-1161
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1161
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Peter Bacsko
> Priority: Major
>
> If we create pods where the nam of the task group does not match the {{task-group-name}} annotation, then the real pods will not transition to Running state when the placeholder pods expire.
> For example, extend the sleep batch job like that:
> {noformat}
> apiVersion: batch/v1
> kind: Job
> metadata:
> name: batch-sleep-job-9
> spec:
> completions: 5
> parallelism: 5
> template:
> metadata:
> labels:
> app: sleep
> applicationId: "batch-sleep-job-9"
> queue: root.sandbox
> annotations:
> yunikorn.apache.org/task-group-name: sleep-groupxxx
> yunikorn.apache.org/task-groups: |-
> [{
> "name": "sleep-group",
> "minMember": 5,
> "minResource": {
> "cpu": "100m",
> "memory": "2000M"
> },
> "nodeSelector": {},
> "tolerations": []
> }]
> ...
> {noformat}
> Submit the job and restart Yunikorn when the placeholders are already running.
> This will result in "batch-sleep-job-9-nnnnn" pods that are not transitioning to {{Running}} and they have to be manually terminated.
> {noformat}
> default batch-sleep-job-9-hgxxl 0/1 Pending 0 19m
> default batch-sleep-job-9-j6twt 0/1 Pending 0 19m
> default batch-sleep-job-9-l4jhm 0/1 Pending 0 19m
> default batch-sleep-job-9-swlm4 0/1 Pending 0 19m
> default batch-sleep-job-9-z6wqx 0/1 Pending 0 19m
> default yunikorn-admission-controller-78c775cfd9-6pp8d 1/1 Running 4 3d22h
> default yunikorn-scheduler-77dd7c665b-f8kkn 2/2 Running 0 17m
> {noformat}
> Note that without YK restart, they deallocated and removed properly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org