You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2022/03/29 10:21:00 UTC
[jira] [Commented] (YUNIKORN-1161) Pods not linked to placeholders are stuck in Running state if YK is restarted
[ https://issues.apache.org/jira/browse/YUNIKORN-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513977#comment-17513977 ]
Peter Bacsko commented on YUNIKORN-1161:
----------------------------------------
Attached YK logs which has entries about a previous run where the same thing happened.
Also added the output of "kubectl describe pod".
> Pods not linked to placeholders are stuck in Running state if YK is restarted
> -----------------------------------------------------------------------------
>
> Key: YUNIKORN-1161
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1161
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Peter Bacsko
> Priority: Major
> Attachments: logs-from-yunikorn-scheduler-k8s-in-yunikorn-scheduler-after_restart_nomatchingtaskgroupname.txt, logs-from-yunikorn-scheduler-k8s-in-yunikorn-scheduler-before_restart_nomatchingtaskgroupname.txt, pods_nomatchingtaskgroupname.txt
>
>
> If we create pods where the nam of the task group does not match the {{task-group-name}} annotation, then the real pods will not transition to Running state when the placeholder pods expire and Yunikorn was restarted in the meantime.
> For example, extend the sleep batch job like that:
> {noformat}
> apiVersion: batch/v1
> kind: Job
> metadata:
> name: batch-sleep-job-9
> spec:
> completions: 5
> parallelism: 5
> template:
> metadata:
> labels:
> app: sleep
> applicationId: "batch-sleep-job-9"
> queue: root.sandbox
> annotations:
> yunikorn.apache.org/task-group-name: sleep-groupxxx
> yunikorn.apache.org/task-groups: |-
> [{
> "name": "sleep-group",
> "minMember": 5,
> "minResource": {
> "cpu": "100m",
> "memory": "2000M"
> },
> "nodeSelector": {},
> "tolerations": []
> }]
> ...
> {noformat}
> Submit the job and restart Yunikorn when the placeholders are already running.
> This will result in "batch-sleep-job-9-nnnnn" pods that are not transitioning to {{Running}} and they have to be manually terminated.
> {noformat}
> $ kubectl get pods -A | grep -E "(batch-sleep-job-9|yunikorn)"
> default batch-sleep-job-9-hgxxl 0/1 Pending 0 20m
> default batch-sleep-job-9-j6twt 0/1 Pending 0 20m
> default batch-sleep-job-9-l4jhm 0/1 Pending 0 20m
> default batch-sleep-job-9-swlm4 0/1 Pending 0 20m
> default batch-sleep-job-9-z6wqx 0/1 Pending 0 20m
> default yunikorn-admission-controller-78c775cfd9-6pp8d 1/1 Running 4 3d22h
> default yunikorn-scheduler-77dd7c665b-f8kkn 2/2 Running 0 18m
> {noformat}
> Note that without YK restart, they are deallocated and removed properly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org