You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Ayub Pathan (Jira)" <ji...@apache.org> on 2021/01/27 03:29:00 UTC
[jira] [Commented] (YUNIKORN-518) Scheduler restarts observed due
to admission controller "FailedPostStartHook"
[ https://issues.apache.org/jira/browse/YUNIKORN-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272534#comment-17272534 ]
Ayub Pathan commented on YUNIKORN-518:
--------------------------------------
In one of the instances, scheduler was restarted 32 times.
{noformat}
NAMESPACE NAME READY STATUS RESTARTS AGE
default batch-sleep-job-1-7lrd5 0/1 Completed 0 3h55m
default batch-sleep-job-1-lw4t9 0/1 Completed 0 3h55m
default batch-sleep-job-2-c95dg 0/1 Completed 0 3h54m
default batch-sleep-job-2-vnfjb 0/1 Completed 0 3h54m
default batch-sleep-job-2-x4mcz 0/1 Completed 0 3h54m
default batch-sleep-job-2-ztnfq 0/1 Completed 0 3h54m
default batch-sleep-job-3-59mnj 0/1 Pending 0 3h53m
default batch-sleep-job-3-7tp5t 0/1 Pending 0 3h53m
default batch-sleep-job-3-bm4fd 0/1 Pending 0 3h53m
default batch-sleep-job-3-c4mxg 0/1 Pending 0 3h53m
default batch-sleep-job-3-cljfj 0/1 Pending 0 3h53m
default batch-sleep-job-3-gcvnp 0/1 Pending 0 3h53m
default batch-sleep-job-3-gwgnn 0/1 Pending 0 3h53m
default batch-sleep-job-3-kj88t 0/1 Pending 0 3h53m
default batch-sleep-job-3-p8c7w 0/1 Pending 0 3h53m
default batch-sleep-job-3-td575 0/1 Pending 0 3h53m
kube-system calico-kube-controllers-db689dff4-jns7j 1/1 Running 0 12d
kube-system calico-node-5k8mv 1/1 Running 0 12d
kube-system calico-node-8trmk 1/1 Running 0 12d
kube-system calico-node-jxgsg 1/1 Running 0 12d
kube-system calico-node-qfhxd 1/1 Running 0 12d
kube-system calico-typha-7d9d44d8db-c6nvk 1/1 Running 0 12d
kube-system calico-typha-7d9d44d8db-rq6kw 1/1 Running 0 12d
kube-system cluster-autoscaler-f7499f8dc-cqpsl 1/1 Running 1 12d
kube-system coredns-58b7b9bd9d-bx2hz 1/1 Running 0 12d
kube-system coredns-58b7b9bd9d-zbct9 1/1 Running 0 12d
kube-system heapster-67c8c5b767-lhvp4 1/1 Running 0 12d
kube-system kube-proxy-2fd7m 1/1 Running 0 12d
kube-system kube-proxy-d8bj9 1/1 Running 0 12d
kube-system kube-proxy-lx95n 1/1 Running 0 12d
kube-system kube-proxy-xktfn 1/1 Running 0 12d
kube-system kubernetes-dashboard-d88d55985-jnv98 1/1 Running 0 12d
kube-system monitoring-influxdb-6574bdd4bf-xl8hz 1/1 Running 1 12d
kube-system spot-instance-termination-handler-aws-node-termination-han2jm6w 1/1 Running 0 12d
kube-system spot-instance-termination-handler-aws-node-termination-hanjr9sb 1/1 Running 0 12d
kube-system spot-instance-termination-handler-aws-node-termination-hanjwpns 1/1 Running 0 12d
kube-system spot-instance-termination-handler-aws-node-termination-hannhgr9 1/1 Running 0 12d
tiller tiller-deploy-f897b44dd-8tcrk 1/1 Running 0 12d
yunikorn yunikorn-scheduler-6577f789d8-pl76n 1/2 CrashLoopBackOff 32 142m {noformat}
> Scheduler restarts observed due to admission controller "FailedPostStartHook"
> -----------------------------------------------------------------------------
>
> Key: YUNIKORN-518
> URL: https://issues.apache.org/jira/browse/YUNIKORN-518
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 0.10
> Reporter: Ayub Pathan
> Priority: Major
>
> {noformat}
> Name: yunikorn-scheduler-6577f789d8-vc5cc
> Namespace: yunikorn
> Priority: 0
> Node: ip-10-192-153-109.ca-central-1.compute.internal/10.192.153.109
> Start Time: Tue, 26 Jan 2021 19:17:12 -0800
> Labels: app=yunikorn
> component=yunikorn-scheduler
> pod-template-hash=6577f789d8
> release=yunikorn
> Annotations: cni.projectcalico.org/podIP: 100.100.166.78/32
> cni.projectcalico.org/podIPs: 100.100.166.78/32
> kubernetes.io/psp: eks.privileged
> Status: Running
> IP: 100.100.166.78
> IPs:
> IP: 100.100.166.78
> Controlled By: ReplicaSet/yunikorn-scheduler-6577f789d8
> Containers:
> yunikorn-scheduler-k8s:
> Container ID: docker://759f2b2f14ba37f46a42cdc59a5c51ed19d442ed717b81ee98d30177b7a184e6
> Image: container-dev.repo.cloudera.com/cloudera/yunikorn-scheduler:0.10.0-b9
> Image ID: docker-pullable://container-dev.repo.cloudera.com/cloudera/yunikorn-scheduler@sha256:878300a91cfd3b9d6dc515948afbfab23572a475b0df7006f06480ee06d1aceb
> Port: 9080/TCP
> Host Port: 0/TCP
> State: Running
> Started: Tue, 26 Jan 2021 19:18:01 -0800
> Last State: Terminated
> Reason: Error
> Exit Code: 1
> Started: Tue, 26 Jan 2021 19:17:33 -0800
> Finished: Tue, 26 Jan 2021 19:17:33 -0800
> Ready: True
> Restart Count: 3
> Limits:
> cpu: 4
> memory: 2Gi
> Requests:
> cpu: 200m
> memory: 1Gi
> Environment:
> NAMESPACE: yunikorn (v1:metadata.namespace)
> ADMISSION_CONTROLLER_IMAGE_REGISTRY: container-dev.repo.cloudera.com/cloudera/yunikorn-admission
> ADMISSION_CONTROLLER_IMAGE_TAG: 0.10.0-b9
> ADMISSION_CONTROLLER_IMAGE_PULL_POLICY: Always
> ADMISSION_CONTROLLER_IMAGE_PULL_SECRETS: [dockercreds]
> Mounts:
> /etc/yunikorn/ from config-volume (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from yunikorn-admin-token-dnq4h (ro)
> yunikorn-scheduler-web:
> Container ID: docker://0b8205bb8292f193765bbc563ea10010106fd316257e523c3446c5685ee0d5bf
> Image: container-dev.repo.cloudera.com/cloudera/yunikorn-web:0.10.0-b9
> Image ID: docker-pullable://container-dev.repo.cloudera.com/cloudera/yunikorn-web@sha256:a64b986df2dc737958701838f41f9fae7f2e4a353a497949ba6b9e75b4b44b66
> Port: 9889/TCP
> Host Port: 0/TCP
> State: Running
> Started: Tue, 26 Jan 2021 19:17:17 -0800
> Ready: True
> Restart Count: 0
> Limits:
> cpu: 200m
> memory: 500Mi
> Requests:
> cpu: 100m
> memory: 100Mi
> Environment: <none>
> Mounts:
> /var/run/secrets/kubernetes.io/serviceaccount from yunikorn-admin-token-dnq4h (ro)
> Conditions:
> Type Status
> Initialized True
> Ready True
> ContainersReady True
> PodScheduled True
> Volumes:
> config-volume:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: yunikorn-configs
> Optional: false
> yunikorn-admin-token-dnq4h:
> Type: Secret (a volume populated by a Secret)
> SecretName: yunikorn-admin-token-dnq4h
> Optional: false
> QoS Class: Burstable
> Node-Selectors: role.node.kubernetes.io/liftie-infra=true
> Tolerations: CriticalAddonsOnly op=Exists
> node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
> node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
> role.node.kubernetes.io/liftie-infra=true:NoSchedule
> Events:
> Type Reason Age From Message
> ---- ------ ---- ---- -------
> Normal Scheduled 61s default-scheduler Successfully assigned yunikorn/yunikorn-scheduler-6577f789d8-vc5cc to ip-10-192-153-109.ca-central-1.compute.internal
> Normal Pulling 57s kubelet Pulling image "container-dev.repo.cloudera.com/cloudera/yunikorn-web:0.10.0-b9"
> Normal Started 56s kubelet Started container yunikorn-scheduler-web
> Normal Created 56s kubelet Created container yunikorn-scheduler-web
> Normal Pulled 56s kubelet Successfully pulled image "container-dev.repo.cloudera.com/cloudera/yunikorn-web:0.10.0-b9"
> Warning FailedPreStopHook 55s (x2 over 58s) kubelet Exec lifecycle hook ([/bin/sh /admission_util.sh delete]) for Container "yunikorn-scheduler-k8s" in Pod "yunikorn-scheduler-6577f789d8-vc5cc_yunikorn(082e1cc7-8765-4aa3-baac-48e3b048cfc6)" failed - error: command '/bin/sh /admission_util.sh delete' exited with 126: , message: "cannot exec in a stopped state: unknown\r\n"
> Normal Killing 55s (x2 over 58s) kubelet FailedPostStartHook
> Warning BackOff 53s (x2 over 54s) kubelet Back-off restarting failed container
> Normal Pulling 41s (x3 over 60s) kubelet Pulling image "container-dev.repo.cloudera.com/cloudera/yunikorn-scheduler:0.10.0-b9"
> Warning FailedPostStartHook 40s (x3 over 58s) kubelet Exec lifecycle hook ([/bin/sh /admission_util.sh create]) for Container "yunikorn-scheduler-k8s" in Pod "yunikorn-scheduler-6577f789d8-vc5cc_yunikorn(082e1cc7-8765-4aa3-baac-48e3b048cfc6)" failed - error: command '/bin/sh /admission_util.sh create' exited with 137: , message: ""
> Normal Started 40s (x3 over 58s) kubelet Started container yunikorn-scheduler-k8s
> Normal Created 40s (x3 over 58s) kubelet Created container yunikorn-scheduler-k8s
> Normal Pulled 40s (x3 over 58s) kubelet Successfully pulled image "container-dev.repo.cloudera.com/cloudera/yunikorn-scheduler:0.10.0-b9" {noformat}
> This is not a blocker but the scheduler was restarted multiple(3) times, hence reporting. This could be due to issue in admission controller start script/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org