You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by "Praveen (Jira)" <ji...@apache.org> on 2023/02/21 21:38:00 UTC
[jira] [Created] (YUNIKORN-1596) Pods marked unschedulable when dynamic PVC times out
Praveen created YUNIKORN-1596:
---------------------------------
Summary: Pods marked unschedulable when dynamic PVC times out
Key: YUNIKORN-1596
URL: https://issues.apache.org/jira/browse/YUNIKORN-1596
Project: Apache YuniKorn
Issue Type: Bug
Reporter: Praveen
We are seeing a behavior when a scheduled pod requesting for PVC times out, its marked as unschedulable. There are no retries on such pod and remain in 'pending' state. With pods in pending, autoscaler does not scale down nodes. This seems similar to issue discussed here:
[https://github.com/kubernetes/autoscaler/issues/3409]
Error from Yunikorn logs :
ERROR cache/context.go:527 Failed to bind pod volumes \{"podName": "<PODNAME>", "nodeName": "<IP>", "dynamicProvisions": 1, "staticBindings": 0}
...
...
/workspace/pkg/cache/task.go:382
2023-02-20T00:02:22.368Z ERROR cache/task.go:265 task failed \{"appID": "<APPID>", "taskID": "45981d91-e543-459b-9657-bdc03b57e26f", "reason": "bind pod volumes failed, name: <NS/PODNAME>, binding volumes: timed out waiting for the condition”}
{{From Autoscalar logs}}
I0220 20:47:01.775653 1 static_autoscaler.go:502] Scale down status: unneededOnly=true lastScaleUpTime=2023-02-20 19:20:56.429598603 +0000 UTC m=+249612.380355315 lastScaleDownDeleteTime=2023-02-20 06:36:50.929515212 +0000 UTC m=+203766.880271921 lastScaleDownFailTime=2023-02-17 22:01:33.693397034 +0000 UTC m=+49.644153730 scaleDownForbidden=true isDeleteInProgress=false scaleDownInCooldown=true
I0220 20:47:11.787999 1 static_autoscaler.go:228] Starting main loop
I0220 20:47:11.792789 1 filter_out_schedulable.go:65] Filtering out schedulables
I0220 20:47:11.792953 1 scheduler_binder.go:829] All bound volumes for Pod "<podname>" match with Node <node>"
I0220 20:47:11.792981 1 filter_out_schedulable.go:118] Pod <podname> marked as unschedulable can be scheduled on node <node> (based on hinting). Ignoring in scale up.
# Can Yunikorn introduce retries for such scenarios?
# Can pods be set to error state after retries?
{{Note: pod name, nodename and ip masked above}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org