[jira] [Created] (YUNIKORN-3494) Flaky admission controller e2e: pod in test namespace not Running within 10s

["TingYao Huang (Jira)" <ji...@apache.org>]

TingYao Huang created YUNIKORN-3494:
---------------------------------------

             Summary: Flaky admission controller e2e: pod in test namespace not Running within 10s
                 Key: YUNIKORN-3494
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3494
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: test - e2e
            Reporter: TingYao Huang


The e2e spec {{AdmissionController [It] Verifying a pod is created in the test namespace}} fails intermittently:
{noformat}
[FAILED] Unexpected error:
    <context.deadlineExceededError>: context deadline exceeded
In [It] at: test/e2e/admission_controller/admission_controller_test.go:133
{noformat}
Seen in: https://github.com/apache/yunikorn-k8shim/actions/runs/36550600452/job/109448390699 (v1.35.8). A re-run of the same commit passed.

*Root cause*
The spec waits only 10s for the pod to reach Running. YuniKorn bound the pod within 2.7s:
||Time||Event||
|15:29:25.13|Pod created|
|15:29:27.86|Pod bound to yk8s-worker|
|15:29:35.30|10s timeout expires, pod still not Running|

The rest of the time goes to kubelet pulling the image and starting the container. {{testPod}} uses {{alpine:latest}} without an {{imagePullPolicy}}, so it defaults to {{Always}}. It is also the first pod in the e2e run that actually starts on a node: admission_controller is the first suite, and the earlier specs pin their pods to a non-existent node. So it does a cold pull from Docker Hub.
In a passing run (v1.24.17) the same spec took 5.8s from creation to Running, leaving about 4s of headroom.

The scheduler and the admission controller behaved correctly. This is a test timeout issue.

*Proposed fix*
* Raise the Running timeout in admission_controller_test.go:131 and :143 from 10s to 30s, as done in YUNIKORN-3468.
* Optional: set {{ImagePullPolicy: v1.PullIfNotPresent}} on {{testPod}} in admission_controller_suite_test.go.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org