You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yu-Lin Chen (Jira)" <ji...@apache.org> on 2024/01/22 01:47:00 UTC
[jira] [Commented] (YUNIKORN-2313) Flaky E2E Test: "Verify_basic_preemption" experiences flakiness due to race condition
[ https://issues.apache.org/jira/browse/YUNIKORN-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17809210#comment-17809210 ]
Yu-Lin Chen commented on YUNIKORN-2313:
---------------------------------------
The root cause of this issue was found in [YUNIKORN-2327] Race condition during update Occupied Resource from Shim to Core and fixed in [YUNIKORN-2329] Ensure RMProxy events are processed in-order.
Rename the title to include "race condition" for providing a more descriptive expression.
> Flaky E2E Test: "Verify_basic_preemption" experiences flakiness due to race condition
> --------------------------------------------------------------------------------------
>
> Key: YUNIKORN-2313
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2313
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: test - e2e
> Reporter: Yu-Lin Chen
> Assignee: Yu-Lin Chen
> Priority: Major
> Labels: pull-request-available
> Attachments: 2_e2e-tests (v1.28.0) （Verify_basic_preemption）.txt
>
>
> The failed e2e test "Verify_basic_preemption" attempted to request more resources than the available resource on the K8S node. The root cause is that some pods running on the node are invisible to YuniKorn scheduler.
> The e2e test mistakenly believed that it could acquire all the available resources recorded in YuniKorn.
> [https://github.com/apache/yunikorn-k8shim/actions/runs/7420644533/job/20212796775?pr=759#step:6:1483]
>
> *Logs:* (Verify_basic_preemption)
> {*}K8S describe node result{*}: (node: yk8s-worker)
> allocable resource: 14610305024 B
> non-terminated pods's memory usage:
> * sleepjob1 - 4870M (33%)
> * sleepjob2 - 4870M (33%)
> * kindnet-tr9jc - {color:#de350b}50Mi{color}(0%)
> * kube-proxy-k58rn - 0 B
> * yunikorn-admission-controller-56c8c8b766-55kz8 - {color:#de350b}500Mi{color} (3%)
> → Total 70%
> {*}YuniKorn Node REST API response{*}: (node: yk8s-worker)
> * memory capacity: 14610305024 B
> * memory occupied: *{color:#de350b}Empty{color}*
> * memory allocated: 9740000000 B (4870M + 4870M) (66%)
> * memory available: 4870305024 B
> Node only have 30% resource remaining, so the sleepjob3(4870M, 33%) never entered Running state.
> YuniKorn is not aware of 'kindnet-tr9jc' and 'yunikorn-admission-controller-56c8c8b766-55kz8' running on the node. The “YuniKorn node available resource” is not equal to “K8S node available resource”.
> Remove dynamic limit for the sleepJob and set a fixed pod/queue limit could solve the issue on the e2e test. But still need to investigate why those running pod are invisible to YuniKorn.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org