You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yu-Lin Chen (Jira)" <ji...@apache.org> on 2023/12/07 07:32:00 UTC
[jira] [Commented] (YUNIKORN-2244) Fix flaky e2e user_group_limit_test
[ https://issues.apache.org/jira/browse/YUNIKORN-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794063#comment-17794063 ]
Yu-Lin Chen commented on YUNIKORN-2244:
---------------------------------------
Just have a quick check on the error, the error happened on ginkgo.AfterSuite and might not specific to user_group_limit.
{code:go}
ginkgo.By("Check Yunikorn's health")
checks, err := yunikorn.GetFailedHealthChecks()
{code}
There have data inconsistency when health check compare sum of nodes total allocation/resources to partition's total allocation/resources.
The root cause might be that those numbers are hold by different objects
* Partition allocation: record in root *Queue
* Partition resources: record in *PartitionContext
* Node allocation: record in *Node
* Node resources: record in *Node
YuniKorn didn't hold partition/node locks together when performing update. The primary concern is likely related to performance. Therefore, the inconsistency is as expected.
I think one possible solution is we can add retry to AfterSuite's health check or just remove it from AfterSuite.
Please correct me if I'm wrong.
> Fix flaky e2e user_group_limit_test
> -----------------------------------
>
> Key: YUNIKORN-2244
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2244
> Project: Apache YuniKorn
> Issue Type: Bug
> Reporter: Chia-Ping Tsai
> Assignee: Lyu Bo Cian
> Priority: Minor
>
> from https://github.com/apache/yunikorn-k8shim/actions/runs/7117154586/job/19381185213?pr=746
> {code:java}
> ------------------------------
> [AfterSuite]
> /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_test.go:93
> STEP: Check Yunikorn's health @ 12/06/23 18:43:31.95
> [FAILED] in [AfterSuite] - /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_test.go:97 @ 12/06/23 18:43:31.951
> [AfterSuite] [FAILED] [0.001 seconds]
> [AfterSuite]
> /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_test.go:93
> [FAILED] Name: Consistency of data Message: Partitions with inconsistent data: ["[mycluster]default"]
> Expected
> <string>: Name: Consistency of data Message: Partitions with inconsistent data: [\"[mycluster]default\"]\n
> to equal
> <string>:
> In [AfterSuite] at: /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_test.go:97 @ 12/06/23 18:43:31.951
> ------------------------------
> [ReportAfterSuite] TestUserGroupLimit
> /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_suite_test.go:37
> [ReportAfterSuite] PASSED [0.002 seconds]
> ------------------------------
> Summarizing 1 Failure:
> [FAIL] [AfterSuite]
> /home/runner/work/yunikorn-k8shim/yunikorn-k8shim/test/e2e/user_group_limit/user_group_limit_test.go:97
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org