You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Wilfred Spiegelenburg (Jira)" <ji...@apache.org> on 2020/03/20 06:13:00 UTC
[jira] [Comment Edited] (YUNIKORN-30) flaky tests cause build
failures on PRs
[ https://issues.apache.org/jira/browse/YUNIKORN-30?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063052#comment-17063052 ]
Wilfred Spiegelenburg edited comment on YUNIKORN-30 at 3/20/20, 6:12 AM:
-------------------------------------------------------------------------
Further details:
I have what I think is the root cause behind some of the failures. I described one of the cases above that I found but some others show a different failure type.
* Incorrect partition name: we seem to call normalise on the partition name twice during our testing.
{code:java}
2020-03-19T21:31:53.9142459Z 2020-03-19T21:31:41.042Z INFO cache/cluster_info.go:618 Failed to find partition for allocation proposal {"partitionName": "[rm:123][rm:123]default"}
{code}
This should never be a problem and shows a bug in the code and we should be able to handle this. The fix is in the normalisation code to check if it is already normalised.
* Event handling: a generic underlying issue. During some local testing I noticed that we do not properly wait for the event handling to process all the events that are generated. In the case observed: allocation releases were still being processed while the end state check was performed. Those issues can be fixed by a proper wait in the test code.
* However in certain failures we see nothing. This could point to a problem with go routines not being scheduled. The logs for these cases show a blank period of about 1 sec (the max time we wait for things) between the normal processing and the wait timing out. I cannot really reproduce those yet.
Working on a PR to fix at least the majority of what I have found.
was (Author: wifreds):
Further details:
I have what I think is the root cause behind some of the failures. I described one of the cases above that I found but some others show a different failure type.
* Incorrect partition name: we seem to call normalise on the partition name twice during our testing.
{code}
2020-03-19T21:31:53.9142459Z 2020-03-19T21:31:41.042Z INFO cache/cluster_info.go:618 Failed to find partition for allocation proposal {"partitionName": "[rm:123][rm:123]default"}
{code}
This should never be a problem and shows a bug in the code and we should be able to handle this. The fix is in the normalisation code to check if it is already normalised.
* Event handling: a generic underlying issue. During some local testing I noticed that we do not properly wait for the event handling to process all the events that are generated. In the case observed: allocation releases were still being processed while the end state check was performed. Those issues can be fixed by a proper wait in the test code.
* However in certain failures we see nothing. This could point to a problem with go routines not being scheduled. The logs for these cases show a blank period of about 1 sec (the max time we wait for things) between the normal processing and the wait timing out. I cannot really reproduce those yet.
Working on a PR to fix at least the majority of what I have found.
> flaky tests cause build failures on PRs
> ---------------------------------------
>
> Key: YUNIKORN-30
> URL: https://issues.apache.org/jira/browse/YUNIKORN-30
> Project: Apache YuniKorn
> Issue Type: Test
> Components: test - smoke
> Reporter: Wilfred Spiegelenburg
> Assignee: Wilfred Spiegelenburg
> Priority: Blocker
> Attachments: TestBasicScheduler_github_fail.log
>
>
> Smoke tests have been failing on PR triggered builds.
> Failures are inconsistent and linked to multiple test cases, failures in the same tests can even happen in different lines of code in different runs without changes:
> {code}
> 2020-03-11T04:39:40.8332236Z --- FAIL: TestSchedulerRecovery (3.07s)
> 2020-03-11T04:39:40.8340886Z ##[error] mock_rm_callback.go:175: Failed to wait for allocations, expected 4, actual 3, called from: TestSchedulerRecovery in scheduler_recovery_test.go:213
> {code}
> {code}
> 2020-03-11T04:39:40.9102758Z --- FAIL: TestBasicScheduler (1.11s)
> 2020-03-11T04:39:40.9103549Z ##[error] mock_rm_callback.go:175: Failed to wait for allocations, expected 4, actual 3, called from: TestBasicScheduler in scheduler_smoke_test.go:341
> {code}
> {code}
> 2020-03-06T07:17:50.4567697Z --- FAIL: TestReservationForTwoQueues (3.10s)
> 2020-03-06T07:17:50.4574239Z ##[error] scheduler_reservation_test.go:276: partition reservations are missing
> {code}
> {code}
> 2020-03-06T08:08:21.8912443Z --- FAIL: TestRemoveReservedNode (1.05s)
> 2020-03-06T08:08:21.8917559Z ##[error] scheduler_utils.go:79: Failed to wait for pending resource, expected 80, actual 60, called from: TestRemoveReservedNode in scheduler_reservation_test.go:356
> {code}
> {code}
> 2020-03-04T10:42:16.5788872Z --- FAIL: TestRemoveReservedNode (0.07s)
> 2020-03-04T10:42:16.5789359Z ##[error] scheduler_reservation_test.go:357: assertion failed: 2 (int) != 1 (int): reservations missing from app
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org