You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Andres de la Peña (Jira)" <ji...@apache.org> on 2021/05/04 12:15:00 UTC
[jira] [Comment Edited] (CASSANDRA-16625) Add a CircleCI job to run
some tests repeatedly
[ https://issues.apache.org/jira/browse/CASSANDRA-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338922#comment-17338922 ]
Andres de la Peña edited comment on CASSANDRA-16625 at 5/4/21, 12:14 PM:
-------------------------------------------------------------------------
I have added a loop-based CircleCI job for running repeated JUnit tests. That job can be used for regular unit tests, long unit tests, jvm dtests, etc. The Ant target is set in the env var [{{REPEATED_UTEST_TARGET}}|https://github.com/apache/cassandra/blob/a63fc6b9dee772caa5014a543046a12a5d23eda5/.circleci/config-2_1.yml#L38-L44]. I still have to add some doc, but I think that the PR is ready for a first round of review.
As an example, I have used the repeated test jobs to reproduce some of the flaky tests that we have opened for 4.0:
||branch||config||run||
|CASSANDRA-16644|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16644]|[3270|https://app.circleci.com/pipelines/github/adelapena/cassandra/376/workflows/eb463f0a-ad93-488a-89a3-bb44fbc0c578/jobs/3270]|
|CASSANDRA-16627|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16627]|[3271|https://app.circleci.com/pipelines/github/adelapena/cassandra/375/workflows/1d48154f-b6eb-4d0b-bb88-fe40a8d40f27/jobs/3271]|
|CASSANDRA-16637|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16637]|[3255|https://app.circleci.com/pipelines/github/adelapena/cassandra/374/workflows/df3c4361-c6f3-4cbf-801f-dfedeef95652/jobs/3255]|
|CASSANDRA-16598|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16598]|[3260|https://app.circleci.com/pipelines/github/adelapena/cassandra/373/workflows/f1612e92-8a59-483b-bc79-d6f29a7f5766/jobs/3260]|
All the test failures reported in the tickets are reproduced. For some of the runs the CircleCI UI is not able to properly parse and highlight the failed test output, probably due to its large size. However, the failures can still be seen in the downloadable test output file.
Unfortunately there isn't an easy way to specify the repeated tests parameters in the CircleCI UI. It's however possible to [trigger a pipeline|https://circleci.com/docs/2.0/pipeline-variables/] with a POST request specifying parameters. The problem is that pipeline parameters should be in the final {{config.yml}} file, but {{circleci config process}} removes them from the configuration in {{config-2_1.yml}} (see [here|https://github.com/CircleCI-Public/circleci-cli/issues/359]), which is a CircleCI limitation if not a bug. We could generate patches to add the repeated tests parameters to the resource files that are generated from the config, so we can start repeated runs with a simple http request without editing the config files. However patching automatically generated files is probably too risky, given that they might change depending on the used circleCI CLI. wdyt?
was (Author: adelapena):
I have added a loop-based CircleCI job for running repeated JUnit tests. That job can be used for regular unit tests, long unit tests, jvm dtests, etc. The Ant target is set in the env var [{{REPEATED_UTEST_TARGET}}|https://github.com/apache/cassandra/blob/a63fc6b9dee772caa5014a543046a12a5d23eda5/.circleci/config-2_1.yml#L38-L44]. I still have to add some doc, but I think that the PR is ready for a first round of review.
As an example, I have used the repeated test jobs to reproduce some of the flaky tests that we have opened for 4.0:
||branch||config||run||
|CASSANDRA-16644|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16644]|[3270|https://app.circleci.com/pipelines/github/adelapena/cassandra/376/workflows/eb463f0a-ad93-488a-89a3-bb44fbc0c578/jobs/3270]|
|CASSANDRA-16627|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16627]|[3271|https://app.circleci.com/pipelines/github/adelapena/cassandra/375/workflows/1d48154f-b6eb-4d0b-bb88-fe40a8d40f27/jobs/3271]|
|CASSANDRA-16637|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16637]|[3255|https://app.circleci.com/pipelines/github/adelapena/cassandra/374/workflows/df3c4361-c6f3-4cbf-801f-dfedeef95652/jobs/3255]|
|CASSANDRA-16598|[diff|https://github.com/adelapena/cassandra/compare/16625-trunk..16625-trunk-run-16598]|[3260|https://app.circleci.com/pipelines/github/adelapena/cassandra/373/workflows/f1612e92-8a59-483b-bc79-d6f29a7f5766/jobs/3260]|
All the test failures reported in the tickets are reproduced. For some of the runs the CircleCI UI is not able to properly parse and highlight the failed test output, probably due to its large size. However, the failures can still be seen in the downloadable test output file.
Unfortunately there isn't an easy way to specify the repeated tests parameters in the CircleCI UI. It's however possible to [trigger a pipeline|https://circleci.com/docs/2.0/api-job-trigger/] with a POST request specifying parameters. The problem is that pipeline parameters should be in the final {{config.yml}} file, but {{circleci config process}} removes them from the configuration in {{config-2_1.yml}} (see [here|https://github.com/CircleCI-Public/circleci-cli/issues/359]), which is a CircleCI limitation if not a bug. We could generate patches to add the repeated tests parameters to the resource files that are generated from the config, so we can start repeated runs with a simple http request without editing the config files. However patching automatically generated files is probably too risky, given that they might change depending on the used circleCI CLI. wdyt?
> Add a CircleCI job to run some tests repeatedly
> -----------------------------------------------
>
> Key: CASSANDRA-16625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16625
> Project: Cassandra
> Issue Type: Task
> Components: CI
> Reporter: Andres de la Peña
> Assignee: Andres de la Peña
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think it could be useful to have an optional CircleCI job to run some specific tests n times. That way, tickets could attach CircleCI runs showing that the changes don't make a certain ticket flaky or, conversely, that they fix a flaky test. Doing this systematically should mitigate the risk of introducing new flaky tests, and I guess it would be more convenient and easy to share than running the tests locally or on a private CI system.
> It would also be nice to have something similar in Jenkins, but I'm focusing this ticket on CircleCI because it's available also for non-committers, so assignees can run their tests before setting the tickets as ready for review.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org