You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Andrés de la Peña <ad...@apache.org> on 2022/10/18 16:33:24 UTC
New CircleCI test multiplexer
Just to let you know that CASSANDRA-17939 has just been committed.
It changes the way the CircleCI multiplexer works, in line with the recent
changes in our release criteria:
* The default number of repeated tests iterations is 500, except for long
and upgrade tests.
* It is possible to specify multiple test classes and methods to be
repeated into the same config push. So patches altering dozens of tests
won't require dozens of config pushes anymore.
* Running the .circleci/generate.sh script with -l/-m/-h flags will use git
diff to automatically detect the new or modified tests and will add them to
the lists of tests to be repeated. The pre-commit workflow will
automatically start repeated runs for these tests. The only exception to
this are Python dtests, that should be specified manually.
* The CircleCI jobs are rearranged so for every regular job there is a
companion job to run the repeated tests associated to that job. Those
companion jobs will only be visible if there are repeated tests to run. Here
<https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d>
is an example run with repeated tests for all the test suites, and here
<https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412>
is the same workflow without any repeated tests.
Some documentation on how to use it can be found here:
https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop
Re: New CircleCI test multiplexer
Posted by Ekaterina Dimitrova <e....@gmail.com>.
I am not familiar with the content of his scripts but if they are helping
to balance even more our resource usage without breaking the bank and
failing the environment, all is more than welcome. Thank you!
On Tue, 18 Oct 2022 at 16:40, Josh McKenzie <jm...@apache.org> wrote:
> Thanks for the context Ekaterina. My Grand Plan is to get whatever wisdom
> David encoded into his script in terms of job bucketing and parallelism
> into the mainline circle config and all of us benefit from that.
>
> On Tue, Oct 18, 2022, at 2:17 PM, Ekaterina Dimitrova wrote:
>
> First I want to take a moment to say thanks to Andres and Josh for this
> new improvement. I truly believe it will save us a lot of time and efforts
> in the flaky tests fight
>
> Pre-CASSANDRA-15712 there was default config for free tier and HIGHRES
> option(bumping all resources to max).
> We added MIDRES as a balanced version which gives you the chance to be
> able to run all tests but with less resources. What do I mean? It is still
> to be used by people with paid accounts but who want to consider the
> cost/resources spent. It was based on experimental runs which proved that
> with half the resources(compared to max out) for some suites we get similar
> and reasonable time to run for those test suites.
> One confusion I want to clear here - MIDRES doesn’t mean we will use
> medium containers but mid/less than the HIGHRES resources. Some Python
> tests cannot be run with medium containers, for example.
> Now whether some people still use HIGHRES or not I can neither confirm nor
> deny. I guess we can make a survey on that.
> I think improvement in choosing parallelism dynamically will be probably a
> great improvement too. We need to test. But if I recall correctly the
> moment we bump container size is when we bump to use way more credits. To
> be double-checked though, it’s been a while. Hope this info helps.
> On the topic of docs - I have one draft version for CircleCI
> doc/instructions but it has to be updated with all latest developments. I
> created it last year but never published as there was always something
> about to happen and for me to have to wait and update it accordingly. I can
> share with anyone willing to finish it so they don’t have to start from
> scratch. I don’t think I will have the time to do it in the next few weeks
> at least.
>
>
> On Tue, 18 Oct 2022 at 13:30, Andrés de la Peña <ad...@apache.org>
> wrote:
>
> The -h profile works but it spends a lot of resources for slightly faster
> results. The -m profile is better value in terms of speed per resources. I
> guess -h can be used if one wants to get results as soon as possible, no
> matter the cost. Ekaterina might be better informed than me, given her work
> on CASSANDRA-15712.
>
> In any case, the new multiplexer doesn't change the resources
> configuration at all. We might want to reevaluate that config in the
> future, and probably follow David's suggestion of deciding parallelism as a
> function of the number of tests to be run.
>
> On Tue, 18 Oct 2022 at 18:13, Josh McKenzie <jm...@apache.org> wrote:
>
>
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
> git diff to automatically detect the new or modified tests and will add
> them to the lists of tests to be repeated. The pre-commit workflow will
> automatically start repeated runs for these tests. The only exception to
> this are Python dtests, that should be specified manually.
>
> Of note: the -h profile should not be used (correct me if I'm wrong here
> Andres). Use -l for the free tier on circle or -m for paid.
>
> Will have some follow up tickets regarding job naming, default config
> type, and updating documentation shortly.
>
> On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote:
>
> Just to let you know that CASSANDRA-17939 has just been committed.
>
> It changes the way the CircleCI multiplexer works, in line with the recent
> changes in our release criteria:
>
> * The default number of repeated tests iterations is 500, except for long
> and upgrade tests.
> * It is possible to specify multiple test classes and methods to be
> repeated into the same config push. So patches altering dozens of tests
> won't require dozens of config pushes anymore.
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
> git diff to automatically detect the new or modified tests and will add
> them to the lists of tests to be repeated. The pre-commit workflow will
> automatically start repeated runs for these tests. The only exception to
> this are Python dtests, that should be specified manually.
> * The CircleCI jobs are rearranged so for every regular job there is a
> companion job to run the repeated tests associated to that job. Those
> companion jobs will only be visible if there are repeated tests to run.
> Here
> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d>
> is an example run with repeated tests for all the test suites, and here
> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412>
> is the same workflow without any repeated tests.
>
> Some documentation on how to use it can be found here:
> https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop
>
>
>
>
Re: New CircleCI test multiplexer
Posted by Josh McKenzie <jm...@apache.org>.
Thanks for the context Ekaterina. My Grand Plan is to get whatever wisdom David encoded into his script in terms of job bucketing and parallelism into the mainline circle config and all of us benefit from that.
On Tue, Oct 18, 2022, at 2:17 PM, Ekaterina Dimitrova wrote:
> First I want to take a moment to say thanks to Andres and Josh for this new improvement. I truly believe it will save us a lot of time and efforts in the flaky tests fight
>
> Pre-CASSANDRA-15712 there was default config for free tier and HIGHRES option(bumping all resources to max).
> We added MIDRES as a balanced version which gives you the chance to be able to run all tests but with less resources. What do I mean? It is still to be used by people with paid accounts but who want to consider the cost/resources spent. It was based on experimental runs which proved that with half the resources(compared to max out) for some suites we get similar and reasonable time to run for those test suites.
> One confusion I want to clear here - MIDRES doesn’t mean we will use medium containers but mid/less than the HIGHRES resources. Some Python tests cannot be run with medium containers, for example.
> Now whether some people still use HIGHRES or not I can neither confirm nor deny. I guess we can make a survey on that.
> I think improvement in choosing parallelism dynamically will be probably a great improvement too. We need to test. But if I recall correctly the moment we bump container size is when we bump to use way more credits. To be double-checked though, it’s been a while. Hope this info helps.
> On the topic of docs - I have one draft version for CircleCI doc/instructions but it has to be updated with all latest developments. I created it last year but never published as there was always something about to happen and for me to have to wait and update it accordingly. I can share with anyone willing to finish it so they don’t have to start from scratch. I don’t think I will have the time to do it in the next few weeks at least.
>
>
> On Tue, 18 Oct 2022 at 13:30, Andrés de la Peña <ad...@apache.org> wrote:
>> The -h profile works but it spends a lot of resources for slightly faster results. The -m profile is better value in terms of speed per resources. I guess -h can be used if one wants to get results as soon as possible, no matter the cost. Ekaterina might be better informed than me, given her work on CASSANDRA-15712.
>>
>> In any case, the new multiplexer doesn't change the resources configuration at all. We might want to reevaluate that config in the future, and probably follow David's suggestion of deciding parallelism as a function of the number of tests to be run.
>>
>> On Tue, 18 Oct 2022 at 18:13, Josh McKenzie <jm...@apache.org> wrote:
>>> __
>>>> * Running the .circleci/generate.sh script with -l/-m/-h flags will use git diff to automatically detect the new or modified tests and will add them to the lists of tests to be repeated. The pre-commit workflow will automatically start repeated runs for these tests. The only exception to this are Python dtests, that should be specified manually.
>>> Of note: the -h profile should not be used (correct me if I'm wrong here Andres). Use -l for the free tier on circle or -m for paid.
>>>
>>> Will have some follow up tickets regarding job naming, default config type, and updating documentation shortly.
>>>
>>> On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote:
>>>> Just to let you know that CASSANDRA-17939 has just been committed.
>>>>
>>>> It changes the way the CircleCI multiplexer works, in line with the recent changes in our release criteria:
>>>>
>>>> * The default number of repeated tests iterations is 500, except for long and upgrade tests.
>>>> * It is possible to specify multiple test classes and methods to be repeated into the same config push. So patches altering dozens of tests won't require dozens of config pushes anymore.
>>>> * Running the .circleci/generate.sh script with -l/-m/-h flags will use git diff to automatically detect the new or modified tests and will add them to the lists of tests to be repeated. The pre-commit workflow will automatically start repeated runs for these tests. The only exception to this are Python dtests, that should be specified manually.
>>>> * The CircleCI jobs are rearranged so for every regular job there is a companion job to run the repeated tests associated to that job. Those companion jobs will only be visible if there are repeated tests to run. Here <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d> is an example run with repeated tests for all the test suites, and here <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412> is the same workflow without any repeated tests.
>>>>
>>>> Some documentation on how to use it can be found here: https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop
>>>
Re: New CircleCI test multiplexer
Posted by Ekaterina Dimitrova <e....@gmail.com>.
First I want to take a moment to say thanks to Andres and Josh for this
new improvement. I truly believe it will save us a lot of time and efforts
in the flaky tests fight
Pre-CASSANDRA-15712 there was default config for free tier and HIGHRES
option(bumping all resources to max).
We added MIDRES as a balanced version which gives you the chance to be
able to run all tests but with less resources. What do I mean? It is still
to be used by people with paid accounts but who want to consider the
cost/resources spent. It was based on experimental runs which proved that
with half the resources(compared to max out) for some suites we get similar
and reasonable time to run for those test suites.
One confusion I want to clear here - MIDRES doesn’t mean we will use
medium containers but mid/less than the HIGHRES resources. Some Python
tests cannot be run with medium containers, for example.
Now whether some people still use HIGHRES or not I can neither confirm nor
deny. I guess we can make a survey on that.
I think improvement in choosing parallelism dynamically will be probably a
great improvement too. We need to test. But if I recall correctly the
moment we bump container size is when we bump to use way more credits. To
be double-checked though, it’s been a while. Hope this info helps.
On the topic of docs - I have one draft version for CircleCI
doc/instructions but it has to be updated with all latest developments. I
created it last year but never published as there was always something
about to happen and for me to have to wait and update it accordingly. I can
share with anyone willing to finish it so they don’t have to start from
scratch. I don’t think I will have the time to do it in the next few weeks
at least.
On Tue, 18 Oct 2022 at 13:30, Andrés de la Peña <ad...@apache.org>
wrote:
> The -h profile works but it spends a lot of resources for slightly faster
> results. The -m profile is better value in terms of speed per resources. I
> guess -h can be used if one wants to get results as soon as possible, no
> matter the cost. Ekaterina might be better informed than me, given her work
> on CASSANDRA-15712.
>
> In any case, the new multiplexer doesn't change the resources
> configuration at all. We might want to reevaluate that config in the
> future, and probably follow David's suggestion of deciding parallelism as a
> function of the number of tests to be run.
>
> On Tue, 18 Oct 2022 at 18:13, Josh McKenzie <jm...@apache.org> wrote:
>
>> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
>> git diff to automatically detect the new or modified tests and will add
>> them to the lists of tests to be repeated. The pre-commit workflow will
>> automatically start repeated runs for these tests. The only exception to
>> this are Python dtests, that should be specified manually.
>>
>> Of note: the -h profile should not be used (correct me if I'm wrong here
>> Andres). Use -l for the free tier on circle or -m for paid.
>>
>> Will have some follow up tickets regarding job naming, default config
>> type, and updating documentation shortly.
>>
>> On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote:
>>
>> Just to let you know that CASSANDRA-17939 has just been committed.
>>
>> It changes the way the CircleCI multiplexer works, in line with the
>> recent changes in our release criteria:
>>
>> * The default number of repeated tests iterations is 500, except for long
>> and upgrade tests.
>> * It is possible to specify multiple test classes and methods to be
>> repeated into the same config push. So patches altering dozens of tests
>> won't require dozens of config pushes anymore.
>> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
>> git diff to automatically detect the new or modified tests and will add
>> them to the lists of tests to be repeated. The pre-commit workflow will
>> automatically start repeated runs for these tests. The only exception to
>> this are Python dtests, that should be specified manually.
>> * The CircleCI jobs are rearranged so for every regular job there is a
>> companion job to run the repeated tests associated to that job. Those
>> companion jobs will only be visible if there are repeated tests to run.
>> Here
>> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d>
>> is an example run with repeated tests for all the test suites, and here
>> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412>
>> is the same workflow without any repeated tests.
>>
>> Some documentation on how to use it can be found here:
>> https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop
>>
>>
>>
Re: New CircleCI test multiplexer
Posted by Andrés de la Peña <ad...@apache.org>.
The -h profile works but it spends a lot of resources for slightly faster
results. The -m profile is better value in terms of speed per resources. I
guess -h can be used if one wants to get results as soon as possible, no
matter the cost. Ekaterina might be better informed than me, given her work
on CASSANDRA-15712.
In any case, the new multiplexer doesn't change the resources configuration
at all. We might want to reevaluate that config in the future, and probably
follow David's suggestion of deciding parallelism as a function of the
number of tests to be run.
On Tue, 18 Oct 2022 at 18:13, Josh McKenzie <jm...@apache.org> wrote:
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
> git diff to automatically detect the new or modified tests and will add
> them to the lists of tests to be repeated. The pre-commit workflow will
> automatically start repeated runs for these tests. The only exception to
> this are Python dtests, that should be specified manually.
>
> Of note: the -h profile should not be used (correct me if I'm wrong here
> Andres). Use -l for the free tier on circle or -m for paid.
>
> Will have some follow up tickets regarding job naming, default config
> type, and updating documentation shortly.
>
> On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote:
>
> Just to let you know that CASSANDRA-17939 has just been committed.
>
> It changes the way the CircleCI multiplexer works, in line with the recent
> changes in our release criteria:
>
> * The default number of repeated tests iterations is 500, except for long
> and upgrade tests.
> * It is possible to specify multiple test classes and methods to be
> repeated into the same config push. So patches altering dozens of tests
> won't require dozens of config pushes anymore.
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use
> git diff to automatically detect the new or modified tests and will add
> them to the lists of tests to be repeated. The pre-commit workflow will
> automatically start repeated runs for these tests. The only exception to
> this are Python dtests, that should be specified manually.
> * The CircleCI jobs are rearranged so for every regular job there is a
> companion job to run the repeated tests associated to that job. Those
> companion jobs will only be visible if there are repeated tests to run.
> Here
> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d>
> is an example run with repeated tests for all the test suites, and here
> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412>
> is the same workflow without any repeated tests.
>
> Some documentation on how to use it can be found here:
> https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop
>
>
>
Re: New CircleCI test multiplexer
Posted by Josh McKenzie <jm...@apache.org>.
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use git diff to automatically detect the new or modified tests and will add them to the lists of tests to be repeated. The pre-commit workflow will automatically start repeated runs for these tests. The only exception to this are Python dtests, that should be specified manually.
Of note: the -h profile should not be used (correct me if I'm wrong here Andres). Use -l for the free tier on circle or -m for paid.
Will have some follow up tickets regarding job naming, default config type, and updating documentation shortly.
On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote:
> Just to let you know that CASSANDRA-17939 has just been committed.
>
> It changes the way the CircleCI multiplexer works, in line with the recent changes in our release criteria:
>
> * The default number of repeated tests iterations is 500, except for long and upgrade tests.
> * It is possible to specify multiple test classes and methods to be repeated into the same config push. So patches altering dozens of tests won't require dozens of config pushes anymore.
> * Running the .circleci/generate.sh script with -l/-m/-h flags will use git diff to automatically detect the new or modified tests and will add them to the lists of tests to be repeated. The pre-commit workflow will automatically start repeated runs for these tests. The only exception to this are Python dtests, that should be specified manually.
> * The CircleCI jobs are rearranged so for every regular job there is a companion job to run the repeated tests associated to that job. Those companion jobs will only be visible if there are repeated tests to run. Here <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d> is an example run with repeated tests for all the test suites, and here <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412> is the same workflow without any repeated tests.
>
> Some documentation on how to use it can be found here: https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop