You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Andrés de la Peña <ad...@apache.org> on 2021/10/06 14:21:23 UTC
Re: Save CircleCI resources with optional test jobs
Hello all,
The changes in CircleCI proposed in the previous messages are implemented
in CASSANDRA-16882. They try to include the feedback from both the
reviewers and the Slack discussion at
https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
The patch modifies the CircleCI config to have two pairs of j8/j11
workflows:
* The javaX_pre-commit_tests workflows are meant to be used when a patch is
definitively ready for final review and/or commit. They have a single
approval step that starts all the basic tests, including unit tests,
dtests, etc. Additionally, it has approval steps for those tests that are
not generally required in every ticket, such as upgrade tests and the
multiplexer. This pair of workflows is quite similar to what we currently
have, and the main difference is that there is an approval step to start
the build.
* The javaX_separate_tests workflows are meant to be used in intermediate
steps and special cases such as fixing flaky tests. All the jobs in these
workflows have an individual approval step, so one can run any test job
without wasting resources in the others. For example, it makes possible to
repeatedly run a unit test without firing the entire JVM dtests.
Here is an example of how the workflows would look like:
https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
Hopefully this approach would help us to save resources in intermediate
development steps and special cases, while keeping the current concept of
mandatory tests.
If no one disagrees with this approach I'll commit the changes in a few
days.
On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
wrote:
> Hi,
>
> CASSANDRA-16882 has patches for any of the mentioned configurations aimed
> to save CircleCI resources.
>
> There are CircleCI runs showing how each approach would look like, plus an
> additional fourth option:
>
> 1. Make the entire workflow optional:
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> 2. Make all the test jobs optional:
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> 3. Make all the mandatory test jobs depend on a single optional job:
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> 4. Combine 2 and 3 to have approval jobs for both individual tests and the
> grouped mandatory tests:
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
>
> I think that the fourth option gives us more flexibility, because it
> allows us to start any combination of tests we want while it keeps the
> concept of required tests.
>
> On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <a....@gmail.com>
> wrote:
>
>> Hello all,
>>
>> The current CircleCI configuration automatically runs the unit tests, JVM
>> dtests and cqhshlib tests. This is done by default for every commit or,
>> with some configuration, for every push.
>>
>> Along the lifecycle of a ticket it is quite frequent to have multiple
>> commits and pushes, all running these test jobs. I'd say that frequently it
>> is not necessary to run the tests for some of those intermediate commits
>> and pushes. For example, one can show proofs of concept, or have multiple
>> rounds of review before actually running the tests. Running the tests for
>> every change can produce an unnecessary expense of CircleCI resources.
>>
>> I think we could make running those tests optional, as well as clearly
>> specifying in the documentation what are the tests runs that are mandatory
>> before actually committing. We could do this in different ways:
>>
>> 1. Make the entire CircleCI workflow optional, so the build job requires
>> manual approval. Once the build is approved the mandatory test jobs would
>> be run without any further approval, exactly as it's currently done.
>>
>> 2. Make all the test jobs optional, so every test job requires manual
>> approval, and the documentation specifies which tests are mandatory in the
>> final steps of a ticket.
>>
>> 3. Make all the mandatory test jobs depend on a single optional job, so
>> we have a single button to optionally run all the mandatory tests.
>>
>> I think any of these changes, or a combination of them, would
>> significantly reduce the usage of resources without making things less
>> tested. The only downside I can think of is that we would need some
>> additional clicks on the CircleCI GUI. What do you think?
>>
>
Re: Save CircleCI resources with optional test jobs
Posted by Andrés de la Peña <ad...@apache.org>.
Hi all,
I have just created CASSANDRA-17113 adding scripting options to select the
workflow to be used, trying to implement Benedict's suggestion.
It adds some flags to the existing .circleci/generate.sh config generation
script. A -p flag generates only the pre-commit workflows, whereas a -s
flag generates only the workflows with separate approval steps for each
test job. Both flags can be used together to generate the two pairs of
workflows. The default option is generating all the workflows, so users can
decide what workflow are they going to use in the CircleCI GUI, after
pushing their changes. We can easily change the workflows that are
generated by default to anything that suits us better.
Additionally, there is a -r flag that disables the first approval step of
the generated workflows. For the separate_tests workflows it means that the
build is automatically run, but the individual steps still need to be
manually approved in the GUI. For the pre-commit_tests workflows, the -r
flag will automatically run the build and the most relevant tests. That
way, users pushing a mostly ready patch and wanting to run the tests with
HIGHRES would probably want to generate their config file with generate.sh
-hpr.
What do you think?
On Thu, 21 Oct 2021 at 14:37, Andrés de la Peña <ad...@apache.org>
wrote:
> The two separate workflows try to serve both types of pushes, early and
> almost ready. The separate-test j8/j11 workflows are for those who push
> early patches, or special cases. The j8/j11 pre-commit workflows are for
> final stages, and it’s probably what those who push mostly-final patches
> would use. I think that these folks can just ignore the separate-tests
> workflows and use the “run everything” approval step on the pre-commit
> workflow.
>
> Focusing on those who push mostly final patches, and therefore use the
> pre-commit workflow, would there any value on running the build
> automatically and then requesting manual approval to run all the tests? If
> the patch is almost ready the tests have to be run, so I think there isn’t
> a difference between running the build step or not. In the end, we still
> have to press a single approval step, and if the build fails the tests
> aren’t going to start.
>
>
> El El jue, 21 oct 2021 a las 14:47, Oleksandr Petrov <
> oleksandr.petrov@gmail.com> escribió:
>
>> Thank you for responding,
>>
>> If there's an option that Benedict has suggested (to allow folks who push
>> mostly-final patches, and the folks who push rather-early patches and then
>> update them continuously, to coexist and be able to quickly switch between
>> configs) I'd be more in favour of this rather than just enabling a
>> build/compile step for everyone.
>>
>> On Wed, Oct 20, 2021 at 8:25 PM Andrés de la Peña <ad...@apache.org>
>> wrote:
>>
>> > Hi all,
>> >
>> > As Ekaterina has mentioned I’ll be OOO until Monday, and I won’t be
>> able to
>> > make any changes until then.
>> >
>> > Alex, I missed to answer your suggestion about building on every commit,
>> > apologies for that. The discussion was open on multiple fronts and I
>> > somehow missed that one. I can easily change it back to automatic
>> builds on
>> > Monday if we prefer to do so.
>> >
>> > The pre-commit workflows have a single button to start the build and all
>> > the tests that were previously run automatically. If we had automatic
>> > builds we would still have a button to start the tests. Thus, I think
>> that
>> > automatic builds in the pre-commit workflow would’t make a difference in
>> > terms of usability, and we would be wasting resources in intermediate
>> > commits.
>> >
>> > As for the workflows with separate approval steps, the automatic build
>> > would save us a click at the cost of wasting resources in some cases.
>> Since
>> > the cost of building is not that high it might make more sense to have
>> > automatic builds in these workflows. Alternatively, a detail that could
>> > improve things a bit in the separate-tests workflows is making the
>> approval
>> > steps for running the tests depend on the approval for the build. That
>> > wouldn’t save us any clicks but it would make impossible to miss the
>> build
>> > approval, since we would need to click it in order to enable the buttons
>> > starting the tests.
>> >
>> > It would be really great if the build started when one approves a
>> certain
>> > test job, but as it has been mentioned that doesn’t seem possible with
>> the
>> > current CircleCI features.
>> >
>> > Having a config that entirely satisfies the needs of everyone doesn’t
>> seem
>> > possible, and I also think that we’ll eventually need better tooling to
>> > generate the config file, although we still need to agree on a default
>> > config. CASSANDRA-16989 has recently added flags to the config
>> generation
>> > script allowing to swap resources and specify the environment vars. We
>> > should probably continue adding similar options to be able to manipulate
>> > the approval steps, parallelism, etc.
>> >
>> > Regards,
>> >
>> > El El mié, 20 oct 2021 a las 18:03, benedict@apache.org <
>> > benedict@apache.org>
>> > escribió:
>> >
>> > > Perhaps instead of a one-size-fits-all policy we should look to
>> improve
>> > > scripting here anyway. We already have to overwrite the config file,
>> it
>> > > would seem easy enough to instead invoke a script that enables the
>> > relevant
>> > > workflows?
>> > >
>> > > If we choose to just stick with a single workflow, I don’t have a
>> super
>> > > strong preference for one approach or the other.
>> > >
>> > > From: Stefan Miklosovic <st...@instaclustr.com>
>> > > Date: Wednesday, 20 October 2021 at 16:58
>> > > To: dev@cassandra.apache.org <de...@cassandra.apache.org>
>> > > Subject: Re: Save CircleCI resources with optional test jobs
>> > > Hi Ekaterina, all,
>> > >
>> > > If you eventually decide to switch it to automatic build on push (I
>> > > like how it is now though), I would appreciate it if there was some
>> > > option in generation scripts (generate.sh) so I could just have some
>> > > shortcut / alias which would disable this very easily.
>> > >
>> > > It would be very nice if there was also an option to specify
>> > > parallelism for highres nodes. My workflow so far was to 1) copy
>> > > highres to config.yml, open editor and replace "100" with "70" for
>> > > parallelism value. I am not sure what plan you guys are on but we can
>> > > at most spin sometime like 80 nodes or so at once.
>> > >
>> > > Thanks
>> > >
>> > > Stefan
>> > >
>> > > On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <
>> e.dimitrova@gmail.com
>> > >
>> > > wrote:
>> > > >
>> > > > Hi everyone,
>> > > >
>> > > > First of all, I know Andres is off these days so he might not be
>> > > following
>> > > > the thread today. I will try to clear the situation as the committer
>> > who
>> > > > approved the change.
>> > > >
>> > > > Alex, I would like to apologize to you! You are right, I personally
>> > > didn’t
>> > > > have your email (sometimes gmail plays games with the mailing list
>> > > emails,
>> > > > not sure why) and I assumed the lazy consensus. Andres sent email
>> that
>> > > this
>> > > > was implemented a month after the discussion was open and no one
>> > > responded
>> > > > here during that time. In his latest email he pointed to the Slack
>> > > > discussion that happened and where agreement was reached.
>> > > > I see your email was sent two days after he said it is implemented.
>> I
>> > > guess
>> > > > we had to be more explicit that this email announces lazy
>> consensus.
>> > > >
>> > > > Anyway, I am really sorry your email was missed and please, believe
>> me
>> > > when
>> > > > I say I would have considered it as the ticket was not committed
>> yet at
>> > > > that point! I am almost sure something similar happened to Andres
>> > knowing
>> > > > how extremely punctual and fair he is.
>> > > >
>> > > > The main goal was not to push on every commit as many times it
>> would be
>> > > > unnecessary waste of resources and spending credits. At the same
>> time
>> > new
>> > > > workflows with one precommit button were added to ensure people can
>> run
>> > > all
>> > > > tests we as a community require with one click before a commit.
>> Links
>> > > with
>> > > > the different versions/suggestions of the workflow are published on
>> the
>> > > > ticket.
>> > > >
>> > > > Yes, now we need to click on the build. It was agreed that many
>> times
>> > > > people push even just to preserve intermediate work and continue
>> later
>> > > > without caring of build or anything. If for some reason it is
>> important
>> > > to
>> > > > you or anyone else from the community to build on every commit, we
>> can
>> > > open
>> > > > a ticket to change that. It will save a click or two in the case of
>> the
>> > > > in-jvm upgrade tests. I would like to point out that Andres was also
>> > > > experimenting to ensure that the graph will be still as much
>> readable
>> > as
>> > > > possible.
>> > > >
>> > > > Benedict, on your comment of feature request - we can do that. I am
>> > also
>> > > > sceptic as you if that will happen but this doesn’t mean we can’t
>> give
>> > > it a
>> > > > try. Who knows, maybe others are also raising the topic and they
>> might
>> > > > consider it.
>> > > >
>> > > > Honestly, I personally support the current workflow and approval
>> > required
>> > > > but if it is not acceptable to others and skipping the build
>> approval
>> > > click
>> > > > is what others would prefer, I will open a ticket to restore that
>> part.
>> > > > Please let me know and one more time, apologize for missing your
>> email,
>> > > > Alex.
>> > > >
>> > > > Best regards,
>> > > > Ekaterina
>> > > >
>> > > >
>> > > >
>> > > > On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <
>> benedict@apache.org>
>> > > > wrote:
>> > > >
>> > > > > I think this would be fine if there were a way for approval of
>> later
>> > > steps
>> > > > > to trigger the build automatically. As it is we have to traverse
>> the
>> > > > > dependency graph manually, which is a bit weird.
>> > > > >
>> > > > > I can’t figure out a way to do this with the CircleCI API
>> however. It
>> > > > > might be a feature request, and perhaps we can at least trigger
>> the
>> > > build
>> > > > > until we get that (which may never happen).
>> > > > >
>> > > > > From: Oleksandr Petrov <ol...@gmail.com>
>> > > > > Date: Wednesday, 20 October 2021 at 13:35
>> > > > > To: dev <de...@cassandra.apache.org>
>> > > > > Subject: Re: Save CircleCI resources with optional test jobs
>> > > > > I thought this discussion was still ongoing, but it looks like
>> > > > > CASSANDRA-16882 is now committed.
>> > > > >
>> > > > > Could you give some context on why at least compilation is not
>> done
>> > on
>> > > > > every commit now?
>> > > > >
>> > > > >
>> > > > > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
>> > > > > oleksandr.petrov@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > I personally rarely push tickets/post a patch in an raw state,
>> but
>> > > since
>> > > > > I
>> > > > > > almost always have to approve jobs when it gets close to
>> commit, I
>> > > don't
>> > > > > > mind also confirming utest runs. I'd say it'd be good to run at
>> > very
>> > > > > least
>> > > > > > a compilation step on every commit. I think it's fine if
>> > > > > dtests/utests/jvm
>> > > > > > tests require approval.
>> > > > > >
>> > > > > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <
>> > > adelapena@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hello all,
>> > > > > >>
>> > > > > >> The changes in CircleCI proposed in the previous messages are
>> > > > > implemented
>> > > > > >> in CASSANDRA-16882. They try to include the feedback from both
>> the
>> > > > > >> reviewers and the Slack discussion at
>> > > > > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000
>> .
>> > > > > >>
>> > > > > >> The patch modifies the CircleCI config to have two pairs of
>> j8/j11
>> > > > > >> workflows:
>> > > > > >>
>> > > > > >> * The javaX_pre-commit_tests workflows are meant to be used
>> when a
>> > > patch
>> > > > > >> is
>> > > > > >> definitively ready for final review and/or commit. They have a
>> > > single
>> > > > > >> approval step that starts all the basic tests, including unit
>> > tests,
>> > > > > >> dtests, etc. Additionally, it has approval steps for those
>> tests
>> > > that
>> > > > > are
>> > > > > >> not generally required in every ticket, such as upgrade tests
>> and
>> > > the
>> > > > > >> multiplexer. This pair of workflows is quite similar to what we
>> > > > > currently
>> > > > > >> have, and the main difference is that there is an approval
>> step to
>> > > start
>> > > > > >> the build.
>> > > > > >>
>> > > > > >> * The javaX_separate_tests workflows are meant to be used in
>> > > > > intermediate
>> > > > > >> steps and special cases such as fixing flaky tests. All the
>> jobs
>> > in
>> > > > > these
>> > > > > >> workflows have an individual approval step, so one can run any
>> > test
>> > > job
>> > > > > >> without wasting resources in the others. For example, it makes
>> > > possible
>> > > > > to
>> > > > > >> repeatedly run a unit test without firing the entire JVM
>> dtests.
>> > > > > >>
>> > > > > >> Here is an example of how the workflows would look like:
>> > > > > >>
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
>> > > > > >>
>> > > > > >> Hopefully this approach would help us to save resources in
>> > > intermediate
>> > > > > >> development steps and special cases, while keeping the current
>> > > concept
>> > > > > of
>> > > > > >> mandatory tests.
>> > > > > >>
>> > > > > >> If no one disagrees with this approach I'll commit the changes
>> in
>> > a
>> > > few
>> > > > > >> days.
>> > > > > >>
>> > > > > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <
>> > > adelapena@apache.org>
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >> > Hi,
>> > > > > >> >
>> > > > > >> > CASSANDRA-16882 has patches for any of the mentioned
>> > > configurations
>> > > > > >> aimed
>> > > > > >> > to save CircleCI resources.
>> > > > > >> >
>> > > > > >> > There are CircleCI runs showing how each approach would look
>> > like,
>> > > > > plus
>> > > > > >> an
>> > > > > >> > additional fourth option:
>> > > > > >> >
>> > > > > >> > 1. Make the entire workflow optional:
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
>> > > > > >> > 2. Make all the test jobs optional:
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
>> > > > > >> > 3. Make all the mandatory test jobs depend on a single
>> optional
>> > > job:
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
>> > > > > >> > 4. Combine 2 and 3 to have approval jobs for both individual
>> > > tests and
>> > > > > >> the
>> > > > > >> > grouped mandatory tests:
>> > > > > >> >
>> > > > > >>
>> > > > >
>> > >
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
>> > > > > >> >
>> > > > > >> > I think that the fourth option gives us more flexibility,
>> > because
>> > > it
>> > > > > >> > allows us to start any combination of tests we want while it
>> > > keeps the
>> > > > > >> > concept of required tests.
>> > > > > >> >
>> > > > > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
>> > > > > >> a.penya.garcia@gmail.com>
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> >> Hello all,
>> > > > > >> >>
>> > > > > >> >> The current CircleCI configuration automatically runs the
>> unit
>> > > tests,
>> > > > > >> JVM
>> > > > > >> >> dtests and cqhshlib tests. This is done by default for every
>> > > commit
>> > > > > or,
>> > > > > >> >> with some configuration, for every push.
>> > > > > >> >>
>> > > > > >> >> Along the lifecycle of a ticket it is quite frequent to have
>> > > multiple
>> > > > > >> >> commits and pushes, all running these test jobs. I'd say
>> that
>> > > > > >> frequently it
>> > > > > >> >> is not necessary to run the tests for some of those
>> > intermediate
>> > > > > >> commits
>> > > > > >> >> and pushes. For example, one can show proofs of concept, or
>> > have
>> > > > > >> multiple
>> > > > > >> >> rounds of review before actually running the tests. Running
>> the
>> > > tests
>> > > > > >> for
>> > > > > >> >> every change can produce an unnecessary expense of CircleCI
>> > > > > resources.
>> > > > > >> >>
>> > > > > >> >> I think we could make running those tests optional, as well
>> as
>> > > > > clearly
>> > > > > >> >> specifying in the documentation what are the tests runs that
>> > are
>> > > > > >> mandatory
>> > > > > >> >> before actually committing. We could do this in different
>> ways:
>> > > > > >> >>
>> > > > > >> >> 1. Make the entire CircleCI workflow optional, so the build
>> job
>> > > > > >> requires
>> > > > > >> >> manual approval. Once the build is approved the mandatory
>> test
>> > > jobs
>> > > > > >> would
>> > > > > >> >> be run without any further approval, exactly as it's
>> currently
>> > > done.
>> > > > > >> >>
>> > > > > >> >> 2. Make all the test jobs optional, so every test job
>> requires
>> > > manual
>> > > > > >> >> approval, and the documentation specifies which tests are
>> > > mandatory
>> > > > > in
>> > > > > >> the
>> > > > > >> >> final steps of a ticket.
>> > > > > >> >>
>> > > > > >> >> 3. Make all the mandatory test jobs depend on a single
>> optional
>> > > job,
>> > > > > so
>> > > > > >> >> we have a single button to optionally run all the mandatory
>> > > tests.
>> > > > > >> >>
>> > > > > >> >> I think any of these changes, or a combination of them,
>> would
>> > > > > >> >> significantly reduce the usage of resources without making
>> > things
>> > > > > less
>> > > > > >> >> tested. The only downside I can think of is that we would
>> need
>> > > some
>> > > > > >> >> additional clicks on the CircleCI GUI. What do you think?
>> > > > > >> >>
>> > > > > >> >
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > alex p
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > alex p
>> > > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > For additional commands, e-mail: dev-help@cassandra.apache.org
>> > >
>> >
>>
>>
>> --
>> alex p
>>
>
Re: Save CircleCI resources with optional test jobs
Posted by Andrés de la Peña <ad...@apache.org>.
The two separate workflows try to serve both types of pushes, early and
almost ready. The separate-test j8/j11 workflows are for those who push
early patches, or special cases. The j8/j11 pre-commit workflows are for
final stages, and it’s probably what those who push mostly-final patches
would use. I think that these folks can just ignore the separate-tests
workflows and use the “run everything” approval step on the pre-commit
workflow.
Focusing on those who push mostly final patches, and therefore use the
pre-commit workflow, would there any value on running the build
automatically and then requesting manual approval to run all the tests? If
the patch is almost ready the tests have to be run, so I think there isn’t
a difference between running the build step or not. In the end, we still
have to press a single approval step, and if the build fails the tests
aren’t going to start.
El El jue, 21 oct 2021 a las 14:47, Oleksandr Petrov <
oleksandr.petrov@gmail.com> escribió:
> Thank you for responding,
>
> If there's an option that Benedict has suggested (to allow folks who push
> mostly-final patches, and the folks who push rather-early patches and then
> update them continuously, to coexist and be able to quickly switch between
> configs) I'd be more in favour of this rather than just enabling a
> build/compile step for everyone.
>
> On Wed, Oct 20, 2021 at 8:25 PM Andrés de la Peña <ad...@apache.org>
> wrote:
>
> > Hi all,
> >
> > As Ekaterina has mentioned I’ll be OOO until Monday, and I won’t be able
> to
> > make any changes until then.
> >
> > Alex, I missed to answer your suggestion about building on every commit,
> > apologies for that. The discussion was open on multiple fronts and I
> > somehow missed that one. I can easily change it back to automatic builds
> on
> > Monday if we prefer to do so.
> >
> > The pre-commit workflows have a single button to start the build and all
> > the tests that were previously run automatically. If we had automatic
> > builds we would still have a button to start the tests. Thus, I think
> that
> > automatic builds in the pre-commit workflow would’t make a difference in
> > terms of usability, and we would be wasting resources in intermediate
> > commits.
> >
> > As for the workflows with separate approval steps, the automatic build
> > would save us a click at the cost of wasting resources in some cases.
> Since
> > the cost of building is not that high it might make more sense to have
> > automatic builds in these workflows. Alternatively, a detail that could
> > improve things a bit in the separate-tests workflows is making the
> approval
> > steps for running the tests depend on the approval for the build. That
> > wouldn’t save us any clicks but it would make impossible to miss the
> build
> > approval, since we would need to click it in order to enable the buttons
> > starting the tests.
> >
> > It would be really great if the build started when one approves a certain
> > test job, but as it has been mentioned that doesn’t seem possible with
> the
> > current CircleCI features.
> >
> > Having a config that entirely satisfies the needs of everyone doesn’t
> seem
> > possible, and I also think that we’ll eventually need better tooling to
> > generate the config file, although we still need to agree on a default
> > config. CASSANDRA-16989 has recently added flags to the config generation
> > script allowing to swap resources and specify the environment vars. We
> > should probably continue adding similar options to be able to manipulate
> > the approval steps, parallelism, etc.
> >
> > Regards,
> >
> > El El mié, 20 oct 2021 a las 18:03, benedict@apache.org <
> > benedict@apache.org>
> > escribió:
> >
> > > Perhaps instead of a one-size-fits-all policy we should look to improve
> > > scripting here anyway. We already have to overwrite the config file, it
> > > would seem easy enough to instead invoke a script that enables the
> > relevant
> > > workflows?
> > >
> > > If we choose to just stick with a single workflow, I don’t have a super
> > > strong preference for one approach or the other.
> > >
> > > From: Stefan Miklosovic <st...@instaclustr.com>
> > > Date: Wednesday, 20 October 2021 at 16:58
> > > To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> > > Subject: Re: Save CircleCI resources with optional test jobs
> > > Hi Ekaterina, all,
> > >
> > > If you eventually decide to switch it to automatic build on push (I
> > > like how it is now though), I would appreciate it if there was some
> > > option in generation scripts (generate.sh) so I could just have some
> > > shortcut / alias which would disable this very easily.
> > >
> > > It would be very nice if there was also an option to specify
> > > parallelism for highres nodes. My workflow so far was to 1) copy
> > > highres to config.yml, open editor and replace "100" with "70" for
> > > parallelism value. I am not sure what plan you guys are on but we can
> > > at most spin sometime like 80 nodes or so at once.
> > >
> > > Thanks
> > >
> > > Stefan
> > >
> > > On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <
> e.dimitrova@gmail.com
> > >
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > First of all, I know Andres is off these days so he might not be
> > > following
> > > > the thread today. I will try to clear the situation as the committer
> > who
> > > > approved the change.
> > > >
> > > > Alex, I would like to apologize to you! You are right, I personally
> > > didn’t
> > > > have your email (sometimes gmail plays games with the mailing list
> > > emails,
> > > > not sure why) and I assumed the lazy consensus. Andres sent email
> that
> > > this
> > > > was implemented a month after the discussion was open and no one
> > > responded
> > > > here during that time. In his latest email he pointed to the Slack
> > > > discussion that happened and where agreement was reached.
> > > > I see your email was sent two days after he said it is implemented. I
> > > guess
> > > > we had to be more explicit that this email announces lazy consensus.
> > > >
> > > > Anyway, I am really sorry your email was missed and please, believe
> me
> > > when
> > > > I say I would have considered it as the ticket was not committed yet
> at
> > > > that point! I am almost sure something similar happened to Andres
> > knowing
> > > > how extremely punctual and fair he is.
> > > >
> > > > The main goal was not to push on every commit as many times it would
> be
> > > > unnecessary waste of resources and spending credits. At the same time
> > new
> > > > workflows with one precommit button were added to ensure people can
> run
> > > all
> > > > tests we as a community require with one click before a commit. Links
> > > with
> > > > the different versions/suggestions of the workflow are published on
> the
> > > > ticket.
> > > >
> > > > Yes, now we need to click on the build. It was agreed that many times
> > > > people push even just to preserve intermediate work and continue
> later
> > > > without caring of build or anything. If for some reason it is
> important
> > > to
> > > > you or anyone else from the community to build on every commit, we
> can
> > > open
> > > > a ticket to change that. It will save a click or two in the case of
> the
> > > > in-jvm upgrade tests. I would like to point out that Andres was also
> > > > experimenting to ensure that the graph will be still as much readable
> > as
> > > > possible.
> > > >
> > > > Benedict, on your comment of feature request - we can do that. I am
> > also
> > > > sceptic as you if that will happen but this doesn’t mean we can’t
> give
> > > it a
> > > > try. Who knows, maybe others are also raising the topic and they
> might
> > > > consider it.
> > > >
> > > > Honestly, I personally support the current workflow and approval
> > required
> > > > but if it is not acceptable to others and skipping the build approval
> > > click
> > > > is what others would prefer, I will open a ticket to restore that
> part.
> > > > Please let me know and one more time, apologize for missing your
> email,
> > > > Alex.
> > > >
> > > > Best regards,
> > > > Ekaterina
> > > >
> > > >
> > > >
> > > > On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <
> benedict@apache.org>
> > > > wrote:
> > > >
> > > > > I think this would be fine if there were a way for approval of
> later
> > > steps
> > > > > to trigger the build automatically. As it is we have to traverse
> the
> > > > > dependency graph manually, which is a bit weird.
> > > > >
> > > > > I can’t figure out a way to do this with the CircleCI API however.
> It
> > > > > might be a feature request, and perhaps we can at least trigger the
> > > build
> > > > > until we get that (which may never happen).
> > > > >
> > > > > From: Oleksandr Petrov <ol...@gmail.com>
> > > > > Date: Wednesday, 20 October 2021 at 13:35
> > > > > To: dev <de...@cassandra.apache.org>
> > > > > Subject: Re: Save CircleCI resources with optional test jobs
> > > > > I thought this discussion was still ongoing, but it looks like
> > > > > CASSANDRA-16882 is now committed.
> > > > >
> > > > > Could you give some context on why at least compilation is not done
> > on
> > > > > every commit now?
> > > > >
> > > > >
> > > > > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > > > > oleksandr.petrov@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I personally rarely push tickets/post a patch in an raw state,
> but
> > > since
> > > > > I
> > > > > > almost always have to approve jobs when it gets close to commit,
> I
> > > don't
> > > > > > mind also confirming utest runs. I'd say it'd be good to run at
> > very
> > > > > least
> > > > > > a compilation step on every commit. I think it's fine if
> > > > > dtests/utests/jvm
> > > > > > tests require approval.
> > > > > >
> > > > > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <
> > > adelapena@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> Hello all,
> > > > > >>
> > > > > >> The changes in CircleCI proposed in the previous messages are
> > > > > implemented
> > > > > >> in CASSANDRA-16882. They try to include the feedback from both
> the
> > > > > >> reviewers and the Slack discussion at
> > > > > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > > > > >>
> > > > > >> The patch modifies the CircleCI config to have two pairs of
> j8/j11
> > > > > >> workflows:
> > > > > >>
> > > > > >> * The javaX_pre-commit_tests workflows are meant to be used
> when a
> > > patch
> > > > > >> is
> > > > > >> definitively ready for final review and/or commit. They have a
> > > single
> > > > > >> approval step that starts all the basic tests, including unit
> > tests,
> > > > > >> dtests, etc. Additionally, it has approval steps for those tests
> > > that
> > > > > are
> > > > > >> not generally required in every ticket, such as upgrade tests
> and
> > > the
> > > > > >> multiplexer. This pair of workflows is quite similar to what we
> > > > > currently
> > > > > >> have, and the main difference is that there is an approval step
> to
> > > start
> > > > > >> the build.
> > > > > >>
> > > > > >> * The javaX_separate_tests workflows are meant to be used in
> > > > > intermediate
> > > > > >> steps and special cases such as fixing flaky tests. All the jobs
> > in
> > > > > these
> > > > > >> workflows have an individual approval step, so one can run any
> > test
> > > job
> > > > > >> without wasting resources in the others. For example, it makes
> > > possible
> > > > > to
> > > > > >> repeatedly run a unit test without firing the entire JVM dtests.
> > > > > >>
> > > > > >> Here is an example of how the workflows would look like:
> > > > > >>
> > > > > >>
> > > > >
> > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> > > > > >>
> > > > > >> Hopefully this approach would help us to save resources in
> > > intermediate
> > > > > >> development steps and special cases, while keeping the current
> > > concept
> > > > > of
> > > > > >> mandatory tests.
> > > > > >>
> > > > > >> If no one disagrees with this approach I'll commit the changes
> in
> > a
> > > few
> > > > > >> days.
> > > > > >>
> > > > > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <
> > > adelapena@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > CASSANDRA-16882 has patches for any of the mentioned
> > > configurations
> > > > > >> aimed
> > > > > >> > to save CircleCI resources.
> > > > > >> >
> > > > > >> > There are CircleCI runs showing how each approach would look
> > like,
> > > > > plus
> > > > > >> an
> > > > > >> > additional fourth option:
> > > > > >> >
> > > > > >> > 1. Make the entire workflow optional:
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > > > > >> > 2. Make all the test jobs optional:
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > > > > >> > 3. Make all the mandatory test jobs depend on a single
> optional
> > > job:
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > > > > >> > 4. Combine 2 and 3 to have approval jobs for both individual
> > > tests and
> > > > > >> the
> > > > > >> > grouped mandatory tests:
> > > > > >> >
> > > > > >>
> > > > >
> > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> > > > > >> >
> > > > > >> > I think that the fourth option gives us more flexibility,
> > because
> > > it
> > > > > >> > allows us to start any combination of tests we want while it
> > > keeps the
> > > > > >> > concept of required tests.
> > > > > >> >
> > > > > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> > > > > >> a.penya.garcia@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Hello all,
> > > > > >> >>
> > > > > >> >> The current CircleCI configuration automatically runs the
> unit
> > > tests,
> > > > > >> JVM
> > > > > >> >> dtests and cqhshlib tests. This is done by default for every
> > > commit
> > > > > or,
> > > > > >> >> with some configuration, for every push.
> > > > > >> >>
> > > > > >> >> Along the lifecycle of a ticket it is quite frequent to have
> > > multiple
> > > > > >> >> commits and pushes, all running these test jobs. I'd say that
> > > > > >> frequently it
> > > > > >> >> is not necessary to run the tests for some of those
> > intermediate
> > > > > >> commits
> > > > > >> >> and pushes. For example, one can show proofs of concept, or
> > have
> > > > > >> multiple
> > > > > >> >> rounds of review before actually running the tests. Running
> the
> > > tests
> > > > > >> for
> > > > > >> >> every change can produce an unnecessary expense of CircleCI
> > > > > resources.
> > > > > >> >>
> > > > > >> >> I think we could make running those tests optional, as well
> as
> > > > > clearly
> > > > > >> >> specifying in the documentation what are the tests runs that
> > are
> > > > > >> mandatory
> > > > > >> >> before actually committing. We could do this in different
> ways:
> > > > > >> >>
> > > > > >> >> 1. Make the entire CircleCI workflow optional, so the build
> job
> > > > > >> requires
> > > > > >> >> manual approval. Once the build is approved the mandatory
> test
> > > jobs
> > > > > >> would
> > > > > >> >> be run without any further approval, exactly as it's
> currently
> > > done.
> > > > > >> >>
> > > > > >> >> 2. Make all the test jobs optional, so every test job
> requires
> > > manual
> > > > > >> >> approval, and the documentation specifies which tests are
> > > mandatory
> > > > > in
> > > > > >> the
> > > > > >> >> final steps of a ticket.
> > > > > >> >>
> > > > > >> >> 3. Make all the mandatory test jobs depend on a single
> optional
> > > job,
> > > > > so
> > > > > >> >> we have a single button to optionally run all the mandatory
> > > tests.
> > > > > >> >>
> > > > > >> >> I think any of these changes, or a combination of them, would
> > > > > >> >> significantly reduce the usage of resources without making
> > things
> > > > > less
> > > > > >> >> tested. The only downside I can think of is that we would
> need
> > > some
> > > > > >> >> additional clicks on the CircleCI GUI. What do you think?
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > alex p
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > alex p
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > >
> >
>
>
> --
> alex p
>
Re: Save CircleCI resources with optional test jobs
Posted by Oleksandr Petrov <ol...@gmail.com>.
Thank you for responding,
If there's an option that Benedict has suggested (to allow folks who push
mostly-final patches, and the folks who push rather-early patches and then
update them continuously, to coexist and be able to quickly switch between
configs) I'd be more in favour of this rather than just enabling a
build/compile step for everyone.
On Wed, Oct 20, 2021 at 8:25 PM Andrés de la Peña <ad...@apache.org>
wrote:
> Hi all,
>
> As Ekaterina has mentioned I’ll be OOO until Monday, and I won’t be able to
> make any changes until then.
>
> Alex, I missed to answer your suggestion about building on every commit,
> apologies for that. The discussion was open on multiple fronts and I
> somehow missed that one. I can easily change it back to automatic builds on
> Monday if we prefer to do so.
>
> The pre-commit workflows have a single button to start the build and all
> the tests that were previously run automatically. If we had automatic
> builds we would still have a button to start the tests. Thus, I think that
> automatic builds in the pre-commit workflow would’t make a difference in
> terms of usability, and we would be wasting resources in intermediate
> commits.
>
> As for the workflows with separate approval steps, the automatic build
> would save us a click at the cost of wasting resources in some cases. Since
> the cost of building is not that high it might make more sense to have
> automatic builds in these workflows. Alternatively, a detail that could
> improve things a bit in the separate-tests workflows is making the approval
> steps for running the tests depend on the approval for the build. That
> wouldn’t save us any clicks but it would make impossible to miss the build
> approval, since we would need to click it in order to enable the buttons
> starting the tests.
>
> It would be really great if the build started when one approves a certain
> test job, but as it has been mentioned that doesn’t seem possible with the
> current CircleCI features.
>
> Having a config that entirely satisfies the needs of everyone doesn’t seem
> possible, and I also think that we’ll eventually need better tooling to
> generate the config file, although we still need to agree on a default
> config. CASSANDRA-16989 has recently added flags to the config generation
> script allowing to swap resources and specify the environment vars. We
> should probably continue adding similar options to be able to manipulate
> the approval steps, parallelism, etc.
>
> Regards,
>
> El El mié, 20 oct 2021 a las 18:03, benedict@apache.org <
> benedict@apache.org>
> escribió:
>
> > Perhaps instead of a one-size-fits-all policy we should look to improve
> > scripting here anyway. We already have to overwrite the config file, it
> > would seem easy enough to instead invoke a script that enables the
> relevant
> > workflows?
> >
> > If we choose to just stick with a single workflow, I don’t have a super
> > strong preference for one approach or the other.
> >
> > From: Stefan Miklosovic <st...@instaclustr.com>
> > Date: Wednesday, 20 October 2021 at 16:58
> > To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> > Subject: Re: Save CircleCI resources with optional test jobs
> > Hi Ekaterina, all,
> >
> > If you eventually decide to switch it to automatic build on push (I
> > like how it is now though), I would appreciate it if there was some
> > option in generation scripts (generate.sh) so I could just have some
> > shortcut / alias which would disable this very easily.
> >
> > It would be very nice if there was also an option to specify
> > parallelism for highres nodes. My workflow so far was to 1) copy
> > highres to config.yml, open editor and replace "100" with "70" for
> > parallelism value. I am not sure what plan you guys are on but we can
> > at most spin sometime like 80 nodes or so at once.
> >
> > Thanks
> >
> > Stefan
> >
> > On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <e.dimitrova@gmail.com
> >
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > First of all, I know Andres is off these days so he might not be
> > following
> > > the thread today. I will try to clear the situation as the committer
> who
> > > approved the change.
> > >
> > > Alex, I would like to apologize to you! You are right, I personally
> > didn’t
> > > have your email (sometimes gmail plays games with the mailing list
> > emails,
> > > not sure why) and I assumed the lazy consensus. Andres sent email that
> > this
> > > was implemented a month after the discussion was open and no one
> > responded
> > > here during that time. In his latest email he pointed to the Slack
> > > discussion that happened and where agreement was reached.
> > > I see your email was sent two days after he said it is implemented. I
> > guess
> > > we had to be more explicit that this email announces lazy consensus.
> > >
> > > Anyway, I am really sorry your email was missed and please, believe me
> > when
> > > I say I would have considered it as the ticket was not committed yet at
> > > that point! I am almost sure something similar happened to Andres
> knowing
> > > how extremely punctual and fair he is.
> > >
> > > The main goal was not to push on every commit as many times it would be
> > > unnecessary waste of resources and spending credits. At the same time
> new
> > > workflows with one precommit button were added to ensure people can run
> > all
> > > tests we as a community require with one click before a commit. Links
> > with
> > > the different versions/suggestions of the workflow are published on the
> > > ticket.
> > >
> > > Yes, now we need to click on the build. It was agreed that many times
> > > people push even just to preserve intermediate work and continue later
> > > without caring of build or anything. If for some reason it is important
> > to
> > > you or anyone else from the community to build on every commit, we can
> > open
> > > a ticket to change that. It will save a click or two in the case of the
> > > in-jvm upgrade tests. I would like to point out that Andres was also
> > > experimenting to ensure that the graph will be still as much readable
> as
> > > possible.
> > >
> > > Benedict, on your comment of feature request - we can do that. I am
> also
> > > sceptic as you if that will happen but this doesn’t mean we can’t give
> > it a
> > > try. Who knows, maybe others are also raising the topic and they might
> > > consider it.
> > >
> > > Honestly, I personally support the current workflow and approval
> required
> > > but if it is not acceptable to others and skipping the build approval
> > click
> > > is what others would prefer, I will open a ticket to restore that part.
> > > Please let me know and one more time, apologize for missing your email,
> > > Alex.
> > >
> > > Best regards,
> > > Ekaterina
> > >
> > >
> > >
> > > On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <be...@apache.org>
> > > wrote:
> > >
> > > > I think this would be fine if there were a way for approval of later
> > steps
> > > > to trigger the build automatically. As it is we have to traverse the
> > > > dependency graph manually, which is a bit weird.
> > > >
> > > > I can’t figure out a way to do this with the CircleCI API however. It
> > > > might be a feature request, and perhaps we can at least trigger the
> > build
> > > > until we get that (which may never happen).
> > > >
> > > > From: Oleksandr Petrov <ol...@gmail.com>
> > > > Date: Wednesday, 20 October 2021 at 13:35
> > > > To: dev <de...@cassandra.apache.org>
> > > > Subject: Re: Save CircleCI resources with optional test jobs
> > > > I thought this discussion was still ongoing, but it looks like
> > > > CASSANDRA-16882 is now committed.
> > > >
> > > > Could you give some context on why at least compilation is not done
> on
> > > > every commit now?
> > > >
> > > >
> > > > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > > > oleksandr.petrov@gmail.com>
> > > > wrote:
> > > >
> > > > > I personally rarely push tickets/post a patch in an raw state, but
> > since
> > > > I
> > > > > almost always have to approve jobs when it gets close to commit, I
> > don't
> > > > > mind also confirming utest runs. I'd say it'd be good to run at
> very
> > > > least
> > > > > a compilation step on every commit. I think it's fine if
> > > > dtests/utests/jvm
> > > > > tests require approval.
> > > > >
> > > > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <
> > adelapena@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Hello all,
> > > > >>
> > > > >> The changes in CircleCI proposed in the previous messages are
> > > > implemented
> > > > >> in CASSANDRA-16882. They try to include the feedback from both the
> > > > >> reviewers and the Slack discussion at
> > > > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > > > >>
> > > > >> The patch modifies the CircleCI config to have two pairs of j8/j11
> > > > >> workflows:
> > > > >>
> > > > >> * The javaX_pre-commit_tests workflows are meant to be used when a
> > patch
> > > > >> is
> > > > >> definitively ready for final review and/or commit. They have a
> > single
> > > > >> approval step that starts all the basic tests, including unit
> tests,
> > > > >> dtests, etc. Additionally, it has approval steps for those tests
> > that
> > > > are
> > > > >> not generally required in every ticket, such as upgrade tests and
> > the
> > > > >> multiplexer. This pair of workflows is quite similar to what we
> > > > currently
> > > > >> have, and the main difference is that there is an approval step to
> > start
> > > > >> the build.
> > > > >>
> > > > >> * The javaX_separate_tests workflows are meant to be used in
> > > > intermediate
> > > > >> steps and special cases such as fixing flaky tests. All the jobs
> in
> > > > these
> > > > >> workflows have an individual approval step, so one can run any
> test
> > job
> > > > >> without wasting resources in the others. For example, it makes
> > possible
> > > > to
> > > > >> repeatedly run a unit test without firing the entire JVM dtests.
> > > > >>
> > > > >> Here is an example of how the workflows would look like:
> > > > >>
> > > > >>
> > > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> > > > >>
> > > > >> Hopefully this approach would help us to save resources in
> > intermediate
> > > > >> development steps and special cases, while keeping the current
> > concept
> > > > of
> > > > >> mandatory tests.
> > > > >>
> > > > >> If no one disagrees with this approach I'll commit the changes in
> a
> > few
> > > > >> days.
> > > > >>
> > > > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <
> > adelapena@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > CASSANDRA-16882 has patches for any of the mentioned
> > configurations
> > > > >> aimed
> > > > >> > to save CircleCI resources.
> > > > >> >
> > > > >> > There are CircleCI runs showing how each approach would look
> like,
> > > > plus
> > > > >> an
> > > > >> > additional fourth option:
> > > > >> >
> > > > >> > 1. Make the entire workflow optional:
> > > > >> >
> > > > >>
> > > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > > > >> > 2. Make all the test jobs optional:
> > > > >> >
> > > > >>
> > > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > > > >> > 3. Make all the mandatory test jobs depend on a single optional
> > job:
> > > > >> >
> > > > >>
> > > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > > > >> > 4. Combine 2 and 3 to have approval jobs for both individual
> > tests and
> > > > >> the
> > > > >> > grouped mandatory tests:
> > > > >> >
> > > > >>
> > > >
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> > > > >> >
> > > > >> > I think that the fourth option gives us more flexibility,
> because
> > it
> > > > >> > allows us to start any combination of tests we want while it
> > keeps the
> > > > >> > concept of required tests.
> > > > >> >
> > > > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> > > > >> a.penya.garcia@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Hello all,
> > > > >> >>
> > > > >> >> The current CircleCI configuration automatically runs the unit
> > tests,
> > > > >> JVM
> > > > >> >> dtests and cqhshlib tests. This is done by default for every
> > commit
> > > > or,
> > > > >> >> with some configuration, for every push.
> > > > >> >>
> > > > >> >> Along the lifecycle of a ticket it is quite frequent to have
> > multiple
> > > > >> >> commits and pushes, all running these test jobs. I'd say that
> > > > >> frequently it
> > > > >> >> is not necessary to run the tests for some of those
> intermediate
> > > > >> commits
> > > > >> >> and pushes. For example, one can show proofs of concept, or
> have
> > > > >> multiple
> > > > >> >> rounds of review before actually running the tests. Running the
> > tests
> > > > >> for
> > > > >> >> every change can produce an unnecessary expense of CircleCI
> > > > resources.
> > > > >> >>
> > > > >> >> I think we could make running those tests optional, as well as
> > > > clearly
> > > > >> >> specifying in the documentation what are the tests runs that
> are
> > > > >> mandatory
> > > > >> >> before actually committing. We could do this in different ways:
> > > > >> >>
> > > > >> >> 1. Make the entire CircleCI workflow optional, so the build job
> > > > >> requires
> > > > >> >> manual approval. Once the build is approved the mandatory test
> > jobs
> > > > >> would
> > > > >> >> be run without any further approval, exactly as it's currently
> > done.
> > > > >> >>
> > > > >> >> 2. Make all the test jobs optional, so every test job requires
> > manual
> > > > >> >> approval, and the documentation specifies which tests are
> > mandatory
> > > > in
> > > > >> the
> > > > >> >> final steps of a ticket.
> > > > >> >>
> > > > >> >> 3. Make all the mandatory test jobs depend on a single optional
> > job,
> > > > so
> > > > >> >> we have a single button to optionally run all the mandatory
> > tests.
> > > > >> >>
> > > > >> >> I think any of these changes, or a combination of them, would
> > > > >> >> significantly reduce the usage of resources without making
> things
> > > > less
> > > > >> >> tested. The only downside I can think of is that we would need
> > some
> > > > >> >> additional clicks on the CircleCI GUI. What do you think?
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > alex p
> > > > >
> > > >
> > > >
> > > > --
> > > > alex p
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
>
--
alex p
Re: Save CircleCI resources with optional test jobs
Posted by Andrés de la Peña <ad...@apache.org>.
Hi all,
As Ekaterina has mentioned I’ll be OOO until Monday, and I won’t be able to
make any changes until then.
Alex, I missed to answer your suggestion about building on every commit,
apologies for that. The discussion was open on multiple fronts and I
somehow missed that one. I can easily change it back to automatic builds on
Monday if we prefer to do so.
The pre-commit workflows have a single button to start the build and all
the tests that were previously run automatically. If we had automatic
builds we would still have a button to start the tests. Thus, I think that
automatic builds in the pre-commit workflow would’t make a difference in
terms of usability, and we would be wasting resources in intermediate
commits.
As for the workflows with separate approval steps, the automatic build
would save us a click at the cost of wasting resources in some cases. Since
the cost of building is not that high it might make more sense to have
automatic builds in these workflows. Alternatively, a detail that could
improve things a bit in the separate-tests workflows is making the approval
steps for running the tests depend on the approval for the build. That
wouldn’t save us any clicks but it would make impossible to miss the build
approval, since we would need to click it in order to enable the buttons
starting the tests.
It would be really great if the build started when one approves a certain
test job, but as it has been mentioned that doesn’t seem possible with the
current CircleCI features.
Having a config that entirely satisfies the needs of everyone doesn’t seem
possible, and I also think that we’ll eventually need better tooling to
generate the config file, although we still need to agree on a default
config. CASSANDRA-16989 has recently added flags to the config generation
script allowing to swap resources and specify the environment vars. We
should probably continue adding similar options to be able to manipulate
the approval steps, parallelism, etc.
Regards,
El El mié, 20 oct 2021 a las 18:03, benedict@apache.org <be...@apache.org>
escribió:
> Perhaps instead of a one-size-fits-all policy we should look to improve
> scripting here anyway. We already have to overwrite the config file, it
> would seem easy enough to instead invoke a script that enables the relevant
> workflows?
>
> If we choose to just stick with a single workflow, I don’t have a super
> strong preference for one approach or the other.
>
> From: Stefan Miklosovic <st...@instaclustr.com>
> Date: Wednesday, 20 October 2021 at 16:58
> To: dev@cassandra.apache.org <de...@cassandra.apache.org>
> Subject: Re: Save CircleCI resources with optional test jobs
> Hi Ekaterina, all,
>
> If you eventually decide to switch it to automatic build on push (I
> like how it is now though), I would appreciate it if there was some
> option in generation scripts (generate.sh) so I could just have some
> shortcut / alias which would disable this very easily.
>
> It would be very nice if there was also an option to specify
> parallelism for highres nodes. My workflow so far was to 1) copy
> highres to config.yml, open editor and replace "100" with "70" for
> parallelism value. I am not sure what plan you guys are on but we can
> at most spin sometime like 80 nodes or so at once.
>
> Thanks
>
> Stefan
>
> On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <e....@gmail.com>
> wrote:
> >
> > Hi everyone,
> >
> > First of all, I know Andres is off these days so he might not be
> following
> > the thread today. I will try to clear the situation as the committer who
> > approved the change.
> >
> > Alex, I would like to apologize to you! You are right, I personally
> didn’t
> > have your email (sometimes gmail plays games with the mailing list
> emails,
> > not sure why) and I assumed the lazy consensus. Andres sent email that
> this
> > was implemented a month after the discussion was open and no one
> responded
> > here during that time. In his latest email he pointed to the Slack
> > discussion that happened and where agreement was reached.
> > I see your email was sent two days after he said it is implemented. I
> guess
> > we had to be more explicit that this email announces lazy consensus.
> >
> > Anyway, I am really sorry your email was missed and please, believe me
> when
> > I say I would have considered it as the ticket was not committed yet at
> > that point! I am almost sure something similar happened to Andres knowing
> > how extremely punctual and fair he is.
> >
> > The main goal was not to push on every commit as many times it would be
> > unnecessary waste of resources and spending credits. At the same time new
> > workflows with one precommit button were added to ensure people can run
> all
> > tests we as a community require with one click before a commit. Links
> with
> > the different versions/suggestions of the workflow are published on the
> > ticket.
> >
> > Yes, now we need to click on the build. It was agreed that many times
> > people push even just to preserve intermediate work and continue later
> > without caring of build or anything. If for some reason it is important
> to
> > you or anyone else from the community to build on every commit, we can
> open
> > a ticket to change that. It will save a click or two in the case of the
> > in-jvm upgrade tests. I would like to point out that Andres was also
> > experimenting to ensure that the graph will be still as much readable as
> > possible.
> >
> > Benedict, on your comment of feature request - we can do that. I am also
> > sceptic as you if that will happen but this doesn’t mean we can’t give
> it a
> > try. Who knows, maybe others are also raising the topic and they might
> > consider it.
> >
> > Honestly, I personally support the current workflow and approval required
> > but if it is not acceptable to others and skipping the build approval
> click
> > is what others would prefer, I will open a ticket to restore that part.
> > Please let me know and one more time, apologize for missing your email,
> > Alex.
> >
> > Best regards,
> > Ekaterina
> >
> >
> >
> > On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <be...@apache.org>
> > wrote:
> >
> > > I think this would be fine if there were a way for approval of later
> steps
> > > to trigger the build automatically. As it is we have to traverse the
> > > dependency graph manually, which is a bit weird.
> > >
> > > I can’t figure out a way to do this with the CircleCI API however. It
> > > might be a feature request, and perhaps we can at least trigger the
> build
> > > until we get that (which may never happen).
> > >
> > > From: Oleksandr Petrov <ol...@gmail.com>
> > > Date: Wednesday, 20 October 2021 at 13:35
> > > To: dev <de...@cassandra.apache.org>
> > > Subject: Re: Save CircleCI resources with optional test jobs
> > > I thought this discussion was still ongoing, but it looks like
> > > CASSANDRA-16882 is now committed.
> > >
> > > Could you give some context on why at least compilation is not done on
> > > every commit now?
> > >
> > >
> > > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > > oleksandr.petrov@gmail.com>
> > > wrote:
> > >
> > > > I personally rarely push tickets/post a patch in an raw state, but
> since
> > > I
> > > > almost always have to approve jobs when it gets close to commit, I
> don't
> > > > mind also confirming utest runs. I'd say it'd be good to run at very
> > > least
> > > > a compilation step on every commit. I think it's fine if
> > > dtests/utests/jvm
> > > > tests require approval.
> > > >
> > > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <
> adelapena@apache.org>
> > > > wrote:
> > > >
> > > >> Hello all,
> > > >>
> > > >> The changes in CircleCI proposed in the previous messages are
> > > implemented
> > > >> in CASSANDRA-16882. They try to include the feedback from both the
> > > >> reviewers and the Slack discussion at
> > > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > > >>
> > > >> The patch modifies the CircleCI config to have two pairs of j8/j11
> > > >> workflows:
> > > >>
> > > >> * The javaX_pre-commit_tests workflows are meant to be used when a
> patch
> > > >> is
> > > >> definitively ready for final review and/or commit. They have a
> single
> > > >> approval step that starts all the basic tests, including unit tests,
> > > >> dtests, etc. Additionally, it has approval steps for those tests
> that
> > > are
> > > >> not generally required in every ticket, such as upgrade tests and
> the
> > > >> multiplexer. This pair of workflows is quite similar to what we
> > > currently
> > > >> have, and the main difference is that there is an approval step to
> start
> > > >> the build.
> > > >>
> > > >> * The javaX_separate_tests workflows are meant to be used in
> > > intermediate
> > > >> steps and special cases such as fixing flaky tests. All the jobs in
> > > these
> > > >> workflows have an individual approval step, so one can run any test
> job
> > > >> without wasting resources in the others. For example, it makes
> possible
> > > to
> > > >> repeatedly run a unit test without firing the entire JVM dtests.
> > > >>
> > > >> Here is an example of how the workflows would look like:
> > > >>
> > > >>
> > >
> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> > > >>
> > > >> Hopefully this approach would help us to save resources in
> intermediate
> > > >> development steps and special cases, while keeping the current
> concept
> > > of
> > > >> mandatory tests.
> > > >>
> > > >> If no one disagrees with this approach I'll commit the changes in a
> few
> > > >> days.
> > > >>
> > > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <
> adelapena@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > CASSANDRA-16882 has patches for any of the mentioned
> configurations
> > > >> aimed
> > > >> > to save CircleCI resources.
> > > >> >
> > > >> > There are CircleCI runs showing how each approach would look like,
> > > plus
> > > >> an
> > > >> > additional fourth option:
> > > >> >
> > > >> > 1. Make the entire workflow optional:
> > > >> >
> > > >>
> > >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > > >> > 2. Make all the test jobs optional:
> > > >> >
> > > >>
> > >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > > >> > 3. Make all the mandatory test jobs depend on a single optional
> job:
> > > >> >
> > > >>
> > >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > > >> > 4. Combine 2 and 3 to have approval jobs for both individual
> tests and
> > > >> the
> > > >> > grouped mandatory tests:
> > > >> >
> > > >>
> > >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> > > >> >
> > > >> > I think that the fourth option gives us more flexibility, because
> it
> > > >> > allows us to start any combination of tests we want while it
> keeps the
> > > >> > concept of required tests.
> > > >> >
> > > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> > > >> a.penya.garcia@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Hello all,
> > > >> >>
> > > >> >> The current CircleCI configuration automatically runs the unit
> tests,
> > > >> JVM
> > > >> >> dtests and cqhshlib tests. This is done by default for every
> commit
> > > or,
> > > >> >> with some configuration, for every push.
> > > >> >>
> > > >> >> Along the lifecycle of a ticket it is quite frequent to have
> multiple
> > > >> >> commits and pushes, all running these test jobs. I'd say that
> > > >> frequently it
> > > >> >> is not necessary to run the tests for some of those intermediate
> > > >> commits
> > > >> >> and pushes. For example, one can show proofs of concept, or have
> > > >> multiple
> > > >> >> rounds of review before actually running the tests. Running the
> tests
> > > >> for
> > > >> >> every change can produce an unnecessary expense of CircleCI
> > > resources.
> > > >> >>
> > > >> >> I think we could make running those tests optional, as well as
> > > clearly
> > > >> >> specifying in the documentation what are the tests runs that are
> > > >> mandatory
> > > >> >> before actually committing. We could do this in different ways:
> > > >> >>
> > > >> >> 1. Make the entire CircleCI workflow optional, so the build job
> > > >> requires
> > > >> >> manual approval. Once the build is approved the mandatory test
> jobs
> > > >> would
> > > >> >> be run without any further approval, exactly as it's currently
> done.
> > > >> >>
> > > >> >> 2. Make all the test jobs optional, so every test job requires
> manual
> > > >> >> approval, and the documentation specifies which tests are
> mandatory
> > > in
> > > >> the
> > > >> >> final steps of a ticket.
> > > >> >>
> > > >> >> 3. Make all the mandatory test jobs depend on a single optional
> job,
> > > so
> > > >> >> we have a single button to optionally run all the mandatory
> tests.
> > > >> >>
> > > >> >> I think any of these changes, or a combination of them, would
> > > >> >> significantly reduce the usage of resources without making things
> > > less
> > > >> >> tested. The only downside I can think of is that we would need
> some
> > > >> >> additional clicks on the CircleCI GUI. What do you think?
> > > >> >>
> > > >> >
> > > >>
> > > >
> > > >
> > > > --
> > > > alex p
> > > >
> > >
> > >
> > > --
> > > alex p
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
Re: Save CircleCI resources with optional test jobs
Posted by "benedict@apache.org" <be...@apache.org>.
Perhaps instead of a one-size-fits-all policy we should look to improve scripting here anyway. We already have to overwrite the config file, it would seem easy enough to instead invoke a script that enables the relevant workflows?
If we choose to just stick with a single workflow, I don’t have a super strong preference for one approach or the other.
From: Stefan Miklosovic <st...@instaclustr.com>
Date: Wednesday, 20 October 2021 at 16:58
To: dev@cassandra.apache.org <de...@cassandra.apache.org>
Subject: Re: Save CircleCI resources with optional test jobs
Hi Ekaterina, all,
If you eventually decide to switch it to automatic build on push (I
like how it is now though), I would appreciate it if there was some
option in generation scripts (generate.sh) so I could just have some
shortcut / alias which would disable this very easily.
It would be very nice if there was also an option to specify
parallelism for highres nodes. My workflow so far was to 1) copy
highres to config.yml, open editor and replace "100" with "70" for
parallelism value. I am not sure what plan you guys are on but we can
at most spin sometime like 80 nodes or so at once.
Thanks
Stefan
On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <e....@gmail.com> wrote:
>
> Hi everyone,
>
> First of all, I know Andres is off these days so he might not be following
> the thread today. I will try to clear the situation as the committer who
> approved the change.
>
> Alex, I would like to apologize to you! You are right, I personally didn’t
> have your email (sometimes gmail plays games with the mailing list emails,
> not sure why) and I assumed the lazy consensus. Andres sent email that this
> was implemented a month after the discussion was open and no one responded
> here during that time. In his latest email he pointed to the Slack
> discussion that happened and where agreement was reached.
> I see your email was sent two days after he said it is implemented. I guess
> we had to be more explicit that this email announces lazy consensus.
>
> Anyway, I am really sorry your email was missed and please, believe me when
> I say I would have considered it as the ticket was not committed yet at
> that point! I am almost sure something similar happened to Andres knowing
> how extremely punctual and fair he is.
>
> The main goal was not to push on every commit as many times it would be
> unnecessary waste of resources and spending credits. At the same time new
> workflows with one precommit button were added to ensure people can run all
> tests we as a community require with one click before a commit. Links with
> the different versions/suggestions of the workflow are published on the
> ticket.
>
> Yes, now we need to click on the build. It was agreed that many times
> people push even just to preserve intermediate work and continue later
> without caring of build or anything. If for some reason it is important to
> you or anyone else from the community to build on every commit, we can open
> a ticket to change that. It will save a click or two in the case of the
> in-jvm upgrade tests. I would like to point out that Andres was also
> experimenting to ensure that the graph will be still as much readable as
> possible.
>
> Benedict, on your comment of feature request - we can do that. I am also
> sceptic as you if that will happen but this doesn’t mean we can’t give it a
> try. Who knows, maybe others are also raising the topic and they might
> consider it.
>
> Honestly, I personally support the current workflow and approval required
> but if it is not acceptable to others and skipping the build approval click
> is what others would prefer, I will open a ticket to restore that part.
> Please let me know and one more time, apologize for missing your email,
> Alex.
>
> Best regards,
> Ekaterina
>
>
>
> On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <be...@apache.org>
> wrote:
>
> > I think this would be fine if there were a way for approval of later steps
> > to trigger the build automatically. As it is we have to traverse the
> > dependency graph manually, which is a bit weird.
> >
> > I can’t figure out a way to do this with the CircleCI API however. It
> > might be a feature request, and perhaps we can at least trigger the build
> > until we get that (which may never happen).
> >
> > From: Oleksandr Petrov <ol...@gmail.com>
> > Date: Wednesday, 20 October 2021 at 13:35
> > To: dev <de...@cassandra.apache.org>
> > Subject: Re: Save CircleCI resources with optional test jobs
> > I thought this discussion was still ongoing, but it looks like
> > CASSANDRA-16882 is now committed.
> >
> > Could you give some context on why at least compilation is not done on
> > every commit now?
> >
> >
> > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > oleksandr.petrov@gmail.com>
> > wrote:
> >
> > > I personally rarely push tickets/post a patch in an raw state, but since
> > I
> > > almost always have to approve jobs when it gets close to commit, I don't
> > > mind also confirming utest runs. I'd say it'd be good to run at very
> > least
> > > a compilation step on every commit. I think it's fine if
> > dtests/utests/jvm
> > > tests require approval.
> > >
> > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
> > > wrote:
> > >
> > >> Hello all,
> > >>
> > >> The changes in CircleCI proposed in the previous messages are
> > implemented
> > >> in CASSANDRA-16882. They try to include the feedback from both the
> > >> reviewers and the Slack discussion at
> > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > >>
> > >> The patch modifies the CircleCI config to have two pairs of j8/j11
> > >> workflows:
> > >>
> > >> * The javaX_pre-commit_tests workflows are meant to be used when a patch
> > >> is
> > >> definitively ready for final review and/or commit. They have a single
> > >> approval step that starts all the basic tests, including unit tests,
> > >> dtests, etc. Additionally, it has approval steps for those tests that
> > are
> > >> not generally required in every ticket, such as upgrade tests and the
> > >> multiplexer. This pair of workflows is quite similar to what we
> > currently
> > >> have, and the main difference is that there is an approval step to start
> > >> the build.
> > >>
> > >> * The javaX_separate_tests workflows are meant to be used in
> > intermediate
> > >> steps and special cases such as fixing flaky tests. All the jobs in
> > these
> > >> workflows have an individual approval step, so one can run any test job
> > >> without wasting resources in the others. For example, it makes possible
> > to
> > >> repeatedly run a unit test without firing the entire JVM dtests.
> > >>
> > >> Here is an example of how the workflows would look like:
> > >>
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> > >>
> > >> Hopefully this approach would help us to save resources in intermediate
> > >> development steps and special cases, while keeping the current concept
> > of
> > >> mandatory tests.
> > >>
> > >> If no one disagrees with this approach I'll commit the changes in a few
> > >> days.
> > >>
> > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > CASSANDRA-16882 has patches for any of the mentioned configurations
> > >> aimed
> > >> > to save CircleCI resources.
> > >> >
> > >> > There are CircleCI runs showing how each approach would look like,
> > plus
> > >> an
> > >> > additional fourth option:
> > >> >
> > >> > 1. Make the entire workflow optional:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > >> > 2. Make all the test jobs optional:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > >> > 3. Make all the mandatory test jobs depend on a single optional job:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > >> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
> > >> the
> > >> > grouped mandatory tests:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> > >> >
> > >> > I think that the fourth option gives us more flexibility, because it
> > >> > allows us to start any combination of tests we want while it keeps the
> > >> > concept of required tests.
> > >> >
> > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> > >> a.penya.garcia@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hello all,
> > >> >>
> > >> >> The current CircleCI configuration automatically runs the unit tests,
> > >> JVM
> > >> >> dtests and cqhshlib tests. This is done by default for every commit
> > or,
> > >> >> with some configuration, for every push.
> > >> >>
> > >> >> Along the lifecycle of a ticket it is quite frequent to have multiple
> > >> >> commits and pushes, all running these test jobs. I'd say that
> > >> frequently it
> > >> >> is not necessary to run the tests for some of those intermediate
> > >> commits
> > >> >> and pushes. For example, one can show proofs of concept, or have
> > >> multiple
> > >> >> rounds of review before actually running the tests. Running the tests
> > >> for
> > >> >> every change can produce an unnecessary expense of CircleCI
> > resources.
> > >> >>
> > >> >> I think we could make running those tests optional, as well as
> > clearly
> > >> >> specifying in the documentation what are the tests runs that are
> > >> mandatory
> > >> >> before actually committing. We could do this in different ways:
> > >> >>
> > >> >> 1. Make the entire CircleCI workflow optional, so the build job
> > >> requires
> > >> >> manual approval. Once the build is approved the mandatory test jobs
> > >> would
> > >> >> be run without any further approval, exactly as it's currently done.
> > >> >>
> > >> >> 2. Make all the test jobs optional, so every test job requires manual
> > >> >> approval, and the documentation specifies which tests are mandatory
> > in
> > >> the
> > >> >> final steps of a ticket.
> > >> >>
> > >> >> 3. Make all the mandatory test jobs depend on a single optional job,
> > so
> > >> >> we have a single button to optionally run all the mandatory tests.
> > >> >>
> > >> >> I think any of these changes, or a combination of them, would
> > >> >> significantly reduce the usage of resources without making things
> > less
> > >> >> tested. The only downside I can think of is that we would need some
> > >> >> additional clicks on the CircleCI GUI. What do you think?
> > >> >>
> > >> >
> > >>
> > >
> > >
> > > --
> > > alex p
> > >
> >
> >
> > --
> > alex p
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Save CircleCI resources with optional test jobs
Posted by Stefan Miklosovic <st...@instaclustr.com>.
Hi Ekaterina, all,
If you eventually decide to switch it to automatic build on push (I
like how it is now though), I would appreciate it if there was some
option in generation scripts (generate.sh) so I could just have some
shortcut / alias which would disable this very easily.
It would be very nice if there was also an option to specify
parallelism for highres nodes. My workflow so far was to 1) copy
highres to config.yml, open editor and replace "100" with "70" for
parallelism value. I am not sure what plan you guys are on but we can
at most spin sometime like 80 nodes or so at once.
Thanks
Stefan
On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <e....@gmail.com> wrote:
>
> Hi everyone,
>
> First of all, I know Andres is off these days so he might not be following
> the thread today. I will try to clear the situation as the committer who
> approved the change.
>
> Alex, I would like to apologize to you! You are right, I personally didn’t
> have your email (sometimes gmail plays games with the mailing list emails,
> not sure why) and I assumed the lazy consensus. Andres sent email that this
> was implemented a month after the discussion was open and no one responded
> here during that time. In his latest email he pointed to the Slack
> discussion that happened and where agreement was reached.
> I see your email was sent two days after he said it is implemented. I guess
> we had to be more explicit that this email announces lazy consensus.
>
> Anyway, I am really sorry your email was missed and please, believe me when
> I say I would have considered it as the ticket was not committed yet at
> that point! I am almost sure something similar happened to Andres knowing
> how extremely punctual and fair he is.
>
> The main goal was not to push on every commit as many times it would be
> unnecessary waste of resources and spending credits. At the same time new
> workflows with one precommit button were added to ensure people can run all
> tests we as a community require with one click before a commit. Links with
> the different versions/suggestions of the workflow are published on the
> ticket.
>
> Yes, now we need to click on the build. It was agreed that many times
> people push even just to preserve intermediate work and continue later
> without caring of build or anything. If for some reason it is important to
> you or anyone else from the community to build on every commit, we can open
> a ticket to change that. It will save a click or two in the case of the
> in-jvm upgrade tests. I would like to point out that Andres was also
> experimenting to ensure that the graph will be still as much readable as
> possible.
>
> Benedict, on your comment of feature request - we can do that. I am also
> sceptic as you if that will happen but this doesn’t mean we can’t give it a
> try. Who knows, maybe others are also raising the topic and they might
> consider it.
>
> Honestly, I personally support the current workflow and approval required
> but if it is not acceptable to others and skipping the build approval click
> is what others would prefer, I will open a ticket to restore that part.
> Please let me know and one more time, apologize for missing your email,
> Alex.
>
> Best regards,
> Ekaterina
>
>
>
> On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <be...@apache.org>
> wrote:
>
> > I think this would be fine if there were a way for approval of later steps
> > to trigger the build automatically. As it is we have to traverse the
> > dependency graph manually, which is a bit weird.
> >
> > I can’t figure out a way to do this with the CircleCI API however. It
> > might be a feature request, and perhaps we can at least trigger the build
> > until we get that (which may never happen).
> >
> > From: Oleksandr Petrov <ol...@gmail.com>
> > Date: Wednesday, 20 October 2021 at 13:35
> > To: dev <de...@cassandra.apache.org>
> > Subject: Re: Save CircleCI resources with optional test jobs
> > I thought this discussion was still ongoing, but it looks like
> > CASSANDRA-16882 is now committed.
> >
> > Could you give some context on why at least compilation is not done on
> > every commit now?
> >
> >
> > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > oleksandr.petrov@gmail.com>
> > wrote:
> >
> > > I personally rarely push tickets/post a patch in an raw state, but since
> > I
> > > almost always have to approve jobs when it gets close to commit, I don't
> > > mind also confirming utest runs. I'd say it'd be good to run at very
> > least
> > > a compilation step on every commit. I think it's fine if
> > dtests/utests/jvm
> > > tests require approval.
> > >
> > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
> > > wrote:
> > >
> > >> Hello all,
> > >>
> > >> The changes in CircleCI proposed in the previous messages are
> > implemented
> > >> in CASSANDRA-16882. They try to include the feedback from both the
> > >> reviewers and the Slack discussion at
> > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > >>
> > >> The patch modifies the CircleCI config to have two pairs of j8/j11
> > >> workflows:
> > >>
> > >> * The javaX_pre-commit_tests workflows are meant to be used when a patch
> > >> is
> > >> definitively ready for final review and/or commit. They have a single
> > >> approval step that starts all the basic tests, including unit tests,
> > >> dtests, etc. Additionally, it has approval steps for those tests that
> > are
> > >> not generally required in every ticket, such as upgrade tests and the
> > >> multiplexer. This pair of workflows is quite similar to what we
> > currently
> > >> have, and the main difference is that there is an approval step to start
> > >> the build.
> > >>
> > >> * The javaX_separate_tests workflows are meant to be used in
> > intermediate
> > >> steps and special cases such as fixing flaky tests. All the jobs in
> > these
> > >> workflows have an individual approval step, so one can run any test job
> > >> without wasting resources in the others. For example, it makes possible
> > to
> > >> repeatedly run a unit test without firing the entire JVM dtests.
> > >>
> > >> Here is an example of how the workflows would look like:
> > >>
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> > >>
> > >> Hopefully this approach would help us to save resources in intermediate
> > >> development steps and special cases, while keeping the current concept
> > of
> > >> mandatory tests.
> > >>
> > >> If no one disagrees with this approach I'll commit the changes in a few
> > >> days.
> > >>
> > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > CASSANDRA-16882 has patches for any of the mentioned configurations
> > >> aimed
> > >> > to save CircleCI resources.
> > >> >
> > >> > There are CircleCI runs showing how each approach would look like,
> > plus
> > >> an
> > >> > additional fourth option:
> > >> >
> > >> > 1. Make the entire workflow optional:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > >> > 2. Make all the test jobs optional:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > >> > 3. Make all the mandatory test jobs depend on a single optional job:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > >> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
> > >> the
> > >> > grouped mandatory tests:
> > >> >
> > >>
> > https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> > >> >
> > >> > I think that the fourth option gives us more flexibility, because it
> > >> > allows us to start any combination of tests we want while it keeps the
> > >> > concept of required tests.
> > >> >
> > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> > >> a.penya.garcia@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hello all,
> > >> >>
> > >> >> The current CircleCI configuration automatically runs the unit tests,
> > >> JVM
> > >> >> dtests and cqhshlib tests. This is done by default for every commit
> > or,
> > >> >> with some configuration, for every push.
> > >> >>
> > >> >> Along the lifecycle of a ticket it is quite frequent to have multiple
> > >> >> commits and pushes, all running these test jobs. I'd say that
> > >> frequently it
> > >> >> is not necessary to run the tests for some of those intermediate
> > >> commits
> > >> >> and pushes. For example, one can show proofs of concept, or have
> > >> multiple
> > >> >> rounds of review before actually running the tests. Running the tests
> > >> for
> > >> >> every change can produce an unnecessary expense of CircleCI
> > resources.
> > >> >>
> > >> >> I think we could make running those tests optional, as well as
> > clearly
> > >> >> specifying in the documentation what are the tests runs that are
> > >> mandatory
> > >> >> before actually committing. We could do this in different ways:
> > >> >>
> > >> >> 1. Make the entire CircleCI workflow optional, so the build job
> > >> requires
> > >> >> manual approval. Once the build is approved the mandatory test jobs
> > >> would
> > >> >> be run without any further approval, exactly as it's currently done.
> > >> >>
> > >> >> 2. Make all the test jobs optional, so every test job requires manual
> > >> >> approval, and the documentation specifies which tests are mandatory
> > in
> > >> the
> > >> >> final steps of a ticket.
> > >> >>
> > >> >> 3. Make all the mandatory test jobs depend on a single optional job,
> > so
> > >> >> we have a single button to optionally run all the mandatory tests.
> > >> >>
> > >> >> I think any of these changes, or a combination of them, would
> > >> >> significantly reduce the usage of resources without making things
> > less
> > >> >> tested. The only downside I can think of is that we would need some
> > >> >> additional clicks on the CircleCI GUI. What do you think?
> > >> >>
> > >> >
> > >>
> > >
> > >
> > > --
> > > alex p
> > >
> >
> >
> > --
> > alex p
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Save CircleCI resources with optional test jobs
Posted by Ekaterina Dimitrova <e....@gmail.com>.
Hi everyone,
First of all, I know Andres is off these days so he might not be following
the thread today. I will try to clear the situation as the committer who
approved the change.
Alex, I would like to apologize to you! You are right, I personally didn’t
have your email (sometimes gmail plays games with the mailing list emails,
not sure why) and I assumed the lazy consensus. Andres sent email that this
was implemented a month after the discussion was open and no one responded
here during that time. In his latest email he pointed to the Slack
discussion that happened and where agreement was reached.
I see your email was sent two days after he said it is implemented. I guess
we had to be more explicit that this email announces lazy consensus.
Anyway, I am really sorry your email was missed and please, believe me when
I say I would have considered it as the ticket was not committed yet at
that point! I am almost sure something similar happened to Andres knowing
how extremely punctual and fair he is.
The main goal was not to push on every commit as many times it would be
unnecessary waste of resources and spending credits. At the same time new
workflows with one precommit button were added to ensure people can run all
tests we as a community require with one click before a commit. Links with
the different versions/suggestions of the workflow are published on the
ticket.
Yes, now we need to click on the build. It was agreed that many times
people push even just to preserve intermediate work and continue later
without caring of build or anything. If for some reason it is important to
you or anyone else from the community to build on every commit, we can open
a ticket to change that. It will save a click or two in the case of the
in-jvm upgrade tests. I would like to point out that Andres was also
experimenting to ensure that the graph will be still as much readable as
possible.
Benedict, on your comment of feature request - we can do that. I am also
sceptic as you if that will happen but this doesn’t mean we can’t give it a
try. Who knows, maybe others are also raising the topic and they might
consider it.
Honestly, I personally support the current workflow and approval required
but if it is not acceptable to others and skipping the build approval click
is what others would prefer, I will open a ticket to restore that part.
Please let me know and one more time, apologize for missing your email,
Alex.
Best regards,
Ekaterina
On Wed, 20 Oct 2021 at 9:01, benedict@apache.org <be...@apache.org>
wrote:
> I think this would be fine if there were a way for approval of later steps
> to trigger the build automatically. As it is we have to traverse the
> dependency graph manually, which is a bit weird.
>
> I can’t figure out a way to do this with the CircleCI API however. It
> might be a feature request, and perhaps we can at least trigger the build
> until we get that (which may never happen).
>
> From: Oleksandr Petrov <ol...@gmail.com>
> Date: Wednesday, 20 October 2021 at 13:35
> To: dev <de...@cassandra.apache.org>
> Subject: Re: Save CircleCI resources with optional test jobs
> I thought this discussion was still ongoing, but it looks like
> CASSANDRA-16882 is now committed.
>
> Could you give some context on why at least compilation is not done on
> every commit now?
>
>
> On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> oleksandr.petrov@gmail.com>
> wrote:
>
> > I personally rarely push tickets/post a patch in an raw state, but since
> I
> > almost always have to approve jobs when it gets close to commit, I don't
> > mind also confirming utest runs. I'd say it'd be good to run at very
> least
> > a compilation step on every commit. I think it's fine if
> dtests/utests/jvm
> > tests require approval.
> >
> > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
> > wrote:
> >
> >> Hello all,
> >>
> >> The changes in CircleCI proposed in the previous messages are
> implemented
> >> in CASSANDRA-16882. They try to include the feedback from both the
> >> reviewers and the Slack discussion at
> >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> >>
> >> The patch modifies the CircleCI config to have two pairs of j8/j11
> >> workflows:
> >>
> >> * The javaX_pre-commit_tests workflows are meant to be used when a patch
> >> is
> >> definitively ready for final review and/or commit. They have a single
> >> approval step that starts all the basic tests, including unit tests,
> >> dtests, etc. Additionally, it has approval steps for those tests that
> are
> >> not generally required in every ticket, such as upgrade tests and the
> >> multiplexer. This pair of workflows is quite similar to what we
> currently
> >> have, and the main difference is that there is an approval step to start
> >> the build.
> >>
> >> * The javaX_separate_tests workflows are meant to be used in
> intermediate
> >> steps and special cases such as fixing flaky tests. All the jobs in
> these
> >> workflows have an individual approval step, so one can run any test job
> >> without wasting resources in the others. For example, it makes possible
> to
> >> repeatedly run a unit test without firing the entire JVM dtests.
> >>
> >> Here is an example of how the workflows would look like:
> >>
> >>
> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
> >>
> >> Hopefully this approach would help us to save resources in intermediate
> >> development steps and special cases, while keeping the current concept
> of
> >> mandatory tests.
> >>
> >> If no one disagrees with this approach I'll commit the changes in a few
> >> days.
> >>
> >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > CASSANDRA-16882 has patches for any of the mentioned configurations
> >> aimed
> >> > to save CircleCI resources.
> >> >
> >> > There are CircleCI runs showing how each approach would look like,
> plus
> >> an
> >> > additional fourth option:
> >> >
> >> > 1. Make the entire workflow optional:
> >> >
> >>
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> >> > 2. Make all the test jobs optional:
> >> >
> >>
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> >> > 3. Make all the mandatory test jobs depend on a single optional job:
> >> >
> >>
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> >> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
> >> the
> >> > grouped mandatory tests:
> >> >
> >>
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> >> >
> >> > I think that the fourth option gives us more flexibility, because it
> >> > allows us to start any combination of tests we want while it keeps the
> >> > concept of required tests.
> >> >
> >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> >> a.penya.garcia@gmail.com>
> >> > wrote:
> >> >
> >> >> Hello all,
> >> >>
> >> >> The current CircleCI configuration automatically runs the unit tests,
> >> JVM
> >> >> dtests and cqhshlib tests. This is done by default for every commit
> or,
> >> >> with some configuration, for every push.
> >> >>
> >> >> Along the lifecycle of a ticket it is quite frequent to have multiple
> >> >> commits and pushes, all running these test jobs. I'd say that
> >> frequently it
> >> >> is not necessary to run the tests for some of those intermediate
> >> commits
> >> >> and pushes. For example, one can show proofs of concept, or have
> >> multiple
> >> >> rounds of review before actually running the tests. Running the tests
> >> for
> >> >> every change can produce an unnecessary expense of CircleCI
> resources.
> >> >>
> >> >> I think we could make running those tests optional, as well as
> clearly
> >> >> specifying in the documentation what are the tests runs that are
> >> mandatory
> >> >> before actually committing. We could do this in different ways:
> >> >>
> >> >> 1. Make the entire CircleCI workflow optional, so the build job
> >> requires
> >> >> manual approval. Once the build is approved the mandatory test jobs
> >> would
> >> >> be run without any further approval, exactly as it's currently done.
> >> >>
> >> >> 2. Make all the test jobs optional, so every test job requires manual
> >> >> approval, and the documentation specifies which tests are mandatory
> in
> >> the
> >> >> final steps of a ticket.
> >> >>
> >> >> 3. Make all the mandatory test jobs depend on a single optional job,
> so
> >> >> we have a single button to optionally run all the mandatory tests.
> >> >>
> >> >> I think any of these changes, or a combination of them, would
> >> >> significantly reduce the usage of resources without making things
> less
> >> >> tested. The only downside I can think of is that we would need some
> >> >> additional clicks on the CircleCI GUI. What do you think?
> >> >>
> >> >
> >>
> >
> >
> > --
> > alex p
> >
>
>
> --
> alex p
>
Re: Save CircleCI resources with optional test jobs
Posted by "benedict@apache.org" <be...@apache.org>.
I think this would be fine if there were a way for approval of later steps to trigger the build automatically. As it is we have to traverse the dependency graph manually, which is a bit weird.
I can’t figure out a way to do this with the CircleCI API however. It might be a feature request, and perhaps we can at least trigger the build until we get that (which may never happen).
From: Oleksandr Petrov <ol...@gmail.com>
Date: Wednesday, 20 October 2021 at 13:35
To: dev <de...@cassandra.apache.org>
Subject: Re: Save CircleCI resources with optional test jobs
I thought this discussion was still ongoing, but it looks like
CASSANDRA-16882 is now committed.
Could you give some context on why at least compilation is not done on
every commit now?
On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <ol...@gmail.com>
wrote:
> I personally rarely push tickets/post a patch in an raw state, but since I
> almost always have to approve jobs when it gets close to commit, I don't
> mind also confirming utest runs. I'd say it'd be good to run at very least
> a compilation step on every commit. I think it's fine if dtests/utests/jvm
> tests require approval.
>
> On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
> wrote:
>
>> Hello all,
>>
>> The changes in CircleCI proposed in the previous messages are implemented
>> in CASSANDRA-16882. They try to include the feedback from both the
>> reviewers and the Slack discussion at
>> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
>>
>> The patch modifies the CircleCI config to have two pairs of j8/j11
>> workflows:
>>
>> * The javaX_pre-commit_tests workflows are meant to be used when a patch
>> is
>> definitively ready for final review and/or commit. They have a single
>> approval step that starts all the basic tests, including unit tests,
>> dtests, etc. Additionally, it has approval steps for those tests that are
>> not generally required in every ticket, such as upgrade tests and the
>> multiplexer. This pair of workflows is quite similar to what we currently
>> have, and the main difference is that there is an approval step to start
>> the build.
>>
>> * The javaX_separate_tests workflows are meant to be used in intermediate
>> steps and special cases such as fixing flaky tests. All the jobs in these
>> workflows have an individual approval step, so one can run any test job
>> without wasting resources in the others. For example, it makes possible to
>> repeatedly run a unit test without firing the entire JVM dtests.
>>
>> Here is an example of how the workflows would look like:
>>
>> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
>>
>> Hopefully this approach would help us to save resources in intermediate
>> development steps and special cases, while keeping the current concept of
>> mandatory tests.
>>
>> If no one disagrees with this approach I'll commit the changes in a few
>> days.
>>
>> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
>> wrote:
>>
>> > Hi,
>> >
>> > CASSANDRA-16882 has patches for any of the mentioned configurations
>> aimed
>> > to save CircleCI resources.
>> >
>> > There are CircleCI runs showing how each approach would look like, plus
>> an
>> > additional fourth option:
>> >
>> > 1. Make the entire workflow optional:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
>> > 2. Make all the test jobs optional:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
>> > 3. Make all the mandatory test jobs depend on a single optional job:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
>> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
>> the
>> > grouped mandatory tests:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
>> >
>> > I think that the fourth option gives us more flexibility, because it
>> > allows us to start any combination of tests we want while it keeps the
>> > concept of required tests.
>> >
>> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
>> a.penya.garcia@gmail.com>
>> > wrote:
>> >
>> >> Hello all,
>> >>
>> >> The current CircleCI configuration automatically runs the unit tests,
>> JVM
>> >> dtests and cqhshlib tests. This is done by default for every commit or,
>> >> with some configuration, for every push.
>> >>
>> >> Along the lifecycle of a ticket it is quite frequent to have multiple
>> >> commits and pushes, all running these test jobs. I'd say that
>> frequently it
>> >> is not necessary to run the tests for some of those intermediate
>> commits
>> >> and pushes. For example, one can show proofs of concept, or have
>> multiple
>> >> rounds of review before actually running the tests. Running the tests
>> for
>> >> every change can produce an unnecessary expense of CircleCI resources.
>> >>
>> >> I think we could make running those tests optional, as well as clearly
>> >> specifying in the documentation what are the tests runs that are
>> mandatory
>> >> before actually committing. We could do this in different ways:
>> >>
>> >> 1. Make the entire CircleCI workflow optional, so the build job
>> requires
>> >> manual approval. Once the build is approved the mandatory test jobs
>> would
>> >> be run without any further approval, exactly as it's currently done.
>> >>
>> >> 2. Make all the test jobs optional, so every test job requires manual
>> >> approval, and the documentation specifies which tests are mandatory in
>> the
>> >> final steps of a ticket.
>> >>
>> >> 3. Make all the mandatory test jobs depend on a single optional job, so
>> >> we have a single button to optionally run all the mandatory tests.
>> >>
>> >> I think any of these changes, or a combination of them, would
>> >> significantly reduce the usage of resources without making things less
>> >> tested. The only downside I can think of is that we would need some
>> >> additional clicks on the CircleCI GUI. What do you think?
>> >>
>> >
>>
>
>
> --
> alex p
>
--
alex p
Re: Save CircleCI resources with optional test jobs
Posted by Oleksandr Petrov <ol...@gmail.com>.
I thought this discussion was still ongoing, but it looks like
CASSANDRA-16882 is now committed.
Could you give some context on why at least compilation is not done on
every commit now?
On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <ol...@gmail.com>
wrote:
> I personally rarely push tickets/post a patch in an raw state, but since I
> almost always have to approve jobs when it gets close to commit, I don't
> mind also confirming utest runs. I'd say it'd be good to run at very least
> a compilation step on every commit. I think it's fine if dtests/utests/jvm
> tests require approval.
>
> On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
> wrote:
>
>> Hello all,
>>
>> The changes in CircleCI proposed in the previous messages are implemented
>> in CASSANDRA-16882. They try to include the feedback from both the
>> reviewers and the Slack discussion at
>> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
>>
>> The patch modifies the CircleCI config to have two pairs of j8/j11
>> workflows:
>>
>> * The javaX_pre-commit_tests workflows are meant to be used when a patch
>> is
>> definitively ready for final review and/or commit. They have a single
>> approval step that starts all the basic tests, including unit tests,
>> dtests, etc. Additionally, it has approval steps for those tests that are
>> not generally required in every ticket, such as upgrade tests and the
>> multiplexer. This pair of workflows is quite similar to what we currently
>> have, and the main difference is that there is an approval step to start
>> the build.
>>
>> * The javaX_separate_tests workflows are meant to be used in intermediate
>> steps and special cases such as fixing flaky tests. All the jobs in these
>> workflows have an individual approval step, so one can run any test job
>> without wasting resources in the others. For example, it makes possible to
>> repeatedly run a unit test without firing the entire JVM dtests.
>>
>> Here is an example of how the workflows would look like:
>>
>> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
>>
>> Hopefully this approach would help us to save resources in intermediate
>> development steps and special cases, while keeping the current concept of
>> mandatory tests.
>>
>> If no one disagrees with this approach I'll commit the changes in a few
>> days.
>>
>> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
>> wrote:
>>
>> > Hi,
>> >
>> > CASSANDRA-16882 has patches for any of the mentioned configurations
>> aimed
>> > to save CircleCI resources.
>> >
>> > There are CircleCI runs showing how each approach would look like, plus
>> an
>> > additional fourth option:
>> >
>> > 1. Make the entire workflow optional:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
>> > 2. Make all the test jobs optional:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
>> > 3. Make all the mandatory test jobs depend on a single optional job:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
>> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
>> the
>> > grouped mandatory tests:
>> >
>> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
>> >
>> > I think that the fourth option gives us more flexibility, because it
>> > allows us to start any combination of tests we want while it keeps the
>> > concept of required tests.
>> >
>> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
>> a.penya.garcia@gmail.com>
>> > wrote:
>> >
>> >> Hello all,
>> >>
>> >> The current CircleCI configuration automatically runs the unit tests,
>> JVM
>> >> dtests and cqhshlib tests. This is done by default for every commit or,
>> >> with some configuration, for every push.
>> >>
>> >> Along the lifecycle of a ticket it is quite frequent to have multiple
>> >> commits and pushes, all running these test jobs. I'd say that
>> frequently it
>> >> is not necessary to run the tests for some of those intermediate
>> commits
>> >> and pushes. For example, one can show proofs of concept, or have
>> multiple
>> >> rounds of review before actually running the tests. Running the tests
>> for
>> >> every change can produce an unnecessary expense of CircleCI resources.
>> >>
>> >> I think we could make running those tests optional, as well as clearly
>> >> specifying in the documentation what are the tests runs that are
>> mandatory
>> >> before actually committing. We could do this in different ways:
>> >>
>> >> 1. Make the entire CircleCI workflow optional, so the build job
>> requires
>> >> manual approval. Once the build is approved the mandatory test jobs
>> would
>> >> be run without any further approval, exactly as it's currently done.
>> >>
>> >> 2. Make all the test jobs optional, so every test job requires manual
>> >> approval, and the documentation specifies which tests are mandatory in
>> the
>> >> final steps of a ticket.
>> >>
>> >> 3. Make all the mandatory test jobs depend on a single optional job, so
>> >> we have a single button to optionally run all the mandatory tests.
>> >>
>> >> I think any of these changes, or a combination of them, would
>> >> significantly reduce the usage of resources without making things less
>> >> tested. The only downside I can think of is that we would need some
>> >> additional clicks on the CircleCI GUI. What do you think?
>> >>
>> >
>>
>
>
> --
> alex p
>
--
alex p
Re: Save CircleCI resources with optional test jobs
Posted by Oleksandr Petrov <ol...@gmail.com>.
I personally rarely push tickets/post a patch in an raw state, but since I
almost always have to approve jobs when it gets close to commit, I don't
mind also confirming utest runs. I'd say it'd be good to run at very least
a compilation step on every commit. I think it's fine if dtests/utests/jvm
tests require approval.
On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña <ad...@apache.org>
wrote:
> Hello all,
>
> The changes in CircleCI proposed in the previous messages are implemented
> in CASSANDRA-16882. They try to include the feedback from both the
> reviewers and the Slack discussion at
> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
>
> The patch modifies the CircleCI config to have two pairs of j8/j11
> workflows:
>
> * The javaX_pre-commit_tests workflows are meant to be used when a patch is
> definitively ready for final review and/or commit. They have a single
> approval step that starts all the basic tests, including unit tests,
> dtests, etc. Additionally, it has approval steps for those tests that are
> not generally required in every ticket, such as upgrade tests and the
> multiplexer. This pair of workflows is quite similar to what we currently
> have, and the main difference is that there is an approval step to start
> the build.
>
> * The javaX_separate_tests workflows are meant to be used in intermediate
> steps and special cases such as fixing flaky tests. All the jobs in these
> workflows have an individual approval step, so one can run any test job
> without wasting resources in the others. For example, it makes possible to
> repeatedly run a unit test without firing the entire JVM dtests.
>
> Here is an example of how the workflows would look like:
>
> https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04
>
> Hopefully this approach would help us to save resources in intermediate
> development steps and special cases, while keeping the current concept of
> mandatory tests.
>
> If no one disagrees with this approach I'll commit the changes in a few
> days.
>
> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña <ad...@apache.org>
> wrote:
>
> > Hi,
> >
> > CASSANDRA-16882 has patches for any of the mentioned configurations aimed
> > to save CircleCI resources.
> >
> > There are CircleCI runs showing how each approach would look like, plus
> an
> > additional fourth option:
> >
> > 1. Make the entire workflow optional:
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29
> > 2. Make all the test jobs optional:
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3
> > 3. Make all the mandatory test jobs depend on a single optional job:
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae
> > 4. Combine 2 and 3 to have approval jobs for both individual tests and
> the
> > grouped mandatory tests:
> >
> https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190
> >
> > I think that the fourth option gives us more flexibility, because it
> > allows us to start any combination of tests we want while it keeps the
> > concept of required tests.
> >
> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña <
> a.penya.garcia@gmail.com>
> > wrote:
> >
> >> Hello all,
> >>
> >> The current CircleCI configuration automatically runs the unit tests,
> JVM
> >> dtests and cqhshlib tests. This is done by default for every commit or,
> >> with some configuration, for every push.
> >>
> >> Along the lifecycle of a ticket it is quite frequent to have multiple
> >> commits and pushes, all running these test jobs. I'd say that
> frequently it
> >> is not necessary to run the tests for some of those intermediate commits
> >> and pushes. For example, one can show proofs of concept, or have
> multiple
> >> rounds of review before actually running the tests. Running the tests
> for
> >> every change can produce an unnecessary expense of CircleCI resources.
> >>
> >> I think we could make running those tests optional, as well as clearly
> >> specifying in the documentation what are the tests runs that are
> mandatory
> >> before actually committing. We could do this in different ways:
> >>
> >> 1. Make the entire CircleCI workflow optional, so the build job requires
> >> manual approval. Once the build is approved the mandatory test jobs
> would
> >> be run without any further approval, exactly as it's currently done.
> >>
> >> 2. Make all the test jobs optional, so every test job requires manual
> >> approval, and the documentation specifies which tests are mandatory in
> the
> >> final steps of a ticket.
> >>
> >> 3. Make all the mandatory test jobs depend on a single optional job, so
> >> we have a single button to optionally run all the mandatory tests.
> >>
> >> I think any of these changes, or a combination of them, would
> >> significantly reduce the usage of resources without making things less
> >> tested. The only downside I can think of is that we would need some
> >> additional clicks on the CircleCI GUI. What do you think?
> >>
> >
>
--
alex p