You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Stefan Podkowinski <sp...@apache.org> on 2018/10/26 14:49:47 UTC
Proposed changes to CircleCI testing workflow
I'd like to give you a quick update on the work that has been done
lately on running tests using CircleCI. Please let me know if you have
any objections or don't think this is going into the right direction, or
have any other feedback!
We've been using CircleCI for a while now and results are used on
constant basis for new patches. Not only by committers, but also by
casual contributors to run unit tests. Looks like people find the
service valuable and we should keep using it. Therefor I'd like to make
some improvements that will make it easier to add new tests and to
continue making CircleCI an option for all contributors, both on paid
and free plans.
The general idea of the changes implemented in #14806, is to consolidate
the existing config to make it more modular and have smaller jobs that
can be scheduled ad-hoc by the developer, instead of running a few big
jobs on every commit. Reorganizing and breaking up the existing config
was done using the new 2.1 config features. Starting jobs on request,
instead of automatically, is done using the manual approval feature,
i.e. you now have to click on that job in the workflow page in order to
start it. I'd like to see us having smaller, more specialized groups of
tests that we can run more selectively during development, while still
being able to run bigger tests before committing, or firing up all of
them during testing and releasing. Other example of smaller jobs would
be testing coverage (#14788) or cqlsh tests (#14298). But also
individual jobs for different ant targets, like burn, stress or benchmarks.
We'd now also be able to run tests using different docker images and
different JDKs. I've already updated the used image to also include Java
11 and added unit and dtest jobs to the config for that. It's now really
easy to run tests on Java 11, although these won't pass yet. It seems to
be important to me to have this kind of flexibility, given the
increasingly diverse ecosystem of Java distributions. We can also add
jobs for packaging and doing smoke tests by installing and starting
packages on different docker images (Redhat, Debian, Ubuntu,..) at a
later point.
As for the paid vs free plans issue, I'd also like us to discuss how we
can make tests faster and less resource intensive in general. As a
desired consequence, we'd be able to move away from multi-node dtests,
to something that can be run using the free plan. I'm looking forward to
see if #14821 can get us into that direction. Ideally we can add these
tests into a job that can be completed on the free plan and encourage
contributors to add new tests there, instead of having to write a dtest,
which they won't be able to run on CircleCI without a paid plan.
Whats changing for you as a CircleCI user?
* All tests, except unit tests, will need to be started manually and
will not run on every commit (this can be further discussed and changed
anytime if needed)
* Updating the config.yml file now requires using the CircleCI cli tool
and should not be done directly (see #14806 for technical details)
* High resource settings can be enabled using a script/patch, either run
manually or as commit hook (again see ticket for details)
* Both free and paid plan users now have more tests to run
As already mentioned, please let me know if you have any thoughts on
this, or if you think this is going into the wrong direction.
Thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Proposed changes to CircleCI testing workflow
Posted by Sumanth Pasupuleti <su...@gmail.com>.
Hi Stefan,
+1 on the CircleCI changes.
A few months ago, I took a shot at CircleCI to add support for Java 1.7 and
in the process (perhaps the more important takeaway), explored adding
"aliases" to avoid code duplication in the yml file (
https://github.com/apache/cassandra/pull/248/commits/47855675f4805a09b78059f7231311566f0e66f0).
I believe we can do similar optimization to
https://github.com/spodkowinski/cassandra/blob/WIP-14806/.circleci/config.yml
.
Also, with your changes, I believe we can close out CASSANDRA-14609.
I was also planning to work on CASSANDRA-14718 to validate artifacts
generation - let me know what you think.
Thanks,
Sumanth
On Fri, Oct 26, 2018 at 10:05 AM Ariel Weisberg <ar...@weisberg.ws> wrote:
> Hi,
>
> Thank you for working on this. These all sound like good changes to me.
>
> Ariel
>
> On Fri, Oct 26, 2018, at 10:49 AM, Stefan Podkowinski wrote:
> > I'd like to give you a quick update on the work that has been done
> > lately on running tests using CircleCI. Please let me know if you have
> > any objections or don't think this is going into the right direction, or
> > have any other feedback!
> >
> > We've been using CircleCI for a while now and results are used on
> > constant basis for new patches. Not only by committers, but also by
> > casual contributors to run unit tests. Looks like people find the
> > service valuable and we should keep using it. Therefor I'd like to make
> > some improvements that will make it easier to add new tests and to
> > continue making CircleCI an option for all contributors, both on paid
> > and free plans.
> >
> > The general idea of the changes implemented in #14806, is to consolidate
> > the existing config to make it more modular and have smaller jobs that
> > can be scheduled ad-hoc by the developer, instead of running a few big
> > jobs on every commit. Reorganizing and breaking up the existing config
> > was done using the new 2.1 config features. Starting jobs on request,
> > instead of automatically, is done using the manual approval feature,
> > i.e. you now have to click on that job in the workflow page in order to
> > start it. I'd like to see us having smaller, more specialized groups of
> > tests that we can run more selectively during development, while still
> > being able to run bigger tests before committing, or firing up all of
> > them during testing and releasing. Other example of smaller jobs would
> > be testing coverage (#14788) or cqlsh tests (#14298). But also
> > individual jobs for different ant targets, like burn, stress or
> benchmarks.
> >
> > We'd now also be able to run tests using different docker images and
> > different JDKs. I've already updated the used image to also include Java
> > 11 and added unit and dtest jobs to the config for that. It's now really
> > easy to run tests on Java 11, although these won't pass yet. It seems to
> > be important to me to have this kind of flexibility, given the
> > increasingly diverse ecosystem of Java distributions. We can also add
> > jobs for packaging and doing smoke tests by installing and starting
> > packages on different docker images (Redhat, Debian, Ubuntu,..) at a
> > later point.
> >
> > As for the paid vs free plans issue, I'd also like us to discuss how we
> > can make tests faster and less resource intensive in general. As a
> > desired consequence, we'd be able to move away from multi-node dtests,
> > to something that can be run using the free plan. I'm looking forward to
> > see if #14821 can get us into that direction. Ideally we can add these
> > tests into a job that can be completed on the free plan and encourage
> > contributors to add new tests there, instead of having to write a dtest,
> > which they won't be able to run on CircleCI without a paid plan.
> >
> > Whats changing for you as a CircleCI user?
> > * All tests, except unit tests, will need to be started manually and
> > will not run on every commit (this can be further discussed and changed
> > anytime if needed)
> > * Updating the config.yml file now requires using the CircleCI cli tool
> > and should not be done directly (see #14806 for technical details)
> > * High resource settings can be enabled using a script/patch, either run
> > manually or as commit hook (again see ticket for details)
> > * Both free and paid plan users now have more tests to run
> >
> > As already mentioned, please let me know if you have any thoughts on
> > this, or if you think this is going into the wrong direction.
> >
> > Thanks.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: Proposed changes to CircleCI testing workflow
Posted by Ariel Weisberg <ar...@weisberg.ws>.
Hi,
Thank you for working on this. These all sound like good changes to me.
Ariel
On Fri, Oct 26, 2018, at 10:49 AM, Stefan Podkowinski wrote:
> I'd like to give you a quick update on the work that has been done
> lately on running tests using CircleCI. Please let me know if you have
> any objections or don't think this is going into the right direction, or
> have any other feedback!
>
> We've been using CircleCI for a while now and results are used on
> constant basis for new patches. Not only by committers, but also by
> casual contributors to run unit tests. Looks like people find the
> service valuable and we should keep using it. Therefor I'd like to make
> some improvements that will make it easier to add new tests and to
> continue making CircleCI an option for all contributors, both on paid
> and free plans.
>
> The general idea of the changes implemented in #14806, is to consolidate
> the existing config to make it more modular and have smaller jobs that
> can be scheduled ad-hoc by the developer, instead of running a few big
> jobs on every commit. Reorganizing and breaking up the existing config
> was done using the new 2.1 config features. Starting jobs on request,
> instead of automatically, is done using the manual approval feature,
> i.e. you now have to click on that job in the workflow page in order to
> start it. I'd like to see us having smaller, more specialized groups of
> tests that we can run more selectively during development, while still
> being able to run bigger tests before committing, or firing up all of
> them during testing and releasing. Other example of smaller jobs would
> be testing coverage (#14788) or cqlsh tests (#14298). But also
> individual jobs for different ant targets, like burn, stress or benchmarks.
>
> We'd now also be able to run tests using different docker images and
> different JDKs. I've already updated the used image to also include Java
> 11 and added unit and dtest jobs to the config for that. It's now really
> easy to run tests on Java 11, although these won't pass yet. It seems to
> be important to me to have this kind of flexibility, given the
> increasingly diverse ecosystem of Java distributions. We can also add
> jobs for packaging and doing smoke tests by installing and starting
> packages on different docker images (Redhat, Debian, Ubuntu,..) at a
> later point.
>
> As for the paid vs free plans issue, I'd also like us to discuss how we
> can make tests faster and less resource intensive in general. As a
> desired consequence, we'd be able to move away from multi-node dtests,
> to something that can be run using the free plan. I'm looking forward to
> see if #14821 can get us into that direction. Ideally we can add these
> tests into a job that can be completed on the free plan and encourage
> contributors to add new tests there, instead of having to write a dtest,
> which they won't be able to run on CircleCI without a paid plan.
>
> Whats changing for you as a CircleCI user?
> * All tests, except unit tests, will need to be started manually and
> will not run on every commit (this can be further discussed and changed
> anytime if needed)
> * Updating the config.yml file now requires using the CircleCI cli tool
> and should not be done directly (see #14806 for technical details)
> * High resource settings can be enabled using a script/patch, either run
> manually or as commit hook (again see ticket for details)
> * Both free and paid plan users now have more tests to run
>
> As already mentioned, please let me know if you have any thoughts on
> this, or if you think this is going into the wrong direction.
>
> Thanks.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Proposed changes to CircleCI testing workflow
Posted by Benedict Elliott Smith <be...@apache.org>.
Just want to say +1, and thanks for taking the time to do this. This all sounds great.
(And completely supersedes what I hoped to achieve with CASSANDRA-14648)
> On 26 Oct 2018, at 15:49, Stefan Podkowinski <sp...@apache.org> wrote:
>
> I'd like to give you a quick update on the work that has been done
> lately on running tests using CircleCI. Please let me know if you have
> any objections or don't think this is going into the right direction, or
> have any other feedback!
>
> We've been using CircleCI for a while now and results are used on
> constant basis for new patches. Not only by committers, but also by
> casual contributors to run unit tests. Looks like people find the
> service valuable and we should keep using it. Therefor I'd like to make
> some improvements that will make it easier to add new tests and to
> continue making CircleCI an option for all contributors, both on paid
> and free plans.
>
> The general idea of the changes implemented in #14806, is to consolidate
> the existing config to make it more modular and have smaller jobs that
> can be scheduled ad-hoc by the developer, instead of running a few big
> jobs on every commit. Reorganizing and breaking up the existing config
> was done using the new 2.1 config features. Starting jobs on request,
> instead of automatically, is done using the manual approval feature,
> i.e. you now have to click on that job in the workflow page in order to
> start it. I'd like to see us having smaller, more specialized groups of
> tests that we can run more selectively during development, while still
> being able to run bigger tests before committing, or firing up all of
> them during testing and releasing. Other example of smaller jobs would
> be testing coverage (#14788) or cqlsh tests (#14298). But also
> individual jobs for different ant targets, like burn, stress or benchmarks.
>
> We'd now also be able to run tests using different docker images and
> different JDKs. I've already updated the used image to also include Java
> 11 and added unit and dtest jobs to the config for that. It's now really
> easy to run tests on Java 11, although these won't pass yet. It seems to
> be important to me to have this kind of flexibility, given the
> increasingly diverse ecosystem of Java distributions. We can also add
> jobs for packaging and doing smoke tests by installing and starting
> packages on different docker images (Redhat, Debian, Ubuntu,..) at a
> later point.
>
> As for the paid vs free plans issue, I'd also like us to discuss how we
> can make tests faster and less resource intensive in general. As a
> desired consequence, we'd be able to move away from multi-node dtests,
> to something that can be run using the free plan. I'm looking forward to
> see if #14821 can get us into that direction. Ideally we can add these
> tests into a job that can be completed on the free plan and encourage
> contributors to add new tests there, instead of having to write a dtest,
> which they won't be able to run on CircleCI without a paid plan.
>
> Whats changing for you as a CircleCI user?
> * All tests, except unit tests, will need to be started manually and
> will not run on every commit (this can be further discussed and changed
> anytime if needed)
> * Updating the config.yml file now requires using the CircleCI cli tool
> and should not be done directly (see #14806 for technical details)
> * High resource settings can be enabled using a script/patch, either run
> manually or as commit hook (again see ticket for details)
> * Both free and paid plan users now have more tests to run
>
> As already mentioned, please let me know if you have any thoughts on
> this, or if you think this is going into the wrong direction.
>
> Thanks.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org