You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Joshua McKenzie <jm...@apache.org> on 2021/12/15 22:44:30 UTC
Clarifying CI release criteria
What CI criteria are we using to gate minor and major releases? For
example: All tests from all suites on all configurations? A subset?
All the test suite columns on the following page?
https://ci-cassandra.apache.org/job/Cassandra-trunk/
All the test suites in all combinations on all JDKs on circle-ci?
This is particularly relevant to how we craft the Build Lead role w/regards
to what we'll be triaging. Also, if a bunch of us are using Circle
pre-commit and we're seeing test flakes unique to the ASF Jenkins execution
environment, we're going to naturally accumulate debt as we lead up to a
release if we block on ASF Jenkins being green.
Notably, I can't find this clarified in our documentation on testing or the
release process, nor is it clarified on the confluence wiki article on the
Release Lifecycle.
Thanks.
~Josh
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
Could you note that in the wiki in the comments section Ekaterina?
https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&
Thanks!
On Tue, Jan 4, 2022 at 4:21 PM Ekaterina Dimitrova <e....@gmail.com>
wrote:
> Hey Josh,
> Thank you for leading these discussions and organizing the wiki pages
> (also from the other mail).
> I just wanted to mention about point 4 of Pending work - I have a draft
> version for CircleCI usage, also Andres has updated the rst documents
> around running tests in a loop, etc.
> BUT those are pending on the merge of the ascii docs (to convert his work)
> and add the correct links and any other changes that have happened since I
> wrote it last year. Just saying for awareness before anyone decides to make
> a write up on that not knowing a draft already exists.
>
> Thanks,
> Ekaterina
>
> On Tue, 4 Jan 2022 at 14:51, Joshua McKenzie <jm...@apache.org> wrote:
>
>> Here's a link to a draft article for the confluence wiki covering what we
>> discussed on this thread:
>> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&
>>
>> Assuming this article accurately captures what we discussed here as well
>> as outstanding work, I'll get it published and integrated with the
>> confluence wiki and get the remainder of the work into a JIRA epic to be
>> tracked.
>>
>> On Fri, Dec 17, 2021 at 4:41 PM Joshua McKenzie <jm...@apache.org>
>> wrote:
>>
>>> I'll get this into a draft article on the wiki so we can collab on those
>>> 3 outstanding TBD's without further cluttering up the dev list. :)
>>>
>>> On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <
>>> e.dimitrova@gmail.com> wrote:
>>>
>>>> It’s indeed good call but I thought this will be addressed in a separate
>>>> document where we discuss required test suites to be run pre-commit. If
>>>> not
>>>> - then I guess we should add those things here too?
>>>>
>>>> On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jm...@apache.org>
>>>> wrote:
>>>>
>>>> > Good call; thanks for the reminder.
>>>> >
>>>> > So maybe add a
>>>> >
>>>> > 3.a: Run all new or modified tests through either local or remote
>>>> > multiplexer N (TBD - 50?) times (w/link to instructions, etc)
>>>> > 3.b Non-regressing is defined here...
>>>> > 3.c After merging tickets...
>>>> >
>>>> > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Could we also add something about running new tests through the
>>>> > > multiplexer?
>>>> > >
>>>> > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <
>>>> jmckenzie@apache.org>
>>>> > > wrote:
>>>> > > >
>>>> > > > So to clarify it all in one place, the proposed new CI process we
>>>> > should
>>>> > > > test for consensus is:
>>>> > > >
>>>> > > > 1. Canonical CI for a release is ci-cassandra. We can optionally,
>>>> and
>>>> > in
>>>> > > > practice will, run circle as well but don't codify blocking on
>>>> that.
>>>> > > > 2. (NEW) We don't release unless we get a fully green run.
>>>> > > > 3. Before any merge, you need either a non-regressing (i.e. no new
>>>> > > > failures) run of circleci with a (specific suite of tests TBD) or
>>>> of
>>>> > > > ci-cassandra.
>>>> > > > 3.a Non-regressing is defined here as "Doesn't introduce any
>>>> new
>>>> > > test
>>>> > > > failures; any new failures in CI are clearly not attributable to
>>>> this
>>>> > > diff"
>>>> > > > 3.b: (NEW) After merging tickets, ci-cassandra runs against
>>>> the
>>>> > SHA
>>>> > > > and the author gets an advisory update on the related JIRA for
>>>> any new
>>>> > > > errors on CI. The author of the ticket will take point on
>>>> triaging this
>>>> > > new
>>>> > > > failure and either fixing (if clearly reproducible or related to
>>>> their
>>>> > > > work) or opening a JIRA for the intermittent failure and linking
>>>> it in
>>>> > > > butler (https://butler.cassandra.apache.org/#/)
>>>> > > > 4. (NEW) The Build Lead role + Butler catches and documents all
>>>> > failures
>>>> > > > and anything that slips through the procedural cracks in 3.b;
>>>> > resourcing
>>>> > > > for fixing flakey tests TBD
>>>> > > >
>>>> > > > Our two TBD we can tackle separately from consensus on the above:
>>>> > > > 1. Suite of tests on circle required to be considered ready for
>>>> merge
>>>> > > > 2. How we resource fixing flakey tests that are functionally
>>>> impossible
>>>> > > to
>>>> > > > attribute without essentially fixing the flake
>>>> > > >
>>>> > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
>>>> > > e.dimitrova@gmail.com>
>>>> > > > wrote:
>>>> > > >
>>>> > > > > +1 (nb) on my end too, I second Mick
>>>> > > > > Thanks for putting this together Josh
>>>> > > > >
>>>> > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org>
>>>> > wrote:
>>>> > > > >
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already
>>>> do
>>>> > > this)
>>>> > > > > and
>>>> > > > > > > get an advisory update on the related JIRA for any new
>>>> errors on
>>>> > > the
>>>> > > > > run
>>>> > > > > > of
>>>> > > > > > > the SHA
>>>> > > > > > >
>>>> > > > > > > I strongly prefer we amend our process with 3.c.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > +1 Yup, this is the most important missing piece for me.
>>>> > > > > >
>>>> > > > > > I also wouldn't mind we word the responsibility of the author
>>>> at
>>>> > > > > > post-commit fault to be involved/leading in the fix. This
>>>> > > incentivises
>>>> > > > > > people to do 2+3 properly, and not push it onto the build
>>>> role.
>>>> > > > > >
>>>> > > > >
>>>> > >
>>>> > >
>>>> ---------------------------------------------------------------------
>>>> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>> > > For additional commands, e-mail: dev-help@cassandra.apache.org
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
Re: Clarifying CI release criteria
Posted by Ekaterina Dimitrova <e....@gmail.com>.
Hey Josh,
Thank you for leading these discussions and organizing the wiki pages (also
from the other mail).
I just wanted to mention about point 4 of Pending work - I have a draft
version for CircleCI usage, also Andres has updated the rst documents
around running tests in a loop, etc.
BUT those are pending on the merge of the ascii docs (to convert his work)
and add the correct links and any other changes that have happened since I
wrote it last year. Just saying for awareness before anyone decides to make
a write up on that not knowing a draft already exists.
Thanks,
Ekaterina
On Tue, 4 Jan 2022 at 14:51, Joshua McKenzie <jm...@apache.org> wrote:
> Here's a link to a draft article for the confluence wiki covering what we
> discussed on this thread:
> https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&
>
> Assuming this article accurately captures what we discussed here as well
> as outstanding work, I'll get it published and integrated with the
> confluence wiki and get the remainder of the work into a JIRA epic to be
> tracked.
>
> On Fri, Dec 17, 2021 at 4:41 PM Joshua McKenzie <jm...@apache.org>
> wrote:
>
>> I'll get this into a draft article on the wiki so we can collab on those
>> 3 outstanding TBD's without further cluttering up the dev list. :)
>>
>> On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <
>> e.dimitrova@gmail.com> wrote:
>>
>>> It’s indeed good call but I thought this will be addressed in a separate
>>> document where we discuss required test suites to be run pre-commit. If
>>> not
>>> - then I guess we should add those things here too?
>>>
>>> On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jm...@apache.org>
>>> wrote:
>>>
>>> > Good call; thanks for the reminder.
>>> >
>>> > So maybe add a
>>> >
>>> > 3.a: Run all new or modified tests through either local or remote
>>> > multiplexer N (TBD - 50?) times (w/link to instructions, etc)
>>> > 3.b Non-regressing is defined here...
>>> > 3.c After merging tickets...
>>> >
>>> > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Could we also add something about running new tests through the
>>> > > multiplexer?
>>> > >
>>> > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <
>>> jmckenzie@apache.org>
>>> > > wrote:
>>> > > >
>>> > > > So to clarify it all in one place, the proposed new CI process we
>>> > should
>>> > > > test for consensus is:
>>> > > >
>>> > > > 1. Canonical CI for a release is ci-cassandra. We can optionally,
>>> and
>>> > in
>>> > > > practice will, run circle as well but don't codify blocking on
>>> that.
>>> > > > 2. (NEW) We don't release unless we get a fully green run.
>>> > > > 3. Before any merge, you need either a non-regressing (i.e. no new
>>> > > > failures) run of circleci with a (specific suite of tests TBD) or
>>> of
>>> > > > ci-cassandra.
>>> > > > 3.a Non-regressing is defined here as "Doesn't introduce any
>>> new
>>> > > test
>>> > > > failures; any new failures in CI are clearly not attributable to
>>> this
>>> > > diff"
>>> > > > 3.b: (NEW) After merging tickets, ci-cassandra runs against
>>> the
>>> > SHA
>>> > > > and the author gets an advisory update on the related JIRA for any
>>> new
>>> > > > errors on CI. The author of the ticket will take point on triaging
>>> this
>>> > > new
>>> > > > failure and either fixing (if clearly reproducible or related to
>>> their
>>> > > > work) or opening a JIRA for the intermittent failure and linking
>>> it in
>>> > > > butler (https://butler.cassandra.apache.org/#/)
>>> > > > 4. (NEW) The Build Lead role + Butler catches and documents all
>>> > failures
>>> > > > and anything that slips through the procedural cracks in 3.b;
>>> > resourcing
>>> > > > for fixing flakey tests TBD
>>> > > >
>>> > > > Our two TBD we can tackle separately from consensus on the above:
>>> > > > 1. Suite of tests on circle required to be considered ready for
>>> merge
>>> > > > 2. How we resource fixing flakey tests that are functionally
>>> impossible
>>> > > to
>>> > > > attribute without essentially fixing the flake
>>> > > >
>>> > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
>>> > > e.dimitrova@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > +1 (nb) on my end too, I second Mick
>>> > > > > Thanks for putting this together Josh
>>> > > > >
>>> > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org>
>>> > wrote:
>>> > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already
>>> do
>>> > > this)
>>> > > > > and
>>> > > > > > > get an advisory update on the related JIRA for any new
>>> errors on
>>> > > the
>>> > > > > run
>>> > > > > > of
>>> > > > > > > the SHA
>>> > > > > > >
>>> > > > > > > I strongly prefer we amend our process with 3.c.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > +1 Yup, this is the most important missing piece for me.
>>> > > > > >
>>> > > > > > I also wouldn't mind we word the responsibility of the author
>>> at
>>> > > > > > post-commit fault to be involved/leading in the fix. This
>>> > > incentivises
>>> > > > > > people to do 2+3 properly, and not push it onto the build role.
>>> > > > > >
>>> > > > >
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>> > > For additional commands, e-mail: dev-help@cassandra.apache.org
>>> > >
>>> > >
>>> >
>>>
>>
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
Here's a link to a draft article for the confluence wiki covering what we
discussed on this thread:
https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=199530280&draftShareId=7c72c252-918c-456b-9859-7d12e8fa9309&
Assuming this article accurately captures what we discussed here as well as
outstanding work, I'll get it published and integrated with the confluence
wiki and get the remainder of the work into a JIRA epic to be tracked.
On Fri, Dec 17, 2021 at 4:41 PM Joshua McKenzie <jm...@apache.org>
wrote:
> I'll get this into a draft article on the wiki so we can collab on those 3
> outstanding TBD's without further cluttering up the dev list. :)
>
> On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <
> e.dimitrova@gmail.com> wrote:
>
>> It’s indeed good call but I thought this will be addressed in a separate
>> document where we discuss required test suites to be run pre-commit. If
>> not
>> - then I guess we should add those things here too?
>>
>> On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jm...@apache.org>
>> wrote:
>>
>> > Good call; thanks for the reminder.
>> >
>> > So maybe add a
>> >
>> > 3.a: Run all new or modified tests through either local or remote
>> > multiplexer N (TBD - 50?) times (w/link to instructions, etc)
>> > 3.b Non-regressing is defined here...
>> > 3.c After merging tickets...
>> >
>> > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com>
>> > wrote:
>> >
>> > > Could we also add something about running new tests through the
>> > > multiplexer?
>> > >
>> > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <
>> jmckenzie@apache.org>
>> > > wrote:
>> > > >
>> > > > So to clarify it all in one place, the proposed new CI process we
>> > should
>> > > > test for consensus is:
>> > > >
>> > > > 1. Canonical CI for a release is ci-cassandra. We can optionally,
>> and
>> > in
>> > > > practice will, run circle as well but don't codify blocking on that.
>> > > > 2. (NEW) We don't release unless we get a fully green run.
>> > > > 3. Before any merge, you need either a non-regressing (i.e. no new
>> > > > failures) run of circleci with a (specific suite of tests TBD) or of
>> > > > ci-cassandra.
>> > > > 3.a Non-regressing is defined here as "Doesn't introduce any
>> new
>> > > test
>> > > > failures; any new failures in CI are clearly not attributable to
>> this
>> > > diff"
>> > > > 3.b: (NEW) After merging tickets, ci-cassandra runs against the
>> > SHA
>> > > > and the author gets an advisory update on the related JIRA for any
>> new
>> > > > errors on CI. The author of the ticket will take point on triaging
>> this
>> > > new
>> > > > failure and either fixing (if clearly reproducible or related to
>> their
>> > > > work) or opening a JIRA for the intermittent failure and linking it
>> in
>> > > > butler (https://butler.cassandra.apache.org/#/)
>> > > > 4. (NEW) The Build Lead role + Butler catches and documents all
>> > failures
>> > > > and anything that slips through the procedural cracks in 3.b;
>> > resourcing
>> > > > for fixing flakey tests TBD
>> > > >
>> > > > Our two TBD we can tackle separately from consensus on the above:
>> > > > 1. Suite of tests on circle required to be considered ready for
>> merge
>> > > > 2. How we resource fixing flakey tests that are functionally
>> impossible
>> > > to
>> > > > attribute without essentially fixing the flake
>> > > >
>> > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
>> > > e.dimitrova@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > +1 (nb) on my end too, I second Mick
>> > > > > Thanks for putting this together Josh
>> > > > >
>> > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do
>> > > this)
>> > > > > and
>> > > > > > > get an advisory update on the related JIRA for any new errors
>> on
>> > > the
>> > > > > run
>> > > > > > of
>> > > > > > > the SHA
>> > > > > > >
>> > > > > > > I strongly prefer we amend our process with 3.c.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > +1 Yup, this is the most important missing piece for me.
>> > > > > >
>> > > > > > I also wouldn't mind we word the responsibility of the author at
>> > > > > > post-commit fault to be involved/leading in the fix. This
>> > > incentivises
>> > > > > > people to do 2+3 properly, and not push it onto the build role.
>> > > > > >
>> > > > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > For additional commands, e-mail: dev-help@cassandra.apache.org
>> > >
>> > >
>> >
>>
>
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
I'll get this into a draft article on the wiki so we can collab on those 3
outstanding TBD's without further cluttering up the dev list. :)
On Fri, Dec 17, 2021 at 11:38 AM Ekaterina Dimitrova <e....@gmail.com>
wrote:
> It’s indeed good call but I thought this will be addressed in a separate
> document where we discuss required test suites to be run pre-commit. If not
> - then I guess we should add those things here too?
>
> On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > Good call; thanks for the reminder.
> >
> > So maybe add a
> >
> > 3.a: Run all new or modified tests through either local or remote
> > multiplexer N (TBD - 50?) times (w/link to instructions, etc)
> > 3.b Non-regressing is defined here...
> > 3.c After merging tickets...
> >
> > On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com>
> > wrote:
> >
> > > Could we also add something about running new tests through the
> > > multiplexer?
> > >
> > > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <jmckenzie@apache.org
> >
> > > wrote:
> > > >
> > > > So to clarify it all in one place, the proposed new CI process we
> > should
> > > > test for consensus is:
> > > >
> > > > 1. Canonical CI for a release is ci-cassandra. We can optionally, and
> > in
> > > > practice will, run circle as well but don't codify blocking on that.
> > > > 2. (NEW) We don't release unless we get a fully green run.
> > > > 3. Before any merge, you need either a non-regressing (i.e. no new
> > > > failures) run of circleci with a (specific suite of tests TBD) or of
> > > > ci-cassandra.
> > > > 3.a Non-regressing is defined here as "Doesn't introduce any new
> > > test
> > > > failures; any new failures in CI are clearly not attributable to this
> > > diff"
> > > > 3.b: (NEW) After merging tickets, ci-cassandra runs against the
> > SHA
> > > > and the author gets an advisory update on the related JIRA for any
> new
> > > > errors on CI. The author of the ticket will take point on triaging
> this
> > > new
> > > > failure and either fixing (if clearly reproducible or related to
> their
> > > > work) or opening a JIRA for the intermittent failure and linking it
> in
> > > > butler (https://butler.cassandra.apache.org/#/)
> > > > 4. (NEW) The Build Lead role + Butler catches and documents all
> > failures
> > > > and anything that slips through the procedural cracks in 3.b;
> > resourcing
> > > > for fixing flakey tests TBD
> > > >
> > > > Our two TBD we can tackle separately from consensus on the above:
> > > > 1. Suite of tests on circle required to be considered ready for merge
> > > > 2. How we resource fixing flakey tests that are functionally
> impossible
> > > to
> > > > attribute without essentially fixing the flake
> > > >
> > > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
> > > e.dimitrova@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (nb) on my end too, I second Mick
> > > > > Thanks for putting this together Josh
> > > > >
> > > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org>
> > wrote:
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do
> > > this)
> > > > > and
> > > > > > > get an advisory update on the related JIRA for any new errors
> on
> > > the
> > > > > run
> > > > > > of
> > > > > > > the SHA
> > > > > > >
> > > > > > > I strongly prefer we amend our process with 3.c.
> > > > > >
> > > > > >
> > > > > >
> > > > > > +1 Yup, this is the most important missing piece for me.
> > > > > >
> > > > > > I also wouldn't mind we word the responsibility of the author at
> > > > > > post-commit fault to be involved/leading in the fix. This
> > > incentivises
> > > > > > people to do 2+3 properly, and not push it onto the build role.
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > >
> > >
> >
>
Re: Clarifying CI release criteria
Posted by Ekaterina Dimitrova <e....@gmail.com>.
It’s indeed good call but I thought this will be addressed in a separate
document where we discuss required test suites to be run pre-commit. If not
- then I guess we should add those things here too?
On Fri, 17 Dec 2021 at 11:36, Joshua McKenzie <jm...@apache.org> wrote:
> Good call; thanks for the reminder.
>
> So maybe add a
>
> 3.a: Run all new or modified tests through either local or remote
> multiplexer N (TBD - 50?) times (w/link to instructions, etc)
> 3.b Non-regressing is defined here...
> 3.c After merging tickets...
>
> On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com>
> wrote:
>
> > Could we also add something about running new tests through the
> > multiplexer?
> >
> > On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <jm...@apache.org>
> > wrote:
> > >
> > > So to clarify it all in one place, the proposed new CI process we
> should
> > > test for consensus is:
> > >
> > > 1. Canonical CI for a release is ci-cassandra. We can optionally, and
> in
> > > practice will, run circle as well but don't codify blocking on that.
> > > 2. (NEW) We don't release unless we get a fully green run.
> > > 3. Before any merge, you need either a non-regressing (i.e. no new
> > > failures) run of circleci with a (specific suite of tests TBD) or of
> > > ci-cassandra.
> > > 3.a Non-regressing is defined here as "Doesn't introduce any new
> > test
> > > failures; any new failures in CI are clearly not attributable to this
> > diff"
> > > 3.b: (NEW) After merging tickets, ci-cassandra runs against the
> SHA
> > > and the author gets an advisory update on the related JIRA for any new
> > > errors on CI. The author of the ticket will take point on triaging this
> > new
> > > failure and either fixing (if clearly reproducible or related to their
> > > work) or opening a JIRA for the intermittent failure and linking it in
> > > butler (https://butler.cassandra.apache.org/#/)
> > > 4. (NEW) The Build Lead role + Butler catches and documents all
> failures
> > > and anything that slips through the procedural cracks in 3.b;
> resourcing
> > > for fixing flakey tests TBD
> > >
> > > Our two TBD we can tackle separately from consensus on the above:
> > > 1. Suite of tests on circle required to be considered ready for merge
> > > 2. How we resource fixing flakey tests that are functionally impossible
> > to
> > > attribute without essentially fixing the flake
> > >
> > > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
> > e.dimitrova@gmail.com>
> > > wrote:
> > >
> > > > +1 (nb) on my end too, I second Mick
> > > > Thanks for putting this together Josh
> > > >
> > > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org>
> wrote:
> > > >
> > > > > >
> > > > > >
> > > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do
> > this)
> > > > and
> > > > > > get an advisory update on the related JIRA for any new errors on
> > the
> > > > run
> > > > > of
> > > > > > the SHA
> > > > > >
> > > > > > I strongly prefer we amend our process with 3.c.
> > > > >
> > > > >
> > > > >
> > > > > +1 Yup, this is the most important missing piece for me.
> > > > >
> > > > > I also wouldn't mind we word the responsibility of the author at
> > > > > post-commit fault to be involved/leading in the fix. This
> > incentivises
> > > > > people to do 2+3 properly, and not push it onto the build role.
> > > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
>
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
Good call; thanks for the reminder.
So maybe add a
3.a: Run all new or modified tests through either local or remote
multiplexer N (TBD - 50?) times (w/link to instructions, etc)
3.b Non-regressing is defined here...
3.c After merging tickets...
On Fri, Dec 17, 2021 at 11:29 AM Brandon Williams <dr...@gmail.com> wrote:
> Could we also add something about running new tests through the
> multiplexer?
>
> On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <jm...@apache.org>
> wrote:
> >
> > So to clarify it all in one place, the proposed new CI process we should
> > test for consensus is:
> >
> > 1. Canonical CI for a release is ci-cassandra. We can optionally, and in
> > practice will, run circle as well but don't codify blocking on that.
> > 2. (NEW) We don't release unless we get a fully green run.
> > 3. Before any merge, you need either a non-regressing (i.e. no new
> > failures) run of circleci with a (specific suite of tests TBD) or of
> > ci-cassandra.
> > 3.a Non-regressing is defined here as "Doesn't introduce any new
> test
> > failures; any new failures in CI are clearly not attributable to this
> diff"
> > 3.b: (NEW) After merging tickets, ci-cassandra runs against the SHA
> > and the author gets an advisory update on the related JIRA for any new
> > errors on CI. The author of the ticket will take point on triaging this
> new
> > failure and either fixing (if clearly reproducible or related to their
> > work) or opening a JIRA for the intermittent failure and linking it in
> > butler (https://butler.cassandra.apache.org/#/)
> > 4. (NEW) The Build Lead role + Butler catches and documents all failures
> > and anything that slips through the procedural cracks in 3.b; resourcing
> > for fixing flakey tests TBD
> >
> > Our two TBD we can tackle separately from consensus on the above:
> > 1. Suite of tests on circle required to be considered ready for merge
> > 2. How we resource fixing flakey tests that are functionally impossible
> to
> > attribute without essentially fixing the flake
> >
> > On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <
> e.dimitrova@gmail.com>
> > wrote:
> >
> > > +1 (nb) on my end too, I second Mick
> > > Thanks for putting this together Josh
> > >
> > > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org> wrote:
> > >
> > > > >
> > > > >
> > > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do
> this)
> > > and
> > > > > get an advisory update on the related JIRA for any new errors on
> the
> > > run
> > > > of
> > > > > the SHA
> > > > >
> > > > > I strongly prefer we amend our process with 3.c.
> > > >
> > > >
> > > >
> > > > +1 Yup, this is the most important missing piece for me.
> > > >
> > > > I also wouldn't mind we word the responsibility of the author at
> > > > post-commit fault to be involved/leading in the fix. This
> incentivises
> > > > people to do 2+3 properly, and not push it onto the build role.
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: Clarifying CI release criteria
Posted by Brandon Williams <dr...@gmail.com>.
Could we also add something about running new tests through the multiplexer?
On Fri, Dec 17, 2021 at 10:23 AM Joshua McKenzie <jm...@apache.org> wrote:
>
> So to clarify it all in one place, the proposed new CI process we should
> test for consensus is:
>
> 1. Canonical CI for a release is ci-cassandra. We can optionally, and in
> practice will, run circle as well but don't codify blocking on that.
> 2. (NEW) We don't release unless we get a fully green run.
> 3. Before any merge, you need either a non-regressing (i.e. no new
> failures) run of circleci with a (specific suite of tests TBD) or of
> ci-cassandra.
> 3.a Non-regressing is defined here as "Doesn't introduce any new test
> failures; any new failures in CI are clearly not attributable to this diff"
> 3.b: (NEW) After merging tickets, ci-cassandra runs against the SHA
> and the author gets an advisory update on the related JIRA for any new
> errors on CI. The author of the ticket will take point on triaging this new
> failure and either fixing (if clearly reproducible or related to their
> work) or opening a JIRA for the intermittent failure and linking it in
> butler (https://butler.cassandra.apache.org/#/)
> 4. (NEW) The Build Lead role + Butler catches and documents all failures
> and anything that slips through the procedural cracks in 3.b; resourcing
> for fixing flakey tests TBD
>
> Our two TBD we can tackle separately from consensus on the above:
> 1. Suite of tests on circle required to be considered ready for merge
> 2. How we resource fixing flakey tests that are functionally impossible to
> attribute without essentially fixing the flake
>
> On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <e....@gmail.com>
> wrote:
>
> > +1 (nb) on my end too, I second Mick
> > Thanks for putting this together Josh
> >
> > On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org> wrote:
> >
> > > >
> > > >
> > > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do this)
> > and
> > > > get an advisory update on the related JIRA for any new errors on the
> > run
> > > of
> > > > the SHA
> > > >
> > > > I strongly prefer we amend our process with 3.c.
> > >
> > >
> > >
> > > +1 Yup, this is the most important missing piece for me.
> > >
> > > I also wouldn't mind we word the responsibility of the author at
> > > post-commit fault to be involved/leading in the fix. This incentivises
> > > people to do 2+3 properly, and not push it onto the build role.
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
So to clarify it all in one place, the proposed new CI process we should
test for consensus is:
1. Canonical CI for a release is ci-cassandra. We can optionally, and in
practice will, run circle as well but don't codify blocking on that.
2. (NEW) We don't release unless we get a fully green run.
3. Before any merge, you need either a non-regressing (i.e. no new
failures) run of circleci with a (specific suite of tests TBD) or of
ci-cassandra.
3.a Non-regressing is defined here as "Doesn't introduce any new test
failures; any new failures in CI are clearly not attributable to this diff"
3.b: (NEW) After merging tickets, ci-cassandra runs against the SHA
and the author gets an advisory update on the related JIRA for any new
errors on CI. The author of the ticket will take point on triaging this new
failure and either fixing (if clearly reproducible or related to their
work) or opening a JIRA for the intermittent failure and linking it in
butler (https://butler.cassandra.apache.org/#/)
4. (NEW) The Build Lead role + Butler catches and documents all failures
and anything that slips through the procedural cracks in 3.b; resourcing
for fixing flakey tests TBD
Our two TBD we can tackle separately from consensus on the above:
1. Suite of tests on circle required to be considered ready for merge
2. How we resource fixing flakey tests that are functionally impossible to
attribute without essentially fixing the flake
On Fri, Dec 17, 2021 at 10:56 AM Ekaterina Dimitrova <e....@gmail.com>
wrote:
> +1 (nb) on my end too, I second Mick
> Thanks for putting this together Josh
>
> On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org> wrote:
>
> > >
> > >
> > > 3.c: (NEW) After merging tickets, run ci-cassandra (already do this)
> and
> > > get an advisory update on the related JIRA for any new errors on the
> run
> > of
> > > the SHA
> > >
> > > I strongly prefer we amend our process with 3.c.
> >
> >
> >
> > +1 Yup, this is the most important missing piece for me.
> >
> > I also wouldn't mind we word the responsibility of the author at
> > post-commit fault to be involved/leading in the fix. This incentivises
> > people to do 2+3 properly, and not push it onto the build role.
> >
>
Re: Clarifying CI release criteria
Posted by Ekaterina Dimitrova <e....@gmail.com>.
+1 (nb) on my end too, I second Mick
Thanks for putting this together Josh
On Fri, 17 Dec 2021 at 10:48, Mick Semb Wever <mc...@apache.org> wrote:
> >
> >
> > 3.c: (NEW) After merging tickets, run ci-cassandra (already do this) and
> > get an advisory update on the related JIRA for any new errors on the run
> of
> > the SHA
> >
> > I strongly prefer we amend our process with 3.c.
>
>
>
> +1 Yup, this is the most important missing piece for me.
>
> I also wouldn't mind we word the responsibility of the author at
> post-commit fault to be involved/leading in the fix. This incentivises
> people to do 2+3 properly, and not push it onto the build role.
>
Re: Clarifying CI release criteria
Posted by Mick Semb Wever <mc...@apache.org>.
>
>
> 3.c: (NEW) After merging tickets, run ci-cassandra (already do this) and
> get an advisory update on the related JIRA for any new errors on the run of
> the SHA
>
> I strongly prefer we amend our process with 3.c.
+1 Yup, this is the most important missing piece for me.
I also wouldn't mind we word the responsibility of the author at
post-commit fault to be involved/leading in the fix. This incentivises
people to do 2+3 properly, and not push it onto the build role.
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
What if we tried the following:
1. Canonical CI for a release is ci-cassandra. We can optionally, and in
practice will, run circle as well but don't codify blocking on that.
2. (NEW) We don't release unless we get a fully green run.
3. Before any merge, you need either a non-regressing (i.e. no new
failures) run of circleci or of ci-cassandra.
3.a Non-regressing is defined here as "Doesn't introduce any new test
failures; any new failures in CI are clearly not attributable to this diff"
4. (NEW) The Build Lead role + Butler catches and documents new
intermittent failures; it's unspecified how we resource fixing those
collectively at this time
2 raises the specter of flaky tests unique to apache infra greatly delaying
releases. I can think of a few options to help keep us from regressing on
ci-cassandra (numbered to indicate where they fit in / replace the flow
above):
3: (NEW) Before merging tickets, block on a clean run of ci-cassandra (need
something like merge trains; could automate merging, hard / impossible
w/merge commits)
3: (NEW) Before merging tickets, run ci-cassandra and get an advisory
update on the related JIRA (extra ci runtime burden; long delays w/out CI
tests or infra optimization)
3.c: (NEW) After merging tickets, run ci-cassandra (already do this) and
get an advisory update on the related JIRA for any new errors on the run of
the SHA
I strongly prefer we amend our process with 3.c. I'm pretty sure we could
get granular enough to compute any new test failures and highlight them in
the JIRA ticket and link to the run + the previous run. I believe this
would greatly tighten the loop between a delta and a failure for a variety
of tests, and 4 above would provide the fail-safe for us to catch and
address flakes far earlier than the current model.
~Josh
On Thu, Dec 16, 2021 at 1:20 PM Mick Semb Wever <mc...@apache.org> wrote:
> >
> >
> > > ci-cassandra.a.o needs to be our canonical CI
> >
> > it's the only one fully usable by a volunteer based
> >
> >
> > only green in both counts as green
> >
> > I think today might just be my day to annoy you Mick. :D Sorry!
> >
>
>
> On the day I'm laid up in bed with a cold.
> Go for gold :-)
>
> I think this is contradictory. We can't require circle to be green for a
> > release if the free tier usage of it a) doesn't pass tests, and/or b)
> > requires a license incompatible w/some contributors. That effectively
> would
> > make circle + asf ci our canonical ci, right?
> >
>
>
> That's taking it out (or twisting) my context a bit, let me explain…
>
> First, I did not mean the free tier. It is not usable AFAIK. It could be
> updated so it was constrained in what it could run and was stable, but then
> it's not complete so there's limited value here. IMHO plugging in GitHub
> Actions to do a very basic build+test would hit a larger newcomer audience.
>
> Second, I didn't mean one *had* to run both. Just like post-commit will
> catch things, just so long as that breakage comes around to you and you
> accept your involvement in it. We (the whole community) need to help out
> when the author cannot reproduce/debug the failure, and this isn't just
> limited to premium circleci.
>
>
> less flakies than the previous release
> >
> > This statement makes me wary. :) Why not "no test failures"?
> >
>
>
> More than happy to go for that. And I damn hope we are there for our next
> major release.
> This statement was more just a preference to lean on the more pragmatic
> side. We know our north star, keep moving towards it.
>
Re: Clarifying CI release criteria
Posted by Mick Semb Wever <mc...@apache.org>.
>
>
> > ci-cassandra.a.o needs to be our canonical CI
>
> it's the only one fully usable by a volunteer based
>
>
> only green in both counts as green
>
> I think today might just be my day to annoy you Mick. :D Sorry!
>
On the day I'm laid up in bed with a cold.
Go for gold :-)
I think this is contradictory. We can't require circle to be green for a
> release if the free tier usage of it a) doesn't pass tests, and/or b)
> requires a license incompatible w/some contributors. That effectively would
> make circle + asf ci our canonical ci, right?
>
That's taking it out (or twisting) my context a bit, let me explain…
First, I did not mean the free tier. It is not usable AFAIK. It could be
updated so it was constrained in what it could run and was stable, but then
it's not complete so there's limited value here. IMHO plugging in GitHub
Actions to do a very basic build+test would hit a larger newcomer audience.
Second, I didn't mean one *had* to run both. Just like post-commit will
catch things, just so long as that breakage comes around to you and you
accept your involvement in it. We (the whole community) need to help out
when the author cannot reproduce/debug the failure, and this isn't just
limited to premium circleci.
less flakies than the previous release
>
> This statement makes me wary. :) Why not "no test failures"?
>
More than happy to go for that. And I damn hope we are there for our next
major release.
This statement was more just a preference to lean on the more pragmatic
side. We know our north star, keep moving towards it.
Re: Clarifying CI release criteria
Posted by Joshua McKenzie <jm...@apache.org>.
>
> ci-cassandra.a.o needs to be our canonical CI
it's the only one fully usable by a volunteer based
only green in both counts as green
I think today might just be my day to annoy you Mick. :D Sorry!
I think this is contradictory. We can't require circle to be green for a
release if the free tier usage of it a) doesn't pass tests, and/or b)
requires a license incompatible w/some contributors. That effectively would
make circle + asf ci our canonical ci, right?
less flakies than the previous release
This statement makes me wary. :) Why not "no test failures"?
~Josh
On Thu, Dec 16, 2021 at 10:41 AM Mick Semb Wever <mc...@apache.org> wrote:
> >
> > Mick, did I miss anything?
> >
>
>
> Yup, ci-cassandra.a.o needs to be our canonical CI because of the
> reasons you state, and because it's the only one fully usable by a
> volunteer based PMC, i.e. community donated hardware, controlled by
> ASF, and no need for a proprietary licence.
>
> I think we should be making that post-commit pipeline as comprehensive
> as possible.
>
> I also really appreciate that we have CircleCI, apart from its
> benefits (particularly speed for pre-commit), it provides valuable
> double-accounting over CI results. A few times we have broken because
> one system didn't properly catch errors. It can be frustrating that we
> get green in one system and then it breaks in the other, but if we
> embrace this strengthening tactic: only green in both counts as green;
> then we end up in a better place.
>
> Let's continue to make post-commit CI as comprehensive as possible,
> and figure out as-we-go what the most efficient pre-commit gate is,
> and accept that if you break it post-commit you still broke it.
>
> For releases I think we have to have no hard failures, and less
> flakies than the previous release. Using Butler we can more easily
> visualise failures from flakies from CI infra instabilities, and
> ensure tickets are created and if a dev broke it that they are
> assigned accordingly (and prioritise its fix above all other
> non-critical work in the project).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: Clarifying CI release criteria
Posted by Mick Semb Wever <mc...@apache.org>.
>
> Mick, did I miss anything?
>
Yup, ci-cassandra.a.o needs to be our canonical CI because of the
reasons you state, and because it's the only one fully usable by a
volunteer based PMC, i.e. community donated hardware, controlled by
ASF, and no need for a proprietary licence.
I think we should be making that post-commit pipeline as comprehensive
as possible.
I also really appreciate that we have CircleCI, apart from its
benefits (particularly speed for pre-commit), it provides valuable
double-accounting over CI results. A few times we have broken because
one system didn't properly catch errors. It can be frustrating that we
get green in one system and then it breaks in the other, but if we
embrace this strengthening tactic: only green in both counts as green;
then we end up in a better place.
Let's continue to make post-commit CI as comprehensive as possible,
and figure out as-we-go what the most efficient pre-commit gate is,
and accept that if you break it post-commit you still broke it.
For releases I think we have to have no hard failures, and less
flakies than the previous release. Using Butler we can more easily
visualise failures from flakies from CI infra instabilities, and
ensure tickets are created and if a dev broke it that they are
assigned accordingly (and prioritise its fix above all other
non-critical work in the project).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Clarifying CI release criteria
Posted by Ekaterina Dimitrova <e....@gmail.com>.
As far as I am aware Jenkins is the official CI running on Apache infra. It
preserves historical data and every committer and PMC member has access to
run all tests pre-commit. We have project builds after every commit.
My understanding about CircleCI - free tier is available to everyone but
not all tests can be executed with the free account because of lack of
enough resources.
Not everyone is willing to pay for premium accounts. We don’t have project
account which runs the suite post-commit and has project build history.
Pre-4.0 we used Jenkins to confirm CI situation.
Also, in my opinion actually CircleCI sometimes reveals more issues than
Jenkins as people run it with different configurations and more things can
pop up.
In regards to the Build Lead role, Butler gets the info from Jenkins and
the history is available to everyone - contributor, committer, PMC.
“ Also, if a bunch of us are using Circle
> pre-commit and we're seeing test flakes unique to the ASF Jenkins
execution
> environment, we're going to naturally accumulate debt as we lead up to a
> release if we block on ASF Jenkins being green.”
If we get into the cadence of checking regularly post-commit in Jenkins, I
hope this won’t happen.
Mick, did I miss anything?
On Thu, 16 Dec 2021 at 2:28, Berenguer Blasi <be...@gmail.com>
wrote:
> Iirc we settled on jenkins being the official CI. Hence we need a green
> jenkins. I can't find the reference now.
>
> Green being all test pass except the 2 or 3 flakies that already
> defeated 3 committers like the connections test. Last time I looked 4.0
> was almost there pending solving the early process termination failures.
>
> On 15/12/21 23:44, Joshua McKenzie wrote:
> > What CI criteria are we using to gate minor and major releases? For
> > example: All tests from all suites on all configurations? A subset?
> >
> > All the test suite columns on the following page?
> > https://ci-cassandra.apache.org/job/Cassandra-trunk/
> >
> > All the test suites in all combinations on all JDKs on circle-ci?
> >
> > This is particularly relevant to how we craft the Build Lead role
> w/regards
> > to what we'll be triaging. Also, if a bunch of us are using Circle
> > pre-commit and we're seeing test flakes unique to the ASF Jenkins
> execution
> > environment, we're going to naturally accumulate debt as we lead up to a
> > release if we block on ASF Jenkins being green.
> >
> > Notably, I can't find this clarified in our documentation on testing or
> the
> > release process, nor is it clarified on the confluence wiki article on
> the
> > Release Lifecycle.
> >
> > Thanks.
> >
> > ~Josh
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: Clarifying CI release criteria
Posted by Berenguer Blasi <be...@gmail.com>.
Iirc we settled on jenkins being the official CI. Hence we need a green
jenkins. I can't find the reference now.
Green being all test pass except the 2 or 3 flakies that already
defeated 3 committers like the connections test. Last time I looked 4.0
was almost there pending solving the early process termination failures.
On 15/12/21 23:44, Joshua McKenzie wrote:
> What CI criteria are we using to gate minor and major releases? For
> example: All tests from all suites on all configurations? A subset?
>
> All the test suite columns on the following page?
> https://ci-cassandra.apache.org/job/Cassandra-trunk/
>
> All the test suites in all combinations on all JDKs on circle-ci?
>
> This is particularly relevant to how we craft the Build Lead role w/regards
> to what we'll be triaging. Also, if a bunch of us are using Circle
> pre-commit and we're seeing test flakes unique to the ASF Jenkins execution
> environment, we're going to naturally accumulate debt as we lead up to a
> release if we block on ASF Jenkins being green.
>
> Notably, I can't find this clarified in our documentation on testing or the
> release process, nor is it clarified on the confluence wiki article on the
> Release Lifecycle.
>
> Thanks.
>
> ~Josh
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org