You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Chip Childers <ch...@sungard.com> on 2013/03/05 19:32:31 UTC
[QA][DISCUSS] Should we document formal release criteria?
Hi all,
Some IRC discussion triggered this thought:
Should we document our formal release criteria? By that, I mean -
should we document the criteria that we use to claim that we are in a
position to cut an actual Release Candidate for voting?
I realized that I've had my own view sitting in my head, but obviously
that means that nobody else knows it!
Thoughts about what our criteria should be?
Here's what's in my head:
No Blocker Bugs outstanding
No Critical Bugs outstanding (unless there is a work-around, and only <
5 like that)
Builds all work
Packaging all works for the selected OS's
Docs and Translation updates checked in
Thoughts?
-chip
RE: [QA][DISCUSS] Should we document formal release criteria?
Posted by Alex Huang <Al...@citrix.com>.
ABSOLUTELY!
--Alex
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: Tuesday, March 5, 2013 10:33 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: [QA][DISCUSS] Should we document formal release criteria?
>
> Hi all,
>
> Some IRC discussion triggered this thought:
>
> Should we document our formal release criteria? By that, I mean - should we
> document the criteria that we use to claim that we are in a position to cut an
> actual Release Candidate for voting?
>
> I realized that I've had my own view sitting in my head, but obviously that
> means that nobody else knows it!
>
> Thoughts about what our criteria should be?
>
> Here's what's in my head:
>
> No Blocker Bugs outstanding
> No Critical Bugs outstanding (unless there is a work-around, and only <
> 5 like that)
> Builds all work
> Packaging all works for the selected OS's Docs and Translation updates
> checked in
>
>
> Thoughts?
>
> -chip
RE: [QA][DISCUSS] Should we document formal release criteria?
Posted by Sudha Ponnaganti <su...@citrix.com>.
We are already in defect fix mode right?? ( even though blockers took longer time to get resolved which prevented QA to make good progress)
Now we are getting features blockers but not QA blockers i.e only few feature validation is blocked but QA is making progress on rest of features.
-----Original Message-----
From: Sebastien Goasguen [mailto:runseb@gmail.com]
Sent: Wednesday, March 06, 2013 2:15 AM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [QA][DISCUSS] Should we document formal release criteria?
On Mar 5, 2013, at 2:18 PM, Joe Brockmeier <jz...@zonker.net> wrote:
> On Tue, Mar 5, 2013, at 01:14 PM, Chip Childers wrote:
>> On Tue, Mar 05, 2013 at 11:11:51AM -0800, Alex Huang wrote:
>>> So in this definition, MAJOR and above must be fixed before a release is cut?
>>
>> Personally, I've used the "no blockers and <5 critical with work
>> arounds" as the criteria, using the definitions I offered.
>
> How many "Major" bugs are acceptable? Currently we have 187. Even if
> they're not going to completely disrupt use of the platform, it seems
> to me that releasing with >100 major bugs is a bit much.
Bit off-thread, but could we have a bug sprint ?
>
> Best,
>
> jzb
> --
> Joe Brockmeier
> jzb@zonker.net
> Twitter: @jzb
> http://www.dissociatedpress.net/
Re: [QA][DISCUSS] Should we document formal release criteria?
Posted by Sebastien Goasguen <ru...@gmail.com>.
On Mar 5, 2013, at 2:18 PM, Joe Brockmeier <jz...@zonker.net> wrote:
> On Tue, Mar 5, 2013, at 01:14 PM, Chip Childers wrote:
>> On Tue, Mar 05, 2013 at 11:11:51AM -0800, Alex Huang wrote:
>>> So in this definition, MAJOR and above must be fixed before a release is cut?
>>
>> Personally, I've used the "no blockers and <5 critical with work
>> arounds" as the criteria, using the definitions I offered.
>
> How many "Major" bugs are acceptable? Currently we have 187. Even if
> they're not going to completely disrupt use of the platform, it seems to
> me that releasing with >100 major bugs is a bit much.
Bit off-thread, but could we have a bug sprint ?
>
> Best,
>
> jzb
> --
> Joe Brockmeier
> jzb@zonker.net
> Twitter: @jzb
> http://www.dissociatedpress.net/
RE: [QA][DISCUSS] Should we document formal release criteria?
Posted by Sudha Ponnaganti <su...@citrix.com>.
I would think we need to address up to Major ( Blockers/Critical/Major ).
Major list can be reviewed / Triaged and issues which doesn't need to be fixed can be deferred. As this is time based release, incoming and critical defect criteria that Chip mentioned should be fine.
-----Original Message-----
From: Joe Brockmeier [mailto:jzb@zonker.net]
Sent: Tuesday, March 05, 2013 11:19 AM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [QA][DISCUSS] Should we document formal release criteria?
On Tue, Mar 5, 2013, at 01:14 PM, Chip Childers wrote:
> On Tue, Mar 05, 2013 at 11:11:51AM -0800, Alex Huang wrote:
> > So in this definition, MAJOR and above must be fixed before a release is cut?
>
> Personally, I've used the "no blockers and <5 critical with work
> arounds" as the criteria, using the definitions I offered.
How many "Major" bugs are acceptable? Currently we have 187. Even if they're not going to completely disrupt use of the platform, it seems to me that releasing with >100 major bugs is a bit much.
Best,
jzb
--
Joe Brockmeier
jzb@zonker.net
Twitter: @jzb
http://www.dissociatedpress.net/
Re: [QA][DISCUSS] Should we document formal release criteria?
Posted by Joe Brockmeier <jz...@zonker.net>.
On Tue, Mar 5, 2013, at 01:14 PM, Chip Childers wrote:
> On Tue, Mar 05, 2013 at 11:11:51AM -0800, Alex Huang wrote:
> > So in this definition, MAJOR and above must be fixed before a release is cut?
>
> Personally, I've used the "no blockers and <5 critical with work
> arounds" as the criteria, using the definitions I offered.
How many "Major" bugs are acceptable? Currently we have 187. Even if
they're not going to completely disrupt use of the platform, it seems to
me that releasing with >100 major bugs is a bit much.
Best,
jzb
--
Joe Brockmeier
jzb@zonker.net
Twitter: @jzb
http://www.dissociatedpress.net/
Re: [QA][DISCUSS] Should we document formal release criteria?
Posted by Chip Childers <ch...@sungard.com>.
On Tue, Mar 05, 2013 at 11:11:51AM -0800, Alex Huang wrote:
> So in this definition, MAJOR and above must be fixed before a release is cut?
Personally, I've used the "no blockers and <5 critical with work
arounds" as the criteria, using the definitions I offered.
>
> --Alex
>
> > -----Original Message-----
> > From: Chip Childers [mailto:chip.childers@sungard.com]
> > Sent: Tuesday, March 5, 2013 11:03 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: [QA][DISCUSS] Should we document formal release criteria?
> >
> > On Tue, Mar 05, 2013 at 01:39:50PM -0500, David Nalley wrote:
> > > On Tue, Mar 5, 2013 at 1:32 PM, Chip Childers <ch...@sungard.com>
> > wrote:
> > > > Hi all,
> > > >
> > > > Some IRC discussion triggered this thought:
> > > >
> > > > Should we document our formal release criteria? By that, I mean -
> > > > should we document the criteria that we use to claim that we are in
> > > > a position to cut an actual Release Candidate for voting?
> > > >
> > >
> > > YES!!!
> > >
> > >
> > > > I realized that I've had my own view sitting in my head, but
> > > > obviously that means that nobody else knows it!
> > > >
> > > > Thoughts about what our criteria should be?
> > > >
> > > > Here's what's in my head:
> > > >
> > > > No Blocker Bugs outstanding
> > > > No Critical Bugs outstanding (unless there is a work-around, and
> > > > only <
> > > > 5 like that)
> > >
> > > So define for me what is a blocker and critical.
> > > I have my own internal definition, but if we have a standard that says
> > > 'no blocker and critical bugs' then we are just doing blind queries
> > > against Jira and maybe even changing the value on a bug so we can be
> > > releasable.
> >
> >
> > Here's how I've defined them in other environments. This may not work for
> > us, but it's a starting point:
> >
> > Blocker - Bugs which result in data corruption or inaccuracies in the data
> > produced or manipulated by the product under test. Discrepancies
> > compromising the functionality of the product under test. No workaround is
> > available. Any security vulnerability.
> >
> > Critical - Commonly used elements of the product under test cause errors
> > which result in the product under test to stop functioning as designed.
> > Users are very likely to encounter the error. A workaround may be available.
> >
> > And just to be complete:
> >
> > Major - Errors that does not affect critical functions. Internal errors that will
> > not be seen by the user. A work around may be available.
> >
> > Minor - Cosmetic errors (including grammar and spelling).
> >
> > Trivial - Silly things that really don't matter all that much
>
>
RE: [QA][DISCUSS] Should we document formal release criteria?
Posted by Alex Huang <Al...@citrix.com>.
So in this definition, MAJOR and above must be fixed before a release is cut?
--Alex
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: Tuesday, March 5, 2013 11:03 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [QA][DISCUSS] Should we document formal release criteria?
>
> On Tue, Mar 05, 2013 at 01:39:50PM -0500, David Nalley wrote:
> > On Tue, Mar 5, 2013 at 1:32 PM, Chip Childers <ch...@sungard.com>
> wrote:
> > > Hi all,
> > >
> > > Some IRC discussion triggered this thought:
> > >
> > > Should we document our formal release criteria? By that, I mean -
> > > should we document the criteria that we use to claim that we are in
> > > a position to cut an actual Release Candidate for voting?
> > >
> >
> > YES!!!
> >
> >
> > > I realized that I've had my own view sitting in my head, but
> > > obviously that means that nobody else knows it!
> > >
> > > Thoughts about what our criteria should be?
> > >
> > > Here's what's in my head:
> > >
> > > No Blocker Bugs outstanding
> > > No Critical Bugs outstanding (unless there is a work-around, and
> > > only <
> > > 5 like that)
> >
> > So define for me what is a blocker and critical.
> > I have my own internal definition, but if we have a standard that says
> > 'no blocker and critical bugs' then we are just doing blind queries
> > against Jira and maybe even changing the value on a bug so we can be
> > releasable.
>
>
> Here's how I've defined them in other environments. This may not work for
> us, but it's a starting point:
>
> Blocker - Bugs which result in data corruption or inaccuracies in the data
> produced or manipulated by the product under test. Discrepancies
> compromising the functionality of the product under test. No workaround is
> available. Any security vulnerability.
>
> Critical - Commonly used elements of the product under test cause errors
> which result in the product under test to stop functioning as designed.
> Users are very likely to encounter the error. A workaround may be available.
>
> And just to be complete:
>
> Major - Errors that does not affect critical functions. Internal errors that will
> not be seen by the user. A work around may be available.
>
> Minor - Cosmetic errors (including grammar and spelling).
>
> Trivial - Silly things that really don't matter all that much
Re: [QA][DISCUSS] Should we document formal release criteria?
Posted by Chip Childers <ch...@sungard.com>.
On Tue, Mar 05, 2013 at 01:39:50PM -0500, David Nalley wrote:
> On Tue, Mar 5, 2013 at 1:32 PM, Chip Childers <ch...@sungard.com> wrote:
> > Hi all,
> >
> > Some IRC discussion triggered this thought:
> >
> > Should we document our formal release criteria? By that, I mean -
> > should we document the criteria that we use to claim that we are in a
> > position to cut an actual Release Candidate for voting?
> >
>
> YES!!!
>
>
> > I realized that I've had my own view sitting in my head, but obviously
> > that means that nobody else knows it!
> >
> > Thoughts about what our criteria should be?
> >
> > Here's what's in my head:
> >
> > No Blocker Bugs outstanding
> > No Critical Bugs outstanding (unless there is a work-around, and only <
> > 5 like that)
>
> So define for me what is a blocker and critical.
> I have my own internal definition, but if we have a standard that says
> 'no blocker and critical bugs' then we are just doing blind queries
> against Jira and maybe even changing the value on a bug so we can be
> releasable.
Here's how I've defined them in other environments. This may not work
for us, but it's a starting point:
Blocker - Bugs which result in data corruption or
inaccuracies in the data produced or manipulated by the product under
test. Discrepancies compromising the functionality of the product under
test. No workaround is available. Any security vulnerability.
Critical - Commonly used elements of the product under test cause errors
which result in the product under test to stop functioning as designed.
Users are very likely to encounter the error. A workaround may be available.
And just to be complete:
Major - Errors that does not affect critical functions. Internal
errors that will not be seen by the user. A work around may be available.
Minor - Cosmetic errors (including grammar and spelling).
Trivial - Silly things that really don't matter all that much
Re: [QA][DISCUSS] Should we document formal release criteria?
Posted by David Nalley <da...@gnsa.us>.
On Tue, Mar 5, 2013 at 1:32 PM, Chip Childers <ch...@sungard.com> wrote:
> Hi all,
>
> Some IRC discussion triggered this thought:
>
> Should we document our formal release criteria? By that, I mean -
> should we document the criteria that we use to claim that we are in a
> position to cut an actual Release Candidate for voting?
>
YES!!!
> I realized that I've had my own view sitting in my head, but obviously
> that means that nobody else knows it!
>
> Thoughts about what our criteria should be?
>
> Here's what's in my head:
>
> No Blocker Bugs outstanding
> No Critical Bugs outstanding (unless there is a work-around, and only <
> 5 like that)
So define for me what is a blocker and critical.
I have my own internal definition, but if we have a standard that says
'no blocker and critical bugs' then we are just doing blind queries
against Jira and maybe even changing the value on a bug so we can be
releasable.
> Builds all work
> Packaging all works for the selected OS's
> Docs and Translation updates checked in
>
>