You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by David Nalley <da...@gnsa.us> on 2012/11/10 13:00:49 UTC
Re: [DISCUSS] QA/Testing focus on 4.1
Hi folks - haven't followed up on this in several weeks - also haven't
had a chance to do much - I've been on holiday and traveling to ACEU.
I did have the opportunity to talk to Gavin McDonald of ASF Infra fame
while at ACEU this past week and discuss with him some ideas for our
testing. He did recommend we actually use builders local to ASF infra
for things we want to happen fast (compile tests, unit tests, etc) as
they are effectively on the same network, and thus things like
checkouts happen much faster than connections over the WAN.
I am going to start having my weekly IRC time at 1800 UTC in
#cloudstack-dev on Wednesdays - all are welcome to join, I know it is
inconvenient for many - and so don't hesitate to do things outside of
that time - I am going to start working on migrating some of the
simpler stuff to builds.a.o.
--David
On Wed, Oct 17, 2012 at 6:22 PM, David Nalley <da...@gnsa.us> wrote:
> Warning: This is a long ponderous email. Stop now and get your cup of
> coffee/tea/$beverage.
>
> Hi folks,
>
> I've been having some off hand conversations with folks in other open
> source communities and rereading a number of good books of late (most
> notably Continuous Delivery by Jez Humble) and I would like to toss
> this idea out and see what folks think of it:
>
> CloudStack is large, often complex, piece of software. There are few
> people who truly grok all of CloudStack, and because of the diversity
> of environments it can be deployed in it can be truly bewildering to
> test fully. That said, I noticed a number of things that became last
> minute, blocker bugs, and it wasn't overly esoteric configurations, it
> was code paths that we just hadn't excercised in testing. One if these
> would have even been easily detected and fixed we had been testing
> installation alone. Additionally, as a project we are dramatically
> growing. The number of people adding non-trivial amounts of code is
> growing, and that makes ad hoc QA even more difficult.
>
> So my proposal in short is that we focus on testing, beginning
> immediately and erect what is effectively a continuous deployment
> pipeline - such that our confidence in our codebase is such that we
> could arbitrarily decide to release if we so desired. I don't think
> that this is a one release type of project, indeed I think it's really
> more of a culture shift than a technical project. The big shift is
> that EVERYONE must be responsible for a quality release. To that end,
> I'd propose the following tenets if we choose to adopt this:
>
> * Tests become the Andon cord[1] for the entire project. When a test
> fails we stop - additional commits don't happen - we find out what is
> wrong and fix it. More on this in a bit.
>
> * Tests (specifically automated tests) become part of our culture.
> * New features should come replete with both unit and integration
> tests. I am tempted to suggest a certain percentage of coverage, but I
> worry that it is a red herring; particularly given our dismal current
> unit test coverage.
> * Blocker and critical bugs must have automated tests that get
> committed as part of being qualified for closing.
>
> * We dedicate a non-trivial portion of our energy to enhancing not
> just the quantity and quality of our tests, but also on making it
> highly automated, and capable of delivering fast feedback. Ideally we
> would know within minutes if a commit broke unit tests, within hours
> if a commit failed in integration testing.
>
> I also know that this isn't a new idea. Lots of people have been
> focused on automated testing as part of our ongoing development. The
> only difference here is that I am actively asking you not to solely
> depend on those folks to do the work for you, but to make testing a
> part of the problem that you have to solve here. To be clear the goal
> isn't to be perfect and problem free with every commit - things will
> break. (If you've followed things at all in CloudStack you'll know
> that I've broken builds more than once)
>
> Pipeline I'd like to see for 4.1:
>
> 1. RAT test (fail this and we have IP problems)
> 2. Compile test (does it actually compile)
> 3. Unit tests
> 4. Package building
> 5. Automated installation (multiple platforms, does it actually
> install from the packages)
> 6. Integration tests (aka Marvin running against virtualized or real
> CS deployments)
>
> Clearly the above isn't an end/all be all for testing, but perhaps we
> can get some of this going in the 4.1 timeframe. There are also
> clearly corner cases (building marvin, building api docs, building
> official documentation) that need to be included in the pipeline as
> well. But the principle is that we won't move on past our failure
> until that failure is resolved.
>
> Immediate Action Items:
>
> Whether we adopt this or not, I plan on showing up on IRC once a week
> to work on testing in some form or another for an hour or two. I will
> also be cajoling people to join me. I might be working on
> infrastructure tasks. Obviously we have people scattered across the
> globe, so it's not the only time to work on testing, but you are
> welcome to join me.
>
> I am curious to hear others thoughts, comments, or flames? Is this
> something we should espouse as we are close to 4.0.0 releasing and
> turning our focus on 4.1?
>
> --David
>
> [1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
RE: [DISCUSS] QA/Testing focus on 4.1
Posted by Alex Huang <Al...@citrix.com>.
Hey Dave,
I didn't even know you started this all important email.
Definitely +1 on the ideas.
But I also think cloudstack is too many components to just start writing tests. Can you take a look and comment at my proposal to refactor cloudstack? It will be much easier to write tests once we break cloudstack into smaller pieces. Part of the reason why I'm making that proposal is because I think it will help in testing the different pieces in cloudstack today.
Thanks.
--Alex
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Saturday, November 10, 2012 4:01 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] QA/Testing focus on 4.1
>
> Hi folks - haven't followed up on this in several weeks - also haven't
> had a chance to do much - I've been on holiday and traveling to ACEU.
>
> I did have the opportunity to talk to Gavin McDonald of ASF Infra fame
> while at ACEU this past week and discuss with him some ideas for our
> testing. He did recommend we actually use builders local to ASF infra
> for things we want to happen fast (compile tests, unit tests, etc) as
> they are effectively on the same network, and thus things like
> checkouts happen much faster than connections over the WAN.
>
> I am going to start having my weekly IRC time at 1800 UTC in
> #cloudstack-dev on Wednesdays - all are welcome to join, I know it is
> inconvenient for many - and so don't hesitate to do things outside of
> that time - I am going to start working on migrating some of the
> simpler stuff to builds.a.o.
>
> --David
>
>
> On Wed, Oct 17, 2012 at 6:22 PM, David Nalley <da...@gnsa.us> wrote:
> > Warning: This is a long ponderous email. Stop now and get your cup of
> > coffee/tea/$beverage.
> >
> > Hi folks,
> >
> > I've been having some off hand conversations with folks in other open
> > source communities and rereading a number of good books of late (most
> > notably Continuous Delivery by Jez Humble) and I would like to toss
> > this idea out and see what folks think of it:
> >
> > CloudStack is large, often complex, piece of software. There are few
> > people who truly grok all of CloudStack, and because of the diversity
> > of environments it can be deployed in it can be truly bewildering to
> > test fully. That said, I noticed a number of things that became last
> > minute, blocker bugs, and it wasn't overly esoteric configurations, it
> > was code paths that we just hadn't excercised in testing. One if these
> > would have even been easily detected and fixed we had been testing
> > installation alone. Additionally, as a project we are dramatically
> > growing. The number of people adding non-trivial amounts of code is
> > growing, and that makes ad hoc QA even more difficult.
> >
> > So my proposal in short is that we focus on testing, beginning
> > immediately and erect what is effectively a continuous deployment
> > pipeline - such that our confidence in our codebase is such that we
> > could arbitrarily decide to release if we so desired. I don't think
> > that this is a one release type of project, indeed I think it's really
> > more of a culture shift than a technical project. The big shift is
> > that EVERYONE must be responsible for a quality release. To that end,
> > I'd propose the following tenets if we choose to adopt this:
> >
> > * Tests become the Andon cord[1] for the entire project. When a test
> > fails we stop - additional commits don't happen - we find out what is
> > wrong and fix it. More on this in a bit.
> >
> > * Tests (specifically automated tests) become part of our culture.
> > * New features should come replete with both unit and integration
> > tests. I am tempted to suggest a certain percentage of coverage, but I
> > worry that it is a red herring; particularly given our dismal current
> > unit test coverage.
> > * Blocker and critical bugs must have automated tests that get
> > committed as part of being qualified for closing.
> >
> > * We dedicate a non-trivial portion of our energy to enhancing not
> > just the quantity and quality of our tests, but also on making it
> > highly automated, and capable of delivering fast feedback. Ideally we
> > would know within minutes if a commit broke unit tests, within hours
> > if a commit failed in integration testing.
> >
> > I also know that this isn't a new idea. Lots of people have been
> > focused on automated testing as part of our ongoing development. The
> > only difference here is that I am actively asking you not to solely
> > depend on those folks to do the work for you, but to make testing a
> > part of the problem that you have to solve here. To be clear the goal
> > isn't to be perfect and problem free with every commit - things will
> > break. (If you've followed things at all in CloudStack you'll know
> > that I've broken builds more than once)
> >
> > Pipeline I'd like to see for 4.1:
> >
> > 1. RAT test (fail this and we have IP problems)
> > 2. Compile test (does it actually compile)
> > 3. Unit tests
> > 4. Package building
> > 5. Automated installation (multiple platforms, does it actually
> > install from the packages)
> > 6. Integration tests (aka Marvin running against virtualized or real
> > CS deployments)
> >
> > Clearly the above isn't an end/all be all for testing, but perhaps we
> > can get some of this going in the 4.1 timeframe. There are also
> > clearly corner cases (building marvin, building api docs, building
> > official documentation) that need to be included in the pipeline as
> > well. But the principle is that we won't move on past our failure
> > until that failure is resolved.
> >
> > Immediate Action Items:
> >
> > Whether we adopt this or not, I plan on showing up on IRC once a week
> > to work on testing in some form or another for an hour or two. I will
> > also be cajoling people to join me. I might be working on
> > infrastructure tasks. Obviously we have people scattered across the
> > globe, so it's not the only time to work on testing, but you are
> > welcome to join me.
> >
> > I am curious to hear others thoughts, comments, or flames? Is this
> > something we should espouse as we are close to 4.0.0 releasing and
> > turning our focus on 4.1?
> >
> > --David
> >
> > [1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
RE: [DISCUSS] QA/Testing focus on 4.1
Posted by Sudha Ponnaganti <su...@citrix.com>.
If defect fixes include unit tests, some legacy code can be covered.
-----Original Message-----
From: Animesh Chaturvedi [mailto:animesh.chaturvedi@citrix.com]
Sent: Wednesday, November 21, 2012 6:18 PM
To: cloudstack-dev@incubator.apache.org
Subject: RE: [DISCUSS] QA/Testing focus on 4.1
From: David Nalley [david@gnsa.us]
Sent: Wednesday, November 21, 2012 5:17 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS] QA/Testing focus on 4.1
> is useful. (But of course, having a set of tests to confirm that the
> refactoring didn't break anything should be a prerequisite to making
> that sort of change!
I was thinking the same thing when reading Chiradeeps review on Min's patch.
We should start with a set of tests that pass cleanly with the current code before trying changes.
> Anyone have other ideas about how to prioritize unit test writing?
I think the idea above is good - as well as requiring it for any refactoring in general.
Animesh> For prioritizing unit test writing, all new code should have good unit test coverage. Full unit test coverage is possible if we adopt TDD. For legacy code we should guard it with unit test overtime based on how critical the code is for best ROI. Whenver we refactor or change legacy code we should try to put as much unit test as possible.
--David
RE: [DISCUSS] QA/Testing focus on 4.1
Posted by Animesh Chaturvedi <an...@citrix.com>.
From: David Nalley [david@gnsa.us]
Sent: Wednesday, November 21, 2012 5:17 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS] QA/Testing focus on 4.1
> is useful. (But of course, having a set of tests to confirm that the
> refactoring didn't break anything should be a prerequisite to making
> that sort of change!
I was thinking the same thing when reading Chiradeeps review on Min's patch.
We should start with a set of tests that pass cleanly with the current
code before trying changes.
> Anyone have other ideas about how to prioritize unit test writing?
I think the idea above is good - as well as requiring it for any
refactoring in general.
Animesh> For prioritizing unit test writing, all new code should have good unit test coverage. Full unit test coverage is possible if we adopt TDD. For legacy code we should guard it with unit test overtime based on how critical the code is for best ROI. Whenver we refactor or change legacy code we should try to put as much unit test as possible.
--David
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by David Nalley <da...@gnsa.us>.
> is useful. (But of course, having a set of tests to confirm that the
> refactoring didn't break anything should be a prerequisite to making
> that sort of change!
I was thinking the same thing when reading Chiradeeps review on Min's patch.
We should start with a set of tests that pass cleanly with the current
code before trying changes.
> Anyone have other ideas about how to prioritize unit test writing?
I think the idea above is good - as well as requiring it for any
refactoring in general.
--David
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by Chip Childers <ch...@sungard.com>.
On Wed, Nov 21, 2012 at 5:47 PM, David Nalley <da...@gnsa.us> wrote:
> Hi folks:
>
> Just a heads up on where I currently stand before the holidays in the US
> begin.
> I have master building and they run unit tests - and also cobertura is
> reporting coverage reports in that job.
>
> https://builds.apache.org/job/cloudstack-master-maven/
>
> I also have the RAT job running here:
> https://builds.apache.org/job/cloudstack-rat-master/
>
> Both of these poll git and run if there have been commits.
>
> --David
Super useful... thanks!
I've been thinking about an appropriate strategy for how to start
adding tests that have an impact, and I think the way to go is to look
at analysis.apache.org's class complexity analysis [1]. If we were to
start at the top of the class complexity chart, I think we would be
adding significant value with each test that gets written. We could
probably exclude the generated AWS code bits, as well as the code that
isn't from our project (such as the Xen Java files). Depending on the
individual class or method in question, we may find that refactoring
is useful. (But of course, having a set of tests to confirm that the
refactoring didn't break anything should be a prerequisite to making
that sort of change!
For example: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
[2] is at the top of the chart for class-complexity, and a few minutes
of looking at the source indicates that there is definitely a need to
get some unit tests wrapped around this code! It's currently at 0%
code coverage via unit tests [3], but we can say the same thing about
lots of other files...
Anyone have other ideas about how to prioritize unit test writing?
-chip
[1] https://analysis.apache.org/drilldown/measures/100206?metric=class_complexity
[2] https://analysis.apache.org/drilldown/measures/100206?metric=class_complexity&rids%5B%5D=100226&rids%5B%5D=102648#
[3] https://builds.apache.org/job/cloudstack-master-maven/13/cobertura/com_cloud_hypervisor_xen_discoverer/
RE: [DISCUSS] QA/Testing focus on 4.1
Posted by Sudha Ponnaganti <su...@citrix.com>.
Thanks for the report David. Gives good baseline for pre 4.1
Lot of scope to add unit tests
https://builds.apache.org/job/cloudstack-master-maven/13/cobertura/
-----Original Message-----
From: David Nalley [mailto:david@gnsa.us]
Sent: Wednesday, November 21, 2012 3:39 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS] QA/Testing focus on 4.1
On Wed, Nov 21, 2012 at 6:29 PM, Animesh Chaturvedi < animesh.chaturvedi@citrix.com> wrote:
> David
>
> I certainly vote +1 on your ideas to improve testing in CS and
> enhancing and embracing automation as way of life. To put in my two
> cents , I would like to highlight that unit test automation is the
> base for automation strategy. Here is a good post on test pyramid
> http://martinfowler.com/bliki/TestPyramid.html. As developers if we
> can commit to comprehensive automated unit tests we can release early
> and often. CloudStack with its API layer is already well positioned
> for service/integration layer automation.
>
>
I agree with Martin's sentiment there.
Really what we want to do is get a pipeline going. (and actually now that I write this, I realize that I need to adjust how things are evolving on
b.a.o) so that it looks like this:
* RAT
* compile + junit
* docs + apidocs
* package building
* install tests
* smoketests
* full blown integration tests
A failure at any stage should block every other future test. This also gets us from least expensive (RAT completes in around 1 minute) to most expensive (the full Marvin tests take hours) Running this frequently also allows us to quickly see where problems have been introduced because there would be a limited set of changes in each of these test iterations.
--David
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by Chip Childers <ch...@sungard.com>.
On Wed, Nov 21, 2012 at 6:38 PM, David Nalley <da...@gnsa.us> wrote:
> On Wed, Nov 21, 2012 at 6:29 PM, Animesh Chaturvedi <
> animesh.chaturvedi@citrix.com> wrote:
>
>> David
>>
>> I certainly vote +1 on your ideas to improve testing in CS and enhancing
>> and embracing automation as way of life. To put in my two cents , I would
>> like to highlight that unit test automation is the base for automation
>> strategy. Here is a good post on test pyramid
>> http://martinfowler.com/bliki/TestPyramid.html. As developers if we can
>> commit to comprehensive automated unit tests we can release early and
>> often. CloudStack with its API layer is already well positioned for
>> service/integration layer automation.
>>
>>
> I agree with Martin's sentiment there.
> Really what we want to do is get a pipeline going. (and actually now that I
> write this, I realize that I need to adjust how things are evolving on
> b.a.o) so that it looks like this:
> * RAT
> * compile + junit
> * docs + apidocs
> * package building
> * install tests
> * smoketests
> * full blown integration tests
>
> A failure at any stage should block every other future test. This also gets
> us from least expensive (RAT completes in around 1 minute) to most
> expensive (the full Marvin tests take hours) Running this frequently also
> allows us to quickly see where problems have been introduced because there
> would be a limited set of changes in each of these test iterations.
>
> --David
+1 to that order
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by David Nalley <da...@gnsa.us>.
On Wed, Nov 21, 2012 at 6:29 PM, Animesh Chaturvedi <
animesh.chaturvedi@citrix.com> wrote:
> David
>
> I certainly vote +1 on your ideas to improve testing in CS and enhancing
> and embracing automation as way of life. To put in my two cents , I would
> like to highlight that unit test automation is the base for automation
> strategy. Here is a good post on test pyramid
> http://martinfowler.com/bliki/TestPyramid.html. As developers if we can
> commit to comprehensive automated unit tests we can release early and
> often. CloudStack with its API layer is already well positioned for
> service/integration layer automation.
>
>
I agree with Martin's sentiment there.
Really what we want to do is get a pipeline going. (and actually now that I
write this, I realize that I need to adjust how things are evolving on
b.a.o) so that it looks like this:
* RAT
* compile + junit
* docs + apidocs
* package building
* install tests
* smoketests
* full blown integration tests
A failure at any stage should block every other future test. This also gets
us from least expensive (RAT completes in around 1 minute) to most
expensive (the full Marvin tests take hours) Running this frequently also
allows us to quickly see where problems have been introduced because there
would be a limited set of changes in each of these test iterations.
--David
RE: [DISCUSS] QA/Testing focus on 4.1
Posted by Animesh Chaturvedi <an...@citrix.com>.
David
I certainly vote +1 on your ideas to improve testing in CS and enhancing and embracing automation as way of life. To put in my two cents , I would like to highlight that unit test automation is the base for automation strategy. Here is a good post on test pyramid http://martinfowler.com/bliki/TestPyramid.html. As developers if we can commit to comprehensive automated unit tests we can release early and often. CloudStack with its API layer is already well positioned for service/integration layer automation.
Thanks
Animesh
-----Original Message-----
From: David Nalley [mailto:david@gnsa.us]
Sent: Wednesday, November 21, 2012 2:48 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: [DISCUSS] QA/Testing focus on 4.1
Hi folks:
Just a heads up on where I currently stand before the holidays in the US begin.
I have master building and they run unit tests - and also cobertura is reporting coverage reports in that job.
https://builds.apache.org/job/cloudstack-master-maven/
I also have the RAT job running here:
https://builds.apache.org/job/cloudstack-rat-master/
Both of these poll git and run if there have been commits.
--David
On Sat, Nov 10, 2012 at 7:00 AM, David Nalley <da...@gnsa.us> wrote:
> Hi folks - haven't followed up on this in several weeks - also haven't
> had a chance to do much - I've been on holiday and traveling to ACEU.
>
> I did have the opportunity to talk to Gavin McDonald of ASF Infra fame
> while at ACEU this past week and discuss with him some ideas for our
> testing. He did recommend we actually use builders local to ASF infra
> for things we want to happen fast (compile tests, unit tests, etc) as
> they are effectively on the same network, and thus things like
> checkouts happen much faster than connections over the WAN.
>
> I am going to start having my weekly IRC time at 1800 UTC in
> #cloudstack-dev on Wednesdays - all are welcome to join, I know it is
> inconvenient for many - and so don't hesitate to do things outside of
> that time - I am going to start working on migrating some of the
> simpler stuff to builds.a.o.
>
> --David
>
>
> On Wed, Oct 17, 2012 at 6:22 PM, David Nalley <da...@gnsa.us> wrote:
> > Warning: This is a long ponderous email. Stop now and get your cup
> > of coffee/tea/$beverage.
> >
> > Hi folks,
> >
> > I've been having some off hand conversations with folks in other
> > open source communities and rereading a number of good books of late
> > (most notably Continuous Delivery by Jez Humble) and I would like to
> > toss this idea out and see what folks think of it:
> >
> > CloudStack is large, often complex, piece of software. There are few
> > people who truly grok all of CloudStack, and because of the
> > diversity of environments it can be deployed in it can be truly
> > bewildering to test fully. That said, I noticed a number of things
> > that became last minute, blocker bugs, and it wasn't overly esoteric
> > configurations, it was code paths that we just hadn't excercised in
> > testing. One if these would have even been easily detected and fixed
> > we had been testing installation alone. Additionally, as a project
> > we are dramatically growing. The number of people adding non-trivial
> > amounts of code is growing, and that makes ad hoc QA even more difficult.
> >
> > So my proposal in short is that we focus on testing, beginning
> > immediately and erect what is effectively a continuous deployment
> > pipeline - such that our confidence in our codebase is such that we
> > could arbitrarily decide to release if we so desired. I don't think
> > that this is a one release type of project, indeed I think it's
> > really more of a culture shift than a technical project. The big
> > shift is that EVERYONE must be responsible for a quality release. To
> > that end, I'd propose the following tenets if we choose to adopt this:
> >
> > * Tests become the Andon cord[1] for the entire project. When a test
> > fails we stop - additional commits don't happen - we find out what
> > is wrong and fix it. More on this in a bit.
> >
> > * Tests (specifically automated tests) become part of our culture.
> > * New features should come replete with both unit and integration
> > tests. I am tempted to suggest a certain percentage of coverage, but
> > I worry that it is a red herring; particularly given our dismal
> > current unit test coverage.
> > * Blocker and critical bugs must have automated tests that get
> > committed as part of being qualified for closing.
> >
> > * We dedicate a non-trivial portion of our energy to enhancing not
> > just the quantity and quality of our tests, but also on making it
> > highly automated, and capable of delivering fast feedback. Ideally
> > we would know within minutes if a commit broke unit tests, within
> > hours if a commit failed in integration testing.
> >
> > I also know that this isn't a new idea. Lots of people have been
> > focused on automated testing as part of our ongoing development. The
> > only difference here is that I am actively asking you not to solely
> > depend on those folks to do the work for you, but to make testing a
> > part of the problem that you have to solve here. To be clear the
> > goal isn't to be perfect and problem free with every commit - things
> > will break. (If you've followed things at all in CloudStack you'll
> > know that I've broken builds more than once)
> >
> > Pipeline I'd like to see for 4.1:
> >
> > 1. RAT test (fail this and we have IP problems) 2. Compile test
> > (does it actually compile) 3. Unit tests 4. Package building 5.
> > Automated installation (multiple platforms, does it actually install
> > from the packages) 6. Integration tests (aka Marvin running against
> > virtualized or real CS deployments)
> >
> > Clearly the above isn't an end/all be all for testing, but perhaps
> > we can get some of this going in the 4.1 timeframe. There are also
> > clearly corner cases (building marvin, building api docs, building
> > official documentation) that need to be included in the pipeline as
> > well. But the principle is that we won't move on past our failure
> > until that failure is resolved.
> >
> > Immediate Action Items:
> >
> > Whether we adopt this or not, I plan on showing up on IRC once a
> > week to work on testing in some form or another for an hour or two.
> > I will also be cajoling people to join me. I might be working on
> > infrastructure tasks. Obviously we have people scattered across the
> > globe, so it's not the only time to work on testing, but you are
> > welcome to join me.
> >
> > I am curious to hear others thoughts, comments, or flames? Is this
> > something we should espouse as we are close to 4.0.0 releasing and
> > turning our focus on 4.1?
> >
> > --David
> >
> > [1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
>
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by David Nalley <da...@gnsa.us>.
Hi folks:
Just a heads up on where I currently stand before the holidays in the US
begin.
I have master building and they run unit tests - and also cobertura is
reporting coverage reports in that job.
https://builds.apache.org/job/cloudstack-master-maven/
I also have the RAT job running here:
https://builds.apache.org/job/cloudstack-rat-master/
Both of these poll git and run if there have been commits.
--David
On Sat, Nov 10, 2012 at 7:00 AM, David Nalley <da...@gnsa.us> wrote:
> Hi folks - haven't followed up on this in several weeks - also haven't
> had a chance to do much - I've been on holiday and traveling to ACEU.
>
> I did have the opportunity to talk to Gavin McDonald of ASF Infra fame
> while at ACEU this past week and discuss with him some ideas for our
> testing. He did recommend we actually use builders local to ASF infra
> for things we want to happen fast (compile tests, unit tests, etc) as
> they are effectively on the same network, and thus things like
> checkouts happen much faster than connections over the WAN.
>
> I am going to start having my weekly IRC time at 1800 UTC in
> #cloudstack-dev on Wednesdays - all are welcome to join, I know it is
> inconvenient for many - and so don't hesitate to do things outside of
> that time - I am going to start working on migrating some of the
> simpler stuff to builds.a.o.
>
> --David
>
>
> On Wed, Oct 17, 2012 at 6:22 PM, David Nalley <da...@gnsa.us> wrote:
> > Warning: This is a long ponderous email. Stop now and get your cup of
> > coffee/tea/$beverage.
> >
> > Hi folks,
> >
> > I've been having some off hand conversations with folks in other open
> > source communities and rereading a number of good books of late (most
> > notably Continuous Delivery by Jez Humble) and I would like to toss
> > this idea out and see what folks think of it:
> >
> > CloudStack is large, often complex, piece of software. There are few
> > people who truly grok all of CloudStack, and because of the diversity
> > of environments it can be deployed in it can be truly bewildering to
> > test fully. That said, I noticed a number of things that became last
> > minute, blocker bugs, and it wasn't overly esoteric configurations, it
> > was code paths that we just hadn't excercised in testing. One if these
> > would have even been easily detected and fixed we had been testing
> > installation alone. Additionally, as a project we are dramatically
> > growing. The number of people adding non-trivial amounts of code is
> > growing, and that makes ad hoc QA even more difficult.
> >
> > So my proposal in short is that we focus on testing, beginning
> > immediately and erect what is effectively a continuous deployment
> > pipeline - such that our confidence in our codebase is such that we
> > could arbitrarily decide to release if we so desired. I don't think
> > that this is a one release type of project, indeed I think it's really
> > more of a culture shift than a technical project. The big shift is
> > that EVERYONE must be responsible for a quality release. To that end,
> > I'd propose the following tenets if we choose to adopt this:
> >
> > * Tests become the Andon cord[1] for the entire project. When a test
> > fails we stop - additional commits don't happen - we find out what is
> > wrong and fix it. More on this in a bit.
> >
> > * Tests (specifically automated tests) become part of our culture.
> > * New features should come replete with both unit and integration
> > tests. I am tempted to suggest a certain percentage of coverage, but I
> > worry that it is a red herring; particularly given our dismal current
> > unit test coverage.
> > * Blocker and critical bugs must have automated tests that get
> > committed as part of being qualified for closing.
> >
> > * We dedicate a non-trivial portion of our energy to enhancing not
> > just the quantity and quality of our tests, but also on making it
> > highly automated, and capable of delivering fast feedback. Ideally we
> > would know within minutes if a commit broke unit tests, within hours
> > if a commit failed in integration testing.
> >
> > I also know that this isn't a new idea. Lots of people have been
> > focused on automated testing as part of our ongoing development. The
> > only difference here is that I am actively asking you not to solely
> > depend on those folks to do the work for you, but to make testing a
> > part of the problem that you have to solve here. To be clear the goal
> > isn't to be perfect and problem free with every commit - things will
> > break. (If you've followed things at all in CloudStack you'll know
> > that I've broken builds more than once)
> >
> > Pipeline I'd like to see for 4.1:
> >
> > 1. RAT test (fail this and we have IP problems)
> > 2. Compile test (does it actually compile)
> > 3. Unit tests
> > 4. Package building
> > 5. Automated installation (multiple platforms, does it actually
> > install from the packages)
> > 6. Integration tests (aka Marvin running against virtualized or real
> > CS deployments)
> >
> > Clearly the above isn't an end/all be all for testing, but perhaps we
> > can get some of this going in the 4.1 timeframe. There are also
> > clearly corner cases (building marvin, building api docs, building
> > official documentation) that need to be included in the pipeline as
> > well. But the principle is that we won't move on past our failure
> > until that failure is resolved.
> >
> > Immediate Action Items:
> >
> > Whether we adopt this or not, I plan on showing up on IRC once a week
> > to work on testing in some form or another for an hour or two. I will
> > also be cajoling people to join me. I might be working on
> > infrastructure tasks. Obviously we have people scattered across the
> > globe, so it's not the only time to work on testing, but you are
> > welcome to join me.
> >
> > I am curious to hear others thoughts, comments, or flames? Is this
> > something we should espouse as we are close to 4.0.0 releasing and
> > turning our focus on 4.1?
> >
> > --David
> >
> > [1] http://en.wikipedia.org/wiki/Andon_(manufacturing)
>
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by prasanna <sr...@gmail.com>.
On 10 November 2012 19:59, prasanna <ts...@apache.org> wrote:
>> I am going to start having my weekly IRC time at 1800 UTC in
>> #cloudstack-dev on Wednesdays - all are welcome to join, I know it is
>> inconvenient for many - and so don't hesitate to do things outside of
>> that time - I am going to start working on migrating some of the
>> simpler stuff to builds.a.o.
>>
>
A bit inconvenient for me. It'll be past 11 PM. But I'll try to make
it. Also I can help take care of the test jobs I setup, apidocs,
marvin builds for the migration.
--
Prasanna.,
Re: [DISCUSS] QA/Testing focus on 4.1
Posted by prasanna <ts...@apache.org>.
> I am going to start having my weekly IRC time at 1800 UTC in
> #cloudstack-dev on Wednesdays - all are welcome to join, I know it is
> inconvenient for many - and so don't hesitate to do things outside of
> that time - I am going to start working on migrating some of the
> simpler stuff to builds.a.o.
>
A bit inconvenient for me. It'll be past 11 PM. But I'll try to make
it. Also I can help take care of the test jobs I setup, apidocs,
marvin builds for the migration.