You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Alex Huang <Al...@citrix.com> on 2013/03/05 19:42:11 UTC
[PROPOSAL] BVT for CloudStack checkins
Hi All,
As most of you are aware, the master branch keeps getting broken by checkins for various reasons. Committers need to be more responsible about their checkins but I don't think we can depend on that happening. There are various reasons. The most obvious to me is that granting committership is not based on code competency. (And I don't think it should.) Given that we need to build a BVT system for ensure that checkins do not break the branch.
Here's my proposal:
Existing components that we'll use.
- Citrix has contributed its testing to Apache.
- Apache CloudStack has already a simulator that's been used for scale testing.
- Marvin
- DevCloud-kvm
Work Proposal:
- Convert the Citrix testing into three phases:
o Setup
o Test
o Verify
- Add a Setup and Verify phase for the simulator
- Add all of the agent commands necessary for the simulator to pass the testing.
- Add a Setup and Verify phase for devCloud-kvm
- Add two more profiles to pom
o Checkin-test-with-simulator: Runs the testing against the simulator
o Checkin-test-with-devCloud: Runs the testing against devcloud
- All of the profiles will attempt to also check the merge list that Chip has proposed.'
- We will also change marvin to easily add zones with actual hardware. It will be based on a data driven document to do the setup.
For a developer to checkin:
- S/he must writes marvin tests for their feature and add it to the BVT.
- S/he must run the checkin tests to verify everything works.
- If the feature contains a hardware/vpx component, simulation must be added.
At this point, everything is about the developer writing in their feature branch and merging in.
On infrastructure side:
- We'll setup continuous BVT based on the simulator.
- I again push that we must use Gerrit to test the code before it gets merge into the branch but I'll leave that for someone else to do that.
Let me know what you guys think. I'll probably break out a bvt branch to work on this. Anyone want to join me?
--Alex
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Alex Huang <Al...@citrix.com>.
That's gerrit part of the proposal. I don't have enough karma with Infra to make that work. Dave or someone else need to take that up.
--Alex
> -----Original Message-----
> From: Kelven Yang [mailto:kelven.yang@citrix.com]
> Sent: Wednesday, March 6, 2013 10:22 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [PROPOSAL] BVT for CloudStack checkins
>
> First +1 on BVT.
>
> Second, should we consider the idea of having a staging area for people to
> check-in? Which is that making master always the stable(reasonable) branch
> for main development, but whenever people make check-ins, it goes into
> staging first, and we have maintainers(could be automatic) to run whatever
> test framework we have and only perform automatic merge to master from
> staging area after a successful test-run?
>
> Kelven
>
> On 3/6/13 4:21 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>
> >Great to see you kick this off Alex! I have a few ideas for this and
> >had been looking for help as well ..
> >
> >I had a draft lying around of an email you sent me long ago about
> >tiered testing and I think your proposal fits very well on this:
> >
> >The tl;dr of that conversation was as below
> >
> >- improve simulator for runtime testability
> >- customize to model and inject failures
> >- make a habit of writing tests around bug reports (I started tagging
> >tests since api_refactoring on JIRA already.
> >look for the integration-test label)
> >- make integration testing easier using factories and DSLs (from
> > Chiradeep)
> >(part 1 of this work was started on this in the marvin-refactor
> >branch)
> >
> >More comments inline
> >
> >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> >> Hi All,
> >>
> >> As most of you are aware, the master branch keeps getting broken by
> >> checkins for various reasons. Committers need to be more responsible
> >> about their checkins but I don't think we can depend on that
> >> happening. There are various reasons. The most obvious to me is
> >> that granting committership is not based on code competency.
> >> (And I don't think it should.) Given that we need to build a BVT
> >> system for ensure that checkins do not break the branch.
> >>
> >> Here's my proposal:
> >>
> >> Existing components that we'll use.
> >>
> >> - Citrix has contributed its testing to Apache.
> >>
> >> - Apache CloudStack has already a simulator that's been used
> >>for scale testing.
> >>
> >> - Marvin
> >>
> >> - DevCloud-kvm
> >>
> >> Work Proposal:
> >>
> >> - Convert the Citrix testing into three phases:
> >>
> >> o Setup
> >>
> >> o Test
> >>
> >> o Verify
> >
> >I do the build, package, setup, test, verify in my integration test
> >pipeline and a similar pipeline for a developer combined into easily
> >runnable maven profiles/lifecycles/goals would be great to have.
> >
> >> - Add a Setup and Verify phase for the simulator
> >> - Add all of the agent commands necessary for the simulator to
> >>pass the testing.
> >> - Add a Setup and Verify phase for devCloud-kvm
> >
> >The setups exists as configs in the marvin sandbox already. Just need
> >to hook it up with mvn. For verify there is a simple python script
> >testSetupSuccess.py already that checks two things
> >
> >- system VMs are up
> >- built-in templates are downloaded
> >
> >This should be a good start IMO.
> >
> >Currently devcloud-kvm is a bit hard to run from a developer
> >environment. But it's great to have in a continuous environment backed
> >by a KVM host with a Linux 3.0 kernel. Marcus has written up some
> >pretty good documentation for this. If someone can help bring up that
> >setup I can assist in bringing up the tests using my devcloud-ci
> >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
> >memory fix.
> >
> >>
> >> - Add two more profiles to pom
> >> o Checkin-test-with-simulator: Runs the testing against the simulator
> >> o Checkin-test-with-devCloud: Runs the testing against devcloud
> >>
> >> - All of the profiles will attempt to also check the merge
> >>list that Chip has proposed.'
> >
> >> - We will also change marvin to easily add zones with
> >> actual hardware. It will be based on a data driven document to do
> >> the setup.
> >This is currently partly doable using a properties file in the sandbox
> >$ python advanced_env.py -i xen.properties -o xen.cfg
> >
> >gives out a advanced zone config for the properties specified.
> >Is data driven similar or you are talking about a DSL that Edison
> >mentioned at CCC12?
> >
> >>
> >> For a developer to checkin:
> >>
> >> - S/he must writes marvin tests for their feature and add
> >> it to the BVT.
> >I made some progress on this in my marvin-refactor branch and will
> >collect my thoughts in an FS that I am drafting. The marvin tests are
> >difficult to write in their current form. I'm following Chiradeep's
> >recommendation for providing factories and once those are baked a DSL
> >language devs can quickly mock down tests against a simulator using the
> >above mentioned mvn profiles.
> >
> >>
> >> - S/he must run the checkin tests to verify everything works.
> >> - If the feature contains a hardware/vpx component, simulation
> >>must be added.
> >>
> >> At this point, everything is about the developer writing in their
> >>feature branch and merging in.
> >>
> >> On infrastructure side:
> >>
> >> - We'll setup continuous BVT based on the simulator.
> >I brought this up on the IRC and the list y'day, so +1 - happy to help
> >>
> >> - I again push that we must use Gerrit to test the code
> >> before it gets merge into the branch but I'll leave that for someone
> >> else to do that.
> >>
> >> Let me know what you guys think. I'll probably break out a bvt
> >> branch to work on this. Anyone want to join me?
> >
> >Count me in!
> >
> >--
> >Prasanna.,
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Simon Waterhouse <Si...@eu.citrix.com>.
+1 for a staging area - a master branch that passes BVT would be a big improvement.
-----Original Message-----
From: Koushik Das [mailto:koushik.das@citrix.com]
Sent: 07 March 2013 06:05
To: cloudstack-dev@incubator.apache.org
Subject: RE: [PROPOSAL] BVT for CloudStack checkins
+1 to the idea of staging area.
Each commit should be put in the staging area and on successful build and BVT run, it would get pushed to master. The staging area should maintain a queue of all the commits and process them in a sequential manner.
If it is possible to set up all these with the existing infra that would be great.
> -----Original Message-----
> From: Kelven Yang [mailto:kelven.yang@citrix.com]
> Sent: Wednesday, March 06, 2013 11:52 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [PROPOSAL] BVT for CloudStack checkins
>
> First +1 on BVT.
>
> Second, should we consider the idea of having a staging area for
> people to check-in? Which is that making master always the
> stable(reasonable) branch for main development, but whenever people
> make check-ins, it goes into staging first, and we have
> maintainers(could be automatic) to run whatever test framework we have
> and only perform automatic merge to master from staging area after a successful test-run?
>
> Kelven
>
> On 3/6/13 4:21 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>
> >Great to see you kick this off Alex! I have a few ideas for this and
> >had been looking for help as well ..
> >
> >I had a draft lying around of an email you sent me long ago about
> >tiered testing and I think your proposal fits very well on this:
> >
> >The tl;dr of that conversation was as below
> >
> >- improve simulator for runtime testability
> >- customize to model and inject failures
> >- make a habit of writing tests around bug reports (I started tagging
> >tests since api_refactoring on JIRA already.
> >look for the integration-test label)
> >- make integration testing easier using factories and DSLs (from
> > Chiradeep)
> >(part 1 of this work was started on this in the marvin-refactor
> >branch)
> >
> >More comments inline
> >
> >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> >> Hi All,
> >>
> >> As most of you are aware, the master branch keeps getting broken by
> >> checkins for various reasons. Committers need to be more
> >> responsible about their checkins but I don't think we can depend on
> >> that happening. There are various reasons. The most obvious to me
> >> is that granting committership is not based on code competency.
> >> (And I don't think it should.) Given that we need to build a BVT
> >> system for ensure that checkins do not break the branch.
> >>
> >> Here's my proposal:
> >>
> >> Existing components that we'll use.
> >>
> >> - Citrix has contributed its testing to Apache.
> >>
> >> - Apache CloudStack has already a simulator that's been used
> >>for scale testing.
> >>
> >> - Marvin
> >>
> >> - DevCloud-kvm
> >>
> >> Work Proposal:
> >>
> >> - Convert the Citrix testing into three phases:
> >>
> >> o Setup
> >>
> >> o Test
> >>
> >> o Verify
> >
> >I do the build, package, setup, test, verify in my integration test
> >pipeline and a similar pipeline for a developer combined into easily
> >runnable maven profiles/lifecycles/goals would be great to have.
> >
> >> - Add a Setup and Verify phase for the simulator
> >> - Add all of the agent commands necessary for the simulator to
> >>pass the testing.
> >> - Add a Setup and Verify phase for devCloud-kvm
> >
> >The setups exists as configs in the marvin sandbox already. Just need
> >to hook it up with mvn. For verify there is a simple python script
> >testSetupSuccess.py already that checks two things
> >
> >- system VMs are up
> >- built-in templates are downloaded
> >
> >This should be a good start IMO.
> >
> >Currently devcloud-kvm is a bit hard to run from a developer
> >environment. But it's great to have in a continuous environment
> >backed by a KVM host with a Linux 3.0 kernel. Marcus has written up
> >some pretty good documentation for this. If someone can help bring up
> >that setup I can assist in bringing up the tests using my devcloud-ci
> >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
> >memory fix.
> >
> >>
> >> - Add two more profiles to pom
> >> o Checkin-test-with-simulator: Runs the testing against the simulator
> >> o Checkin-test-with-devCloud: Runs the testing against devcloud
> >>
> >> - All of the profiles will attempt to also check the merge
> >>list that Chip has proposed.'
> >
> >> - We will also change marvin to easily add zones with
> >> actual hardware. It will be based on a data driven document to do
> >> the setup.
> >This is currently partly doable using a properties file in the
> >sandbox $ python advanced_env.py -i xen.properties -o xen.cfg
> >
> >gives out a advanced zone config for the properties specified.
> >Is data driven similar or you are talking about a DSL that Edison
> >mentioned at CCC12?
> >
> >>
> >> For a developer to checkin:
> >>
> >> - S/he must writes marvin tests for their feature and add
> >> it to the BVT.
> >I made some progress on this in my marvin-refactor branch and will
> >collect my thoughts in an FS that I am drafting. The marvin tests are
> >difficult to write in their current form. I'm following Chiradeep's
> >recommendation for providing factories and once those are baked a DSL
> >language devs can quickly mock down tests against a simulator using
> >the above mentioned mvn profiles.
> >
> >>
> >> - S/he must run the checkin tests to verify everything works.
> >> - If the feature contains a hardware/vpx component, simulation
> >>must be added.
> >>
> >> At this point, everything is about the developer writing in their
> >>feature branch and merging in.
> >>
> >> On infrastructure side:
> >>
> >> - We'll setup continuous BVT based on the simulator.
> >I brought this up on the IRC and the list y'day, so +1 - happy to
> >help
> >>
> >> - I again push that we must use Gerrit to test the code
> >> before it gets merge into the branch but I'll leave that for
> >> someone else to do that.
> >>
> >> Let me know what you guys think. I'll probably break out a bvt
> >> branch to work on this. Anyone want to join me?
> >
> >Count me in!
> >
> >--
> >Prasanna.,
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Koushik Das <ko...@citrix.com>.
+1 to the idea of staging area.
Each commit should be put in the staging area and on successful build and BVT run, it would get pushed to master. The staging area should maintain a queue of all the commits and process them in a sequential manner.
If it is possible to set up all these with the existing infra that would be great.
> -----Original Message-----
> From: Kelven Yang [mailto:kelven.yang@citrix.com]
> Sent: Wednesday, March 06, 2013 11:52 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [PROPOSAL] BVT for CloudStack checkins
>
> First +1 on BVT.
>
> Second, should we consider the idea of having a staging area for people to
> check-in? Which is that making master always the stable(reasonable) branch
> for main development, but whenever people make check-ins, it goes into
> staging first, and we have maintainers(could be automatic) to run whatever
> test framework we have and only perform automatic merge to master from
> staging area after a successful test-run?
>
> Kelven
>
> On 3/6/13 4:21 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>
> >Great to see you kick this off Alex! I have a few ideas for this and
> >had been looking for help as well ..
> >
> >I had a draft lying around of an email you sent me long ago about
> >tiered testing and I think your proposal fits very well on this:
> >
> >The tl;dr of that conversation was as below
> >
> >- improve simulator for runtime testability
> >- customize to model and inject failures
> >- make a habit of writing tests around bug reports (I started tagging
> >tests since api_refactoring on JIRA already.
> >look for the integration-test label)
> >- make integration testing easier using factories and DSLs (from
> > Chiradeep)
> >(part 1 of this work was started on this in the marvin-refactor
> >branch)
> >
> >More comments inline
> >
> >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> >> Hi All,
> >>
> >> As most of you are aware, the master branch keeps getting broken by
> >> checkins for various reasons. Committers need to be more responsible
> >> about their checkins but I don't think we can depend on that
> >> happening. There are various reasons. The most obvious to me is
> >> that granting committership is not based on code competency.
> >> (And I don't think it should.) Given that we need to build a BVT
> >> system for ensure that checkins do not break the branch.
> >>
> >> Here's my proposal:
> >>
> >> Existing components that we'll use.
> >>
> >> - Citrix has contributed its testing to Apache.
> >>
> >> - Apache CloudStack has already a simulator that's been used
> >>for scale testing.
> >>
> >> - Marvin
> >>
> >> - DevCloud-kvm
> >>
> >> Work Proposal:
> >>
> >> - Convert the Citrix testing into three phases:
> >>
> >> o Setup
> >>
> >> o Test
> >>
> >> o Verify
> >
> >I do the build, package, setup, test, verify in my integration test
> >pipeline and a similar pipeline for a developer combined into easily
> >runnable maven profiles/lifecycles/goals would be great to have.
> >
> >> - Add a Setup and Verify phase for the simulator
> >> - Add all of the agent commands necessary for the simulator to
> >>pass the testing.
> >> - Add a Setup and Verify phase for devCloud-kvm
> >
> >The setups exists as configs in the marvin sandbox already. Just need
> >to hook it up with mvn. For verify there is a simple python script
> >testSetupSuccess.py already that checks two things
> >
> >- system VMs are up
> >- built-in templates are downloaded
> >
> >This should be a good start IMO.
> >
> >Currently devcloud-kvm is a bit hard to run from a developer
> >environment. But it's great to have in a continuous environment backed
> >by a KVM host with a Linux 3.0 kernel. Marcus has written up some
> >pretty good documentation for this. If someone can help bring up that
> >setup I can assist in bringing up the tests using my devcloud-ci
> >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
> >memory fix.
> >
> >>
> >> - Add two more profiles to pom
> >> o Checkin-test-with-simulator: Runs the testing against the simulator
> >> o Checkin-test-with-devCloud: Runs the testing against devcloud
> >>
> >> - All of the profiles will attempt to also check the merge
> >>list that Chip has proposed.'
> >
> >> - We will also change marvin to easily add zones with
> >> actual hardware. It will be based on a data driven document to do
> >> the setup.
> >This is currently partly doable using a properties file in the sandbox
> >$ python advanced_env.py -i xen.properties -o xen.cfg
> >
> >gives out a advanced zone config for the properties specified.
> >Is data driven similar or you are talking about a DSL that Edison
> >mentioned at CCC12?
> >
> >>
> >> For a developer to checkin:
> >>
> >> - S/he must writes marvin tests for their feature and add
> >> it to the BVT.
> >I made some progress on this in my marvin-refactor branch and will
> >collect my thoughts in an FS that I am drafting. The marvin tests are
> >difficult to write in their current form. I'm following Chiradeep's
> >recommendation for providing factories and once those are baked a DSL
> >language devs can quickly mock down tests against a simulator using the
> >above mentioned mvn profiles.
> >
> >>
> >> - S/he must run the checkin tests to verify everything works.
> >> - If the feature contains a hardware/vpx component, simulation
> >>must be added.
> >>
> >> At this point, everything is about the developer writing in their
> >>feature branch and merging in.
> >>
> >> On infrastructure side:
> >>
> >> - We'll setup continuous BVT based on the simulator.
> >I brought this up on the IRC and the list y'day, so +1 - happy to help
> >>
> >> - I again push that we must use Gerrit to test the code
> >> before it gets merge into the branch but I'll leave that for someone
> >> else to do that.
> >>
> >> Let me know what you guys think. I'll probably break out a bvt
> >> branch to work on this. Anyone want to join me?
> >
> >Count me in!
> >
> >--
> >Prasanna.,
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Frank Zhang <Fr...@citrix.com>.
> -----Original Message-----
> From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of
> Prasanna Santhanam
> Sent: Thursday, March 07, 2013 2:53 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [PROPOSAL] BVT for CloudStack checkins
>
> On Thu, Mar 07, 2013 at 01:19:19AM +0530, Frank Zhang wrote:
> > > - improve simulator for runtime testability
> > > - customize to model and inject failures
> > > - make a habit of writing tests around bug reports (I started
> > > tagging tests since api_refactoring on JIRA already.
> > > look for the integration-test label)
> > > - make integration testing easier using factories and DSLs (from
> > > Chiradeep)
> >
> > Sorry for long thread again. All my opinion is "don't distract from
> > tools, improving Marvin should be good enough". Below are points that
> > you can simply skip.
>
> Frank, no need to apologize, these are useful insights. My original proposal
> was only to refactor the marvin library as you suggest. It still is to provide a
> good library that involves less typing in python.
>
>
> > 2. capability
> > DSL varying from common language usually lacks loop, condition
> > statement, and sometimes variable. But these things are very important
> > to our test. if you strip these abilities from your DSL, I believe you
> > will finally add them back that ends up with writing a common
> > language, then you have been distracted much from writing test case
> >
> DSL is a good to have for non-programmers - the target audience could be
> those who are not comfortable writing python code and qa engineers who
> aren't experienced writing code. Agree that it will have its limitations. The dsl
> is only for quickly writing a few simple tests, it's not to replace the actual
> integrated test writing using python. I won't be doing that as it will convolute
> things.
>
>
> > 3. it's not our business now
> > The urgent need for CloudStack now is a set of reliable test cases,
> > which I think could be done by improving current Marvin. Building our
> > DSL distracts us at this point.
>
> If not for writing tests I"m sure it can be used for describing deployments for
> marvin. I'll figure out a compromise and your inputs on this is most welcome!
Cool. We used to have that but have been obsoleted.
This is most valuable thing I see for CloudStack, marvin should use XML to deploy
setup. This may be the most widely spread DSL that is comfortable for even
non-programmer
>
> > 4. you don't even need to search a test framework for this time being
> > When I studying DSL, I ever looked into some famous test framework.
> > For Behavior Driven Test I looked Cucumber(Ruby), for Model Driven
> > Test I looked fpmt(the name maybe wrong, written by a French). They
> > both use DSL. My conclusion is the benefit of DSL is for stuff that
> > has nothing to do with test logic itself. For example, driving test
> > cases, producing document/report, scheduling random test case
> > combination ...
>
> I did read enough dsl-bashing on the interwebs when I explored about it
> (terms like cargo cults from the ruby era come to mind) but it might be worth
> a shot. We won't know until we try.
>
> > they are nice to have, but not urgent. So for our purpose, I would
> > suggest using python nose which is a unit test case framework but can
> > still use for integration test. simple, quick, and easy.
>
> The nose support for marvin has existed for sometime and you can use many
> of nose's plugins today including - pdb, attributes, multiprocess etc.
>
> --
> Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Prasanna Santhanam <ts...@apache.org>.
On Thu, Mar 07, 2013 at 01:19:19AM +0530, Frank Zhang wrote:
> > - improve simulator for runtime testability
> > - customize to model and inject failures
> > - make a habit of writing tests around bug reports (I started tagging tests
> > since api_refactoring on JIRA already.
> > look for the integration-test label)
> > - make integration testing easier using factories and DSLs (from
> > Chiradeep)
>
> Sorry for long thread again. All my opinion is "don't distract from
> tools, improving
> Marvin should be good enough". Below are points that you can simply skip.
Frank, no need to apologize, these are useful insights. My original
proposal was only to refactor the marvin library as you suggest. It
still is to provide a good library that involves less typing in
python.
> 2. capability
> DSL varying from common language usually lacks loop, condition
> statement, and sometimes variable. But
> these things are very important to our test. if you strip these
> abilities from your DSL, I believe you will finally
> add them back that ends up with writing a common language, then you
> have been distracted much from writing
> test case
>
DSL is a good to have for non-programmers - the target audience could
be those who are not comfortable writing python code and qa engineers
who aren't experienced writing code. Agree that it will have its
limitations. The dsl is only for quickly writing a few simple tests,
it's not to replace the actual integrated test writing using python. I
won't be doing that as it will convolute things.
> 3. it's not our business now
> The urgent need for CloudStack now is a set of reliable test cases,
> which I think could be done by improving current
> Marvin. Building our DSL distracts us at this point.
If not for writing tests I"m sure it can be used for describing
deployments for marvin. I'll figure out a compromise and your inputs
on this is most welcome!
> 4. you don't even need to search a test framework for this time being
> When I studying DSL, I ever looked into some famous test framework.
> For Behavior Driven Test I looked Cucumber(Ruby),
> for Model Driven Test I looked fpmt(the name maybe wrong, written by
> a French). They both use DSL. My conclusion is the
> benefit of DSL is for stuff that has nothing to do with test logic
> itself. For example, driving test cases, producing document/report,
> scheduling random test case combination ...
I did read enough dsl-bashing on the interwebs when I explored about
it (terms like cargo cults from the ruby era come to mind) but
it might be worth a shot. We won't know until we try.
> they are nice to have, but not urgent. So for our purpose, I would
> suggest using python nose which is a unit test case framework but
> can still use for integration test. simple, quick, and easy.
The nose support for marvin has existed for sometime and you can use
many of nose's plugins today including - pdb, attributes, multiprocess
etc.
--
Prasanna.,
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Frank Zhang <Fr...@citrix.com>.
> - improve simulator for runtime testability
> - customize to model and inject failures
> - make a habit of writing tests around bug reports (I started tagging tests
> since api_refactoring on JIRA already.
> look for the integration-test label)
> - make integration testing easier using factories and DSLs (from
> Chiradeep)
Sorry for long thread again. All my opinion is "don't distract from tools, improving
Marvin should be good enough". Below are points that you can simply skip.
I used to create a DSL after studying "Domain Specific Language" (by Martin Fowler), using Python.
>From my experience, I feel it's not suitable for CloudStack integration test.
Here are points:
1. Syntax puzzle
The DSL I created is known as "internal DSL" which builds on syntax of host language,
"internal DSL" saves your time to write parser but limited by host language's ability. For example:
Target(
name='deploy',
actions = [
Action(id='zone', name='createZone', args=IN(name='testZone')),
Action(id='pod', name='createPod', args=IN(name='testPod', zoneUuid=O('zone').uuid)),
],
depends = ['deploydb'],
)
RunTarget('deploy')
This is an example my DSL applying to CloudStack test, you see a few syntaxes which are not intuitive.
Actually with a well-defined library the case could be as simple as:
import actions
import nose
@nose.time(600)
def test_deploy():
zone = actions.createZone(name='testZone')
pod = actions.createPod(name='testPod', zoneUuid=zone.uuid)
test_deploy()
which is very pythonic every Python programmer can easily understand.
Maybe I am an inexperience DSL writer, but syntax puzzle is common problem for DSL, I really think we gain little
by introducing such thing into our test framework
2. capability
DSL varying from common language usually lacks loop, condition statement, and sometimes variable. But
these things are very important to our test. if you strip these abilities from your DSL, I believe you will finally
add them back that ends up with writing a common language, then you have been distracted much from writing
test case
3. it's not our business now
The urgent need for CloudStack now is a set of reliable test cases, which I think could be done by improving current
Marvin. Building our DSL distracts us at this point.
4. you don't even need to search a test framework for this time being
When I studying DSL, I ever looked into some famous test framework. For Behavior Driven Test I looked Cucumber(Ruby),
for Model Driven Test I looked fpmt(the name maybe wrong, written by a French). They both use DSL. My conclusion is the
benefit of DSL is for stuff that has nothing to do with test logic itself. For example, driving test cases, producing document/report,
scheduling random test case combination ...
they are nice to have, but not urgent. So for our purpose, I would suggest using python nose which is a unit test case framework but
can still use for integration test. simple, quick, and easy.
> (part 1 of this work was started on this in the marvin-refactor
> branch)
>
> More comments inline
>
> On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> > Hi All,
> >
> > As most of you are aware, the master branch keeps getting broken by
> > checkins for various reasons. Committers need to be more responsible
> > about their checkins but I don't think we can depend on that
> > happening. There are various reasons. The most obvious to me is that
> > granting committership is not based on code competency.
> > (And I don't think it should.) Given that we need to build a BVT
> > system for ensure that checkins do not break the branch.
> >
> > Here's my proposal:
> >
> > Existing components that we'll use.
> >
> > - Citrix has contributed its testing to Apache.
> >
> > - Apache CloudStack has already a simulator that's been used for scale
> testing.
> >
> > - Marvin
> >
> > - DevCloud-kvm
> >
> > Work Proposal:
> >
> > - Convert the Citrix testing into three phases:
> >
> > o Setup
> >
> > o Test
> >
> > o Verify
>
> I do the build, package, setup, test, verify in my integration test pipeline and
> a similar pipeline for a developer combined into easily runnable maven
> profiles/lifecycles/goals would be great to have.
>
> > - Add a Setup and Verify phase for the simulator
> > - Add all of the agent commands necessary for the simulator to pass
> the testing.
> > - Add a Setup and Verify phase for devCloud-kvm
>
> The setups exists as configs in the marvin sandbox already. Just need to hook
> it up with mvn. For verify there is a simple python script testSetupSuccess.py
> already that checks two things
>
> - system VMs are up
> - built-in templates are downloaded
>
> This should be a good start IMO.
>
> Currently devcloud-kvm is a bit hard to run from a developer environment.
> But it's great to have in a continuous environment backed by a KVM host
> with a Linux 3.0 kernel. Marcus has written up some pretty good
> documentation for this. If someone can help bring up that setup I can assist
> in bringing up the tests using my devcloud-ci scripts. I'm bringing up devcloud
> after Kelven 'alerted' us of the memory fix.
>
> >
> > - Add two more profiles to pom
> > o Checkin-test-with-simulator: Runs the testing against the simulator
> > o Checkin-test-with-devCloud: Runs the testing against devcloud
> >
> > - All of the profiles will attempt to also check the merge list that Chip
> has proposed.'
>
> > - We will also change marvin to easily add zones with
> > actual hardware. It will be based on a data driven document to do the
> > setup.
> This is currently partly doable using a properties file in the sandbox $ python
> advanced_env.py -i xen.properties -o xen.cfg
>
> gives out a advanced zone config for the properties specified.
> Is data driven similar or you are talking about a DSL that Edison mentioned at
> CCC12?
>
> >
> > For a developer to checkin:
> >
> > - S/he must writes marvin tests for their feature and add
> > it to the BVT.
> I made some progress on this in my marvin-refactor branch and will collect
> my thoughts in an FS that I am drafting. The marvin tests are difficult to write
> in their current form. I'm following Chiradeep's recommendation for
> providing factories and once those are baked a DSL language devs can quickly
> mock down tests against a simulator using the above mentioned mvn profiles.
>
> >
> > - S/he must run the checkin tests to verify everything works.
> > - If the feature contains a hardware/vpx component, simulation must
> be added.
> >
> > At this point, everything is about the developer writing in their feature
> branch and merging in.
> >
> > On infrastructure side:
> >
> > - We'll setup continuous BVT based on the simulator.
> I brought this up on the IRC and the list y'day, so +1 - happy to help
> >
> > - I again push that we must use Gerrit to test the code
> > before it gets merge into the branch but I'll leave that for someone
> > else to do that.
> >
> > Let me know what you guys think. I'll probably break out a bvt branch
> > to work on this. Anyone want to join me?
>
> Count me in!
>
> --
> Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Thu, Mar 7, 2013 at 12:11 AM, Murali Reddy <Mu...@citrix.com> wrote:
> On 06/03/13 11:52 PM, "Kelven Yang" <ke...@citrix.com> wrote:
>
>>First +1 on BVT.
>>
>>Second, should we consider the idea of having a staging area for people to
>>check-in? Which is that making master always the stable(reasonable) branch
>>for main development, but whenever people make check-ins, it goes into
>>staging first,
>
> +1 for having staging-branch. Unless we are able to run, Bvt's on the
> feature branch, there is no confidence that merge of feature branch to
> master won't break master. IMO, its not un-common to have a master branch
> which is kept stable with regular builds and BVT test passes, then have a
> development branch where the features land up first go through the BVT
> then pushed to master. On regular basic there can be 'reverse integration'
> (dev branch to master) and 'forward integrations' (master to dev)
> automated.
>
I don't see why we can't run BVT on a feature branch. Should be
relatively trivial to setup. Especially for branches expected to live
more than a few days, I'd prefer to see testing run on them regularly.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Murali Reddy <Mu...@citrix.com>.
Also with out staging, features developed by non-commiters else where,
even with good code review in place there is always possibility of
regression after feature merge into master.
On 07/03/13 10:41 AM, "Murali Reddy" <Mu...@citrix.com> wrote:
>On 06/03/13 11:52 PM, "Kelven Yang" <ke...@citrix.com> wrote:
>
>>First +1 on BVT.
>>
>>Second, should we consider the idea of having a staging area for people
>>to
>>check-in? Which is that making master always the stable(reasonable)
>>branch
>>for main development, but whenever people make check-ins, it goes into
>>staging first,
>
>+1 for having staging-branch. Unless we are able to run, Bvt's on the
>feature branch, there is no confidence that merge of feature branch to
>master won't break master. IMO, its not un-common to have a master branch
>which is kept stable with regular builds and BVT test passes, then have a
>development branch where the features land up first go through the BVT
>then pushed to master. On regular basic there can be 'reverse integration'
>(dev branch to master) and 'forward integrations' (master to dev)
>automated.
>
>>and we have maintainers(could be automatic) to run whatever
>>test framework we have and only perform automatic merge to master from
>>staging area after a successful test-run?
>
>
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Murali Reddy <Mu...@citrix.com>.
On 06/03/13 11:52 PM, "Kelven Yang" <ke...@citrix.com> wrote:
>First +1 on BVT.
>
>Second, should we consider the idea of having a staging area for people to
>check-in? Which is that making master always the stable(reasonable) branch
>for main development, but whenever people make check-ins, it goes into
>staging first,
+1 for having staging-branch. Unless we are able to run, Bvt's on the
feature branch, there is no confidence that merge of feature branch to
master won't break master. IMO, its not un-common to have a master branch
which is kept stable with regular builds and BVT test passes, then have a
development branch where the features land up first go through the BVT
then pushed to master. On regular basic there can be 'reverse integration'
(dev branch to master) and 'forward integrations' (master to dev)
automated.
>and we have maintainers(could be automatic) to run whatever
>test framework we have and only perform automatic merge to master from
>staging area after a successful test-run?
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Kelven Yang <ke...@citrix.com>.
First +1 on BVT.
Second, should we consider the idea of having a staging area for people to
check-in? Which is that making master always the stable(reasonable) branch
for main development, but whenever people make check-ins, it goes into
staging first, and we have maintainers(could be automatic) to run whatever
test framework we have and only perform automatic merge to master from
staging area after a successful test-run?
Kelven
On 3/6/13 4:21 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>Great to see you kick this off Alex! I have a few ideas for this and
>had been looking for help as well ..
>
>I had a draft lying around of an email you sent me long ago about
>tiered testing and I think your proposal fits very well on this:
>
>The tl;dr of that conversation was as below
>
>- improve simulator for runtime testability
>- customize to model and inject failures
>- make a habit of writing tests around bug reports
>(I started tagging tests since api_refactoring on JIRA already.
>look for the integration-test label)
>- make integration testing easier using factories and DSLs (from
> Chiradeep)
>(part 1 of this work was started on this in the marvin-refactor
>branch)
>
>More comments inline
>
>On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
>> Hi All,
>>
>> As most of you are aware, the master branch keeps getting broken by
>> checkins for various reasons. Committers need to be more
>> responsible about their checkins but I don't think we can depend on
>> that happening. There are various reasons. The most obvious to me
>> is that granting committership is not based on code competency.
>> (And I don't think it should.) Given that we need to build a BVT
>> system for ensure that checkins do not break the branch.
>>
>> Here's my proposal:
>>
>> Existing components that we'll use.
>>
>> - Citrix has contributed its testing to Apache.
>>
>> - Apache CloudStack has already a simulator that's been used
>>for scale testing.
>>
>> - Marvin
>>
>> - DevCloud-kvm
>>
>> Work Proposal:
>>
>> - Convert the Citrix testing into three phases:
>>
>> o Setup
>>
>> o Test
>>
>> o Verify
>
>I do the build, package, setup, test, verify in my integration test
>pipeline and a similar pipeline for a developer combined into easily
>runnable maven profiles/lifecycles/goals would be great to have.
>
>> - Add a Setup and Verify phase for the simulator
>> - Add all of the agent commands necessary for the simulator to
>>pass the testing.
>> - Add a Setup and Verify phase for devCloud-kvm
>
>The setups exists as configs in the marvin sandbox already. Just need
>to hook it up with mvn. For verify there is a simple python script
>testSetupSuccess.py already that checks two things
>
>- system VMs are up
>- built-in templates are downloaded
>
>This should be a good start IMO.
>
>Currently devcloud-kvm is a bit hard to run from a developer
>environment. But it's great to have in a continuous environment backed
>by a KVM host with a Linux 3.0 kernel. Marcus has written up some
>pretty good documentation for this. If someone can help bring up that
>setup I can assist in bringing up the tests using my devcloud-ci
>scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
>memory fix.
>
>>
>> - Add two more profiles to pom
>> o Checkin-test-with-simulator: Runs the testing against the simulator
>> o Checkin-test-with-devCloud: Runs the testing against devcloud
>>
>> - All of the profiles will attempt to also check the merge
>>list that Chip has proposed.'
>
>> - We will also change marvin to easily add zones with
>> actual hardware. It will be based on a data driven document to do
>> the setup.
>This is currently partly doable using a properties file in the sandbox
>$ python advanced_env.py -i xen.properties -o xen.cfg
>
>gives out a advanced zone config for the properties specified.
>Is data driven similar or you are talking about a DSL that Edison
>mentioned at CCC12?
>
>>
>> For a developer to checkin:
>>
>> - S/he must writes marvin tests for their feature and add
>> it to the BVT.
>I made some progress on this in my marvin-refactor branch and will
>collect my thoughts in an FS that I am drafting. The marvin tests are
>difficult to write in their current form. I'm following Chiradeep's
>recommendation for providing factories and once those are baked a DSL
>language devs can quickly mock down tests against a simulator using
>the above mentioned mvn profiles.
>
>>
>> - S/he must run the checkin tests to verify everything works.
>> - If the feature contains a hardware/vpx component, simulation
>>must be added.
>>
>> At this point, everything is about the developer writing in their
>>feature branch and merging in.
>>
>> On infrastructure side:
>>
>> - We'll setup continuous BVT based on the simulator.
>I brought this up on the IRC and the list y'day, so +1 - happy to help
>>
>> - I again push that we must use Gerrit to test the code
>> before it gets merge into the branch but I'll leave that for someone
>> else to do that.
>>
>> Let me know what you guys think. I'll probably break out a bvt
>> branch to work on this. Anyone want to join me?
>
>Count me in!
>
>--
>Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Prasanna Santhanam <ts...@apache.org>.
Great to see you kick this off Alex! I have a few ideas for this and
had been looking for help as well ..
I had a draft lying around of an email you sent me long ago about
tiered testing and I think your proposal fits very well on this:
The tl;dr of that conversation was as below
- improve simulator for runtime testability
- customize to model and inject failures
- make a habit of writing tests around bug reports
(I started tagging tests since api_refactoring on JIRA already.
look for the integration-test label)
- make integration testing easier using factories and DSLs (from
Chiradeep)
(part 1 of this work was started on this in the marvin-refactor
branch)
More comments inline
On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote:
> Hi All,
>
> As most of you are aware, the master branch keeps getting broken by
> checkins for various reasons. Committers need to be more
> responsible about their checkins but I don't think we can depend on
> that happening. There are various reasons. The most obvious to me
> is that granting committership is not based on code competency.
> (And I don't think it should.) Given that we need to build a BVT
> system for ensure that checkins do not break the branch.
>
> Here's my proposal:
>
> Existing components that we'll use.
>
> - Citrix has contributed its testing to Apache.
>
> - Apache CloudStack has already a simulator that's been used for scale testing.
>
> - Marvin
>
> - DevCloud-kvm
>
> Work Proposal:
>
> - Convert the Citrix testing into three phases:
>
> o Setup
>
> o Test
>
> o Verify
I do the build, package, setup, test, verify in my integration test
pipeline and a similar pipeline for a developer combined into easily
runnable maven profiles/lifecycles/goals would be great to have.
> - Add a Setup and Verify phase for the simulator
> - Add all of the agent commands necessary for the simulator to pass the testing.
> - Add a Setup and Verify phase for devCloud-kvm
The setups exists as configs in the marvin sandbox already. Just need
to hook it up with mvn. For verify there is a simple python script
testSetupSuccess.py already that checks two things
- system VMs are up
- built-in templates are downloaded
This should be a good start IMO.
Currently devcloud-kvm is a bit hard to run from a developer
environment. But it's great to have in a continuous environment backed
by a KVM host with a Linux 3.0 kernel. Marcus has written up some
pretty good documentation for this. If someone can help bring up that
setup I can assist in bringing up the tests using my devcloud-ci
scripts. I'm bringing up devcloud after Kelven 'alerted' us of the
memory fix.
>
> - Add two more profiles to pom
> o Checkin-test-with-simulator: Runs the testing against the simulator
> o Checkin-test-with-devCloud: Runs the testing against devcloud
>
> - All of the profiles will attempt to also check the merge list that Chip has proposed.'
> - We will also change marvin to easily add zones with
> actual hardware. It will be based on a data driven document to do
> the setup.
This is currently partly doable using a properties file in the sandbox
$ python advanced_env.py -i xen.properties -o xen.cfg
gives out a advanced zone config for the properties specified.
Is data driven similar or you are talking about a DSL that Edison
mentioned at CCC12?
>
> For a developer to checkin:
>
> - S/he must writes marvin tests for their feature and add
> it to the BVT.
I made some progress on this in my marvin-refactor branch and will
collect my thoughts in an FS that I am drafting. The marvin tests are
difficult to write in their current form. I'm following Chiradeep's
recommendation for providing factories and once those are baked a DSL
language devs can quickly mock down tests against a simulator using
the above mentioned mvn profiles.
>
> - S/he must run the checkin tests to verify everything works.
> - If the feature contains a hardware/vpx component, simulation must be added.
>
> At this point, everything is about the developer writing in their feature branch and merging in.
>
> On infrastructure side:
>
> - We'll setup continuous BVT based on the simulator.
I brought this up on the IRC and the list y'day, so +1 - happy to help
>
> - I again push that we must use Gerrit to test the code
> before it gets merge into the branch but I'll leave that for someone
> else to do that.
>
> Let me know what you guys think. I'll probably break out a bvt
> branch to work on this. Anyone want to join me?
Count me in!
--
Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Tue, Mar 5, 2013 at 1:42 PM, Alex Huang <Al...@citrix.com> wrote:
> Hi All,
>
> As most of you are aware, the master branch keeps getting broken by checkins for various reasons. Committers need to be more responsible about their checkins but I don't think we can depend on that happening. There are various reasons. The most obvious to me is that granting committership is not based on code competency. (And I don't think it should.) Given that we need to build a BVT system for ensure that checkins do not break the branch.
>
I like the idea of this alot. It's not quite continuous integration,
but its closer than where we are now.
A couple of additional comments. This really isn't about code
competency. There are really three problems. The first is that people
are fallible, make mistakes, regardless of their competency. The
second is that CloudStack is a rather large codebase, with a ton of
different codepaths, many that are currently very difficult to
exercise from a developers laptop. Third, CloudStack has lots of
moving pieces, many that you may no know about.
I am ok with gating feature merges on running BVT tests or more. I do
not think that this is a replacement for unit tests, it's just
another, more intensive, layer of testing.
Tell me how I can help with this.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Ahmad Emneina <ae...@gmail.com>.
>
> For a developer to checkin:
>
> - S/he must writes marvin tests for their feature and add it to
> the BVT.
>
> - S/he must run the checkin tests to verify everything works.
>
+1 to the above suggestions. there has been a lot of breakage on master
recently, this should eliminate it.
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Fri, Mar 8, 2013 at 3:36 PM, Chip Childers <ch...@sungard.com> wrote:
> On Fri, Mar 08, 2013 at 03:18:46PM -0500, David Nalley wrote:
>> On Fri, Mar 8, 2013 at 2:30 PM, Chip Childers <ch...@sungard.com> wrote:
>> > On Fri, Mar 08, 2013 at 08:54:05AM -0800, Alex Huang wrote:
>> >> >
>> >> > I'd add:
>> >> > Large scale of our code base
>> >> > Difficulty of testing at scale without plenty of hardware (which was a problem
>> >> > stated during incubation proposal)
>> >>
>> >> I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
>> >>
>> >> Specific hardware will be different but should be a much smaller part of our code base. Also I believe specific hardware code can be broken but mostly won't affect others. Here the main concern is changes impacting the developer community at large.
>> >>
>> >
>> > I think that David is referring to the larger CI aspects of things. If
>> > we were to adopt gerrit, it would be best used (given Hugo's concerns)
>> > as a gate into master (or x branches) after successful CI tests. Hugo's
>> > concern was that he not be blocked, waiting for another timezone to wake
>> > up and review commits.
>> >
>> > IMO, our best path to success here is to have a couple of different
>> > scenarios:
>> >
>> > 1 - Contributors (non-committers) submit a patch that will be tested
>> > within a CI environment, but must also be reviewed / approved by at least one
>> > committer, before being pushed into the repo.
>> >
>> > 2 - Committers submit a patch that will be tested within a CI
>> > environment before being pushed into the repo.
>> >
>> > Optionally, committers need to be able to request that another reviewer
>> > approve the patch before it's pushed (this helps with collaboration).
>> >
>>
>> Yes - I think this is essentially a CI problem to solve.
>>
>> Let's not forget that we can already do this to a degree:
>> http://olamy.blogspot.com/2012/10/test-your-local-patch-on-remote-jenkins.html
>>
>> This allows us to use some of the test scenarios already setup for 4.1
>> and master to test proposed patches. This isn't quite gerrit where
>> it's all automated, but it's a step in the right direction.
>> Though a BVT branch that mirrors master works too (though it will make
>> commits noisy - essentially two messages for each commit) Perhaps
>> olamy's above post to test proposed patches solves a problem.
>
> Remind me again: do we have the required Jenkins plugin installed on
> either / both of builds.a.o and jenkins.cs.o?
We do on jenkins.cs.o...now. :)
No idea on builds.a.o - I am only a job admin not a jenkins admin.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Mar 08, 2013 at 03:18:46PM -0500, David Nalley wrote:
> On Fri, Mar 8, 2013 at 2:30 PM, Chip Childers <ch...@sungard.com> wrote:
> > On Fri, Mar 08, 2013 at 08:54:05AM -0800, Alex Huang wrote:
> >> >
> >> > I'd add:
> >> > Large scale of our code base
> >> > Difficulty of testing at scale without plenty of hardware (which was a problem
> >> > stated during incubation proposal)
> >>
> >> I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
> >>
> >> Specific hardware will be different but should be a much smaller part of our code base. Also I believe specific hardware code can be broken but mostly won't affect others. Here the main concern is changes impacting the developer community at large.
> >>
> >
> > I think that David is referring to the larger CI aspects of things. If
> > we were to adopt gerrit, it would be best used (given Hugo's concerns)
> > as a gate into master (or x branches) after successful CI tests. Hugo's
> > concern was that he not be blocked, waiting for another timezone to wake
> > up and review commits.
> >
> > IMO, our best path to success here is to have a couple of different
> > scenarios:
> >
> > 1 - Contributors (non-committers) submit a patch that will be tested
> > within a CI environment, but must also be reviewed / approved by at least one
> > committer, before being pushed into the repo.
> >
> > 2 - Committers submit a patch that will be tested within a CI
> > environment before being pushed into the repo.
> >
> > Optionally, committers need to be able to request that another reviewer
> > approve the patch before it's pushed (this helps with collaboration).
> >
>
> Yes - I think this is essentially a CI problem to solve.
>
> Let's not forget that we can already do this to a degree:
> http://olamy.blogspot.com/2012/10/test-your-local-patch-on-remote-jenkins.html
>
> This allows us to use some of the test scenarios already setup for 4.1
> and master to test proposed patches. This isn't quite gerrit where
> it's all automated, but it's a step in the right direction.
> Though a BVT branch that mirrors master works too (though it will make
> commits noisy - essentially two messages for each commit) Perhaps
> olamy's above post to test proposed patches solves a problem.
Remind me again: do we have the required Jenkins plugin installed on
either / both of builds.a.o and jenkins.cs.o?
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Fri, Mar 8, 2013 at 2:30 PM, Chip Childers <ch...@sungard.com> wrote:
> On Fri, Mar 08, 2013 at 08:54:05AM -0800, Alex Huang wrote:
>> >
>> > I'd add:
>> > Large scale of our code base
>> > Difficulty of testing at scale without plenty of hardware (which was a problem
>> > stated during incubation proposal)
>>
>> I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
>>
>> Specific hardware will be different but should be a much smaller part of our code base. Also I believe specific hardware code can be broken but mostly won't affect others. Here the main concern is changes impacting the developer community at large.
>>
>
> I think that David is referring to the larger CI aspects of things. If
> we were to adopt gerrit, it would be best used (given Hugo's concerns)
> as a gate into master (or x branches) after successful CI tests. Hugo's
> concern was that he not be blocked, waiting for another timezone to wake
> up and review commits.
>
> IMO, our best path to success here is to have a couple of different
> scenarios:
>
> 1 - Contributors (non-committers) submit a patch that will be tested
> within a CI environment, but must also be reviewed / approved by at least one
> committer, before being pushed into the repo.
>
> 2 - Committers submit a patch that will be tested within a CI
> environment before being pushed into the repo.
>
> Optionally, committers need to be able to request that another reviewer
> approve the patch before it's pushed (this helps with collaboration).
>
Yes - I think this is essentially a CI problem to solve.
Let's not forget that we can already do this to a degree:
http://olamy.blogspot.com/2012/10/test-your-local-patch-on-remote-jenkins.html
This allows us to use some of the test scenarios already setup for 4.1
and master to test proposed patches. This isn't quite gerrit where
it's all automated, but it's a step in the right direction.
Though a BVT branch that mirrors master works too (though it will make
commits noisy - essentially two messages for each commit) Perhaps
olamy's above post to test proposed patches solves a problem.
>> >
>> >
>> > > We should also be prepared to discuss how other ASF projects deal with
>> > > this, and how we are different from them (are we bigger, are we moving
>> > > faster, etc...).
>> > >
>> >
>> >
>> > Honestly the Hadoop ecosystem has some of the same problems. It might be
>> > worth talking with some of those folks.
>> >
>> > > So if we want to head down this path, which again I think we should
>> > > (and I understand some of the concerns raised by others... we need to
>> > > deal with those specific issues), then we need to bring the points
>> > > above to a conclusion.
>> > >
>> > > Who wants to drive this discussion forward?
>> >
>> > The person who drives this, particularly with infrastructure, should be the
>> > person who plans on doing the work to get this accomplished. In my mind
>> > this is a medium term effort - probably a couple of months.
>>
>> I'm perfectly fine with driving the BVT side of things but like I said, I don't think I can do the infra side of things. Someone else needs to take that over.
>>
>> --Alex
>>
>
> Well, if we want it someone needs to do it. I'm looking at hadoop (as
> David suggested), and am willing to help craft the request to infra@.
> We need one or two people to volunteer to implement / maintain any new
> tool before we propose anything to the infra team.
>
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Fri, Mar 8, 2013 at 3:53 PM, Chip Childers <ch...@sungard.com> wrote:
> On Mar 8, 2013, at 3:51 PM, David Nalley <da...@gnsa.us> wrote:
>
>> On Fri, Mar 8, 2013 at 3:47 PM, Chip Childers <ch...@sungard.com> wrote:
>>> On Fri, Mar 08, 2013 at 02:30:27PM -0500, Chip Childers wrote:
>>>> Well, if we want it someone needs to do it. I'm looking at hadoop (as
>>>> David suggested) [snip...]
>>>
>>> Take a look at Hadoop's contributor guide [1], specifically the
>>> "Contributing your work" section. Hadoop is using Jira for patch
>>> submissions, a custom jira workflow, and builds.a.o basically does the
>>> CI +1 of the patch before a committer reviews and commits it.
>>>
>>> They basically have the workflow that I proposed... The downside is
>>> that Jira attachments are a poor way to do peer reviews.
>>>
>>> Thoughts?
>>
>>
>> I thought that was supposed to be the piece that Gerrit solved.
>>
>> --David
>>
>
> I said "workflow" not tool chain. The point was that they have a
> similar process, and a different use of tools to pull it off (than
> what we are discussing).
ACK - I missed the wording.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Mar 8, 2013, at 3:51 PM, David Nalley <da...@gnsa.us> wrote:
> On Fri, Mar 8, 2013 at 3:47 PM, Chip Childers <ch...@sungard.com> wrote:
>> On Fri, Mar 08, 2013 at 02:30:27PM -0500, Chip Childers wrote:
>>> Well, if we want it someone needs to do it. I'm looking at hadoop (as
>>> David suggested) [snip...]
>>
>> Take a look at Hadoop's contributor guide [1], specifically the
>> "Contributing your work" section. Hadoop is using Jira for patch
>> submissions, a custom jira workflow, and builds.a.o basically does the
>> CI +1 of the patch before a committer reviews and commits it.
>>
>> They basically have the workflow that I proposed... The downside is
>> that Jira attachments are a poor way to do peer reviews.
>>
>> Thoughts?
>
>
> I thought that was supposed to be the piece that Gerrit solved.
>
> --David
>
I said "workflow" not tool chain. The point was that they have a
similar process, and a different use of tools to pull it off (than
what we are discussing).
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Fri, Mar 8, 2013 at 3:47 PM, Chip Childers <ch...@sungard.com> wrote:
> On Fri, Mar 08, 2013 at 02:30:27PM -0500, Chip Childers wrote:
>> Well, if we want it someone needs to do it. I'm looking at hadoop (as
>> David suggested) [snip...]
>
> Take a look at Hadoop's contributor guide [1], specifically the
> "Contributing your work" section. Hadoop is using Jira for patch
> submissions, a custom jira workflow, and builds.a.o basically does the
> CI +1 of the patch before a committer reviews and commits it.
>
> They basically have the workflow that I proposed... The downside is
> that Jira attachments are a poor way to do peer reviews.
>
> Thoughts?
I thought that was supposed to be the piece that Gerrit solved.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Mar 08, 2013 at 02:30:27PM -0500, Chip Childers wrote:
> Well, if we want it someone needs to do it. I'm looking at hadoop (as
> David suggested) [snip...]
Take a look at Hadoop's contributor guide [1], specifically the
"Contributing your work" section. Hadoop is using Jira for patch
submissions, a custom jira workflow, and builds.a.o basically does the
CI +1 of the patch before a committer reviews and commits it.
They basically have the workflow that I proposed... The downside is
that Jira attachments are a poor way to do peer reviews.
Thoughts?
[1] http://wiki.apache.org/hadoop/HowToContribute
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 3/8/13 11:30 AM, "Chip Childers" <ch...@sungard.com> wrote:
>On Fri, Mar 08, 2013 at 08:54:05AM -0800, Alex Huang wrote:
>> >
>> > I'd add:
>> > Large scale of our code base
>> > Difficulty of testing at scale without plenty of hardware (which was
>>a problem
>> > stated during incubation proposal)
>>
>> I'm going to start off a branch specifically to do BVT on simulator
>>and devCloud so that we can at least have system vms/vrs and business
>>logic tested.
>>
>> Specific hardware will be different but should be a much smaller part
>>of our code base. Also I believe specific hardware code can be broken
>>but mostly won't affect others. Here the main concern is changes
>>impacting the developer community at large.
>>
>
>I think that David is referring to the larger CI aspects of things. If
>we were to adopt gerrit, it would be best used (given Hugo's concerns)
>as a gate into master (or x branches) after successful CI tests. Hugo's
>concern was that he not be blocked, waiting for another timezone to wake
>up and review commits.
>
>IMO, our best path to success here is to have a couple of different
>scenarios:
>
>1 - Contributors (non-committers) submit a patch that will be tested
>within a CI environment, but must also be reviewed / approved by at least
>one
>committer, before being pushed into the repo.
>
>2 - Committers submit a patch that will be tested within a CI
>environment before being pushed into the repo.
>
>Optionally, committers need to be able to request that another reviewer
>approve the patch before it's pushed (this helps with collaboration).
Nice. I like it.
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Mar 08, 2013 at 08:54:05AM -0800, Alex Huang wrote:
> >
> > I'd add:
> > Large scale of our code base
> > Difficulty of testing at scale without plenty of hardware (which was a problem
> > stated during incubation proposal)
>
> I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
>
> Specific hardware will be different but should be a much smaller part of our code base. Also I believe specific hardware code can be broken but mostly won't affect others. Here the main concern is changes impacting the developer community at large.
>
I think that David is referring to the larger CI aspects of things. If
we were to adopt gerrit, it would be best used (given Hugo's concerns)
as a gate into master (or x branches) after successful CI tests. Hugo's
concern was that he not be blocked, waiting for another timezone to wake
up and review commits.
IMO, our best path to success here is to have a couple of different
scenarios:
1 - Contributors (non-committers) submit a patch that will be tested
within a CI environment, but must also be reviewed / approved by at least one
committer, before being pushed into the repo.
2 - Committers submit a patch that will be tested within a CI
environment before being pushed into the repo.
Optionally, committers need to be able to request that another reviewer
approve the patch before it's pushed (this helps with collaboration).
> >
> >
> > > We should also be prepared to discuss how other ASF projects deal with
> > > this, and how we are different from them (are we bigger, are we moving
> > > faster, etc...).
> > >
> >
> >
> > Honestly the Hadoop ecosystem has some of the same problems. It might be
> > worth talking with some of those folks.
> >
> > > So if we want to head down this path, which again I think we should
> > > (and I understand some of the concerns raised by others... we need to
> > > deal with those specific issues), then we need to bring the points
> > > above to a conclusion.
> > >
> > > Who wants to drive this discussion forward?
> >
> > The person who drives this, particularly with infrastructure, should be the
> > person who plans on doing the work to get this accomplished. In my mind
> > this is a medium term effort - probably a couple of months.
>
> I'm perfectly fine with driving the BVT side of things but like I said, I don't think I can do the infra side of things. Someone else needs to take that over.
>
> --Alex
>
Well, if we want it someone needs to do it. I'm looking at hadoop (as
David suggested), and am willing to help craft the request to infra@.
We need one or two people to volunteer to implement / maintain any new
tool before we propose anything to the infra team.
Any takers?
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Rohit Yadav <bh...@apache.org>.
Awesome work Prasanna!
CloudMonkey only needs the request section, and your changes are in
the response where you go ahead and serialize class objects which have
may objects again, recursively. I'll see the code, otherwise from the
gist you shared, the response json looks fine and won't break
cloudmonkey.
Cheers.
On Wed, Mar 27, 2013 at 10:21 PM, prasanna <ts...@apache.org> wrote:
> On 25 March 2013 18:44, Prasanna Santhanam <ts...@apache.org> wrote:
>> I'm working to include a couple more things:
>>
>> a. Allow a developer to run his/her own integration test written in
>> marvin.
>> b. new APIs to be synced automatically without marvin reinstallation
>
> In doing b. I realized I had to improve the response given out by the
> API discovery plugin. What I'm essential trying to do is allow a
> developer to sync the APIs that he/she might have introduced/modified
> and run integration-tests against those newly introduced APIs. Marvin
> will regenerate its libraries on sync and the mvn integration-test can
> be run against such a server. Since many of our APIs return embedded
> entities I've altered API discovery to include parent+child (if any)
> relationships. The new response for a sample API returned by the
> listApis command is shown at [1]
>
> [1] https://gist.github.com/vogxn/5255883
>
> I've put this change in the bvt branch and tested that cloudmonkey was
> unaffected by it.
>
> Rohit - since you wrote API discovery would you mind taking a look at
> the change?
>
> Thanks,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by prasanna <ts...@apache.org>.
On 25 March 2013 18:44, Prasanna Santhanam <ts...@apache.org> wrote:
> I'm working to include a couple more things:
>
> a. Allow a developer to run his/her own integration test written in
> marvin.
> b. new APIs to be synced automatically without marvin reinstallation
In doing b. I realized I had to improve the response given out by the
API discovery plugin. What I'm essential trying to do is allow a
developer to sync the APIs that he/she might have introduced/modified
and run integration-tests against those newly introduced APIs. Marvin
will regenerate its libraries on sync and the mvn integration-test can
be run against such a server. Since many of our APIs return embedded
entities I've altered API discovery to include parent+child (if any)
relationships. The new response for a sample API returned by the
listApis command is shown at [1]
[1] https://gist.github.com/vogxn/5255883
I've put this change in the bvt branch and tested that cloudmonkey was
unaffected by it.
Rohit - since you wrote API discovery would you mind taking a look at
the change?
Thanks,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Mar 23, 2013 at 09:01:49PM +0530, prasanna wrote:
> On 23 March 2013 17:53, prasanna <ts...@apache.org> wrote:
> > On 23 March 2013 01:49, Chip Childers <ch...@sungard.com> wrote:
> >>
> >> OK - Next errors:
> >>
> >> mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl :cloud-marvin integration-test
> >> [INFO] Scanning for projects...
> >> [INFO]
> >> [INFO] ------------------------------------------------------------------------
> >> [INFO] Building Apache CloudStack marvin 4.2.0-SNAPSHOT
> >> [INFO] ------------------------------------------------------------------------
> >> [INFO]
> >> [INFO] --- exec-maven-plugin:1.2.1:exec (generate-sources) @ cloud-marvin ---
> >> [INFO]
> >> [INFO] --- maven-remote-resources-plugin:1.3:process (default) @ cloud-marvin ---
> >> [INFO]
> >> [INFO] --- maven-site-plugin:3.1:attach-descriptor (attach-descriptor) @ cloud-marvin ---
> >> [INFO]
> >> [INFO] --- exec-maven-plugin:1.2.1:exec (pre-integration-test) @ cloud-marvin ---
> >> [INFO]
> >> [INFO] --- exec-maven-plugin:1.2.1:exec (integration-test) @ cloud-marvin ---
> >> Usage: nosetests [options]
> >>
> >> nosetests: error: no such option: --with-marvin
> >> [INFO] ------------------------------------------------------------------------
> >> [INFO] BUILD FAILURE
> >
> > Awesome! Thanks for trying Chip. I think you may have forgotten to
> > register the marvin-nose plugin with nose
> >
> > $ cd tools/marvin/marvin
> > $ pip install .
> >
> > I'll combine the two pip install steps so it's only one Marvin installation.
>
> This is done. You don't have to register marvin with nose anymore.
> It's done as part of marvin install now. I've corrected the wiki(s)
> with this information.
>
> Also a good time to enhance marvin to use ApiDiscovery like
> cloudmonkey does so that new APIs included can be tested along with
> this mvn+marvin integration.
Couple more changes:
1. Separated the simulator+marvin test into a separate spring context.
This will allow the simulator components to not intervene with the
regular oss and nonoss components. I made this change so that
simulator related components are not injected in an actual production
installation. The context is enabled by the -Dsimulator property in
the developer profile
2. I've also added an additional test apart from the deployVM test -
The test will now include some basic checks of the virtual router.
I'm working to include a couple more things:
a. Allow a developer to run his/her own integration test written in
marvin.
b. new APIs to be synced automatically without marvin reinstallation
The bvt branch is rebased with master and can be merged in the current
state. I will send out a MERGE request after the above two
improvements are done.
--
Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by prasanna <ts...@apache.org>.
On 23 March 2013 17:53, prasanna <ts...@apache.org> wrote:
> On 23 March 2013 01:49, Chip Childers <ch...@sungard.com> wrote:
>>
>> OK - Next errors:
>>
>> mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl :cloud-marvin integration-test
>> [INFO] Scanning for projects...
>> [INFO]
>> [INFO] ------------------------------------------------------------------------
>> [INFO] Building Apache CloudStack marvin 4.2.0-SNAPSHOT
>> [INFO] ------------------------------------------------------------------------
>> [INFO]
>> [INFO] --- exec-maven-plugin:1.2.1:exec (generate-sources) @ cloud-marvin ---
>> [INFO]
>> [INFO] --- maven-remote-resources-plugin:1.3:process (default) @ cloud-marvin ---
>> [INFO]
>> [INFO] --- maven-site-plugin:3.1:attach-descriptor (attach-descriptor) @ cloud-marvin ---
>> [INFO]
>> [INFO] --- exec-maven-plugin:1.2.1:exec (pre-integration-test) @ cloud-marvin ---
>> [INFO]
>> [INFO] --- exec-maven-plugin:1.2.1:exec (integration-test) @ cloud-marvin ---
>> Usage: nosetests [options]
>>
>> nosetests: error: no such option: --with-marvin
>> [INFO] ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>
> Awesome! Thanks for trying Chip. I think you may have forgotten to
> register the marvin-nose plugin with nose
>
> $ cd tools/marvin/marvin
> $ pip install .
>
> I'll combine the two pip install steps so it's only one Marvin installation.
This is done. You don't have to register marvin with nose anymore.
It's done as part of marvin install now. I've corrected the wiki(s)
with this information.
Also a good time to enhance marvin to use ApiDiscovery like
cloudmonkey does so that new APIs included can be tested along with
this mvn+marvin integration.
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by prasanna <ts...@apache.org>.
On 23 March 2013 01:49, Chip Childers <ch...@sungard.com> wrote:
>
> OK - Next errors:
>
> mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl :cloud-marvin integration-test
> [INFO] Scanning for projects...
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Apache CloudStack marvin 4.2.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:exec (generate-sources) @ cloud-marvin ---
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.3:process (default) @ cloud-marvin ---
> [INFO]
> [INFO] --- maven-site-plugin:3.1:attach-descriptor (attach-descriptor) @ cloud-marvin ---
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:exec (pre-integration-test) @ cloud-marvin ---
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:exec (integration-test) @ cloud-marvin ---
> Usage: nosetests [options]
>
> nosetests: error: no such option: --with-marvin
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
Awesome! Thanks for trying Chip. I think you may have forgotten to
register the marvin-nose plugin with nose
$ cd tools/marvin/marvin
$ pip install .
I'll combine the two pip install steps so it's only one Marvin installation.
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Mar 22, 2013 at 01:07:48PM -0700, Chiradeep Vittal wrote:
>
>
> On 3/22/13 12:49 PM, "Chip Childers" <ch...@sungard.com> wrote:
>
> >
> >
> >After running through the steps, I get this error:
> >
> > mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl
> >:cloud-marvin integration-test
> > ERROR: transport error 202: bind failed: Address already in use
> > ERROR: JDWP Transport dt_socket failed to initialize,
> >TRANSPORT_INIT(510)
> > JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports
> >initialized [../../../src/share/back/debugInit.c:741]
> > FATAL ERROR in native method: JDWP No transports initialized,
> >jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
> > [1] 28712 abort (core dumped) mvn -Pdeveloper,marvin
> >-Dmarvin.config=setup/dev/advanced.cfg -pl
> >
> >
> >I think that running the jetty server from mvn may be problematic, if
> >steps
> >afterward are also going to require maven. Is there a config to get
> >around this?
>
> It looks like two java processes are being started up, and each want to
> bind to the same debug port. Both probably have something like -
> -Xrunjdwp:transport=dt_socket,address=8787,
>
> Perhaps leave it out of MAVEN_OPTS?
>
>
Yup.. I completely forgot that I had a debug port set. Damn.
OK - Next errors:
mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl :cloud-marvin integration-test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache CloudStack marvin 4.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (generate-sources) @ cloud-marvin ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.3:process (default) @ cloud-marvin ---
[INFO]
[INFO] --- maven-site-plugin:3.1:attach-descriptor (attach-descriptor) @ cloud-marvin ---
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (pre-integration-test) @ cloud-marvin ---
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (integration-test) @ cloud-marvin ---
Usage: nosetests [options]
nosetests: error: no such option: --with-marvin
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:18.998s
[INFO] Finished at: Fri Mar 22 20:11:51 GMT 2013
[INFO] Final Memory: 23M/214M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (integration-test) on project cloud-marvin: Command execution failed. Process exited with an error: 2 (Exit value: 2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 3/22/13 12:49 PM, "Chip Childers" <ch...@sungard.com> wrote:
>
>
>After running through the steps, I get this error:
>
> mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl
>:cloud-marvin integration-test
> ERROR: transport error 202: bind failed: Address already in use
> ERROR: JDWP Transport dt_socket failed to initialize,
>TRANSPORT_INIT(510)
> JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports
>initialized [../../../src/share/back/debugInit.c:741]
> FATAL ERROR in native method: JDWP No transports initialized,
>jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
> [1] 28712 abort (core dumped) mvn -Pdeveloper,marvin
>-Dmarvin.config=setup/dev/advanced.cfg -pl
>
>
>I think that running the jetty server from mvn may be problematic, if
>steps
>afterward are also going to require maven. Is there a config to get
>around this?
It looks like two java processes are being started up, and each want to
bind to the same debug port. Both probably have something like -
-Xrunjdwp:transport=dt_socket,address=8787,
Perhaps leave it out of MAVEN_OPTS?
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Mar 22, 2013 at 08:27:44PM +0530, Prasanna Santhanam wrote:
> On Sun, Mar 10, 2013 at 11:39:03AM -0400, Prasanna Santhanam wrote:
> > On Sat, Mar 09, 2013 at 05:01:49AM -0500, prasanna wrote:
> > > On 8 March 2013 22:24, Alex Huang <Al...@citrix.com> wrote:
> > > >
> > > > I'm going to start off a branch specifically to do BVT on
> > > > simulator and devCloud so that we can at least have system vms/vrs
> > > > and business logic tested.
> > >
> > > Hey Alex - will you be writing new tests or revamping the simulator in
> > > the bvt branch? On the devcloud side of things I've kept the
> > > devcloud-ci (https://github.com/vogxn/devcloud-ci/commits/master) up
> > > to date with changes reqd for running devcloud workers in a CI
> > > envrionment. I'm looking into the devcloud-kvm infrastructure some of
> > > which I ear marked last week.
> > >
> >
> > Alex - After some hours with maven I've now removed the cyclic
> > dependency in the simulator build and it works again. I've pushed my
> > changes to the bvt branch for the further work on the BVT.
> >
> > Here are the steps to build, run and test the setup of the simulator:
> > $ mvn -Pdeveloper clean install
> > $ mvn -Pdeveloper -pl developer -Ddeploydb
> > $ mvn -Pdeveloper -pl developer -Ddeploydb-simulator
> > $ mvn -pl client jetty:run
> >
> > To deploy an adv. zone and test the server health:
> > $ mvn -Pdeveloper,marvin -Dmarvin.config=`find . -name simulator.cfg` -pl :cloud-marvin test
> >
>
> Long pending follow up:
>
> I've been integrating simulator, marvin and maven together to try and
> put together a basic test that does -
>
> 1. zone deployment
> 2. system VM health check
> 3. deploy VM in the zone
>
> This work is now on the bvt branch. Since this puts together python
> and maven which don't seem to interact in a friendly way I'd like some
> help testing in other developer environments to fix possible failures.
>
> The steps for running this integrated test are on the wiki:
> Checkout the bvt branch before proceeding.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/How+to+build+on+master+branch#Howtobuildonmasterbranch-IntegratedSimulatorMarvintest
>
Awesome!
After running through the steps, I get this error:
mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl :cloud-marvin integration-test
ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized [../../../src/share/back/debugInit.c:741]
FATAL ERROR in native method: JDWP No transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)
[1] 28712 abort (core dumped) mvn -Pdeveloper,marvin -Dmarvin.config=setup/dev/advanced.cfg -pl
I think that running the jetty server from mvn may be problematic, if steps
afterward are also going to require maven. Is there a config to get around this?
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sun, Mar 10, 2013 at 11:39:03AM -0400, Prasanna Santhanam wrote:
> On Sat, Mar 09, 2013 at 05:01:49AM -0500, prasanna wrote:
> > On 8 March 2013 22:24, Alex Huang <Al...@citrix.com> wrote:
> > >
> > > I'm going to start off a branch specifically to do BVT on
> > > simulator and devCloud so that we can at least have system vms/vrs
> > > and business logic tested.
> >
> > Hey Alex - will you be writing new tests or revamping the simulator in
> > the bvt branch? On the devcloud side of things I've kept the
> > devcloud-ci (https://github.com/vogxn/devcloud-ci/commits/master) up
> > to date with changes reqd for running devcloud workers in a CI
> > envrionment. I'm looking into the devcloud-kvm infrastructure some of
> > which I ear marked last week.
> >
>
> Alex - After some hours with maven I've now removed the cyclic
> dependency in the simulator build and it works again. I've pushed my
> changes to the bvt branch for the further work on the BVT.
>
> Here are the steps to build, run and test the setup of the simulator:
> $ mvn -Pdeveloper clean install
> $ mvn -Pdeveloper -pl developer -Ddeploydb
> $ mvn -Pdeveloper -pl developer -Ddeploydb-simulator
> $ mvn -pl client jetty:run
>
> To deploy an adv. zone and test the server health:
> $ mvn -Pdeveloper,marvin -Dmarvin.config=`find . -name simulator.cfg` -pl :cloud-marvin test
>
Long pending follow up:
I've been integrating simulator, marvin and maven together to try and
put together a basic test that does -
1. zone deployment
2. system VM health check
3. deploy VM in the zone
This work is now on the bvt branch. Since this puts together python
and maven which don't seem to interact in a friendly way I'd like some
help testing in other developer environments to fix possible failures.
The steps for running this integrated test are on the wiki:
Checkout the bvt branch before proceeding.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/How+to+build+on+master+branch#Howtobuildonmasterbranch-IntegratedSimulatorMarvintest
Thanks,
--
Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Mar 09, 2013 at 05:01:49AM -0500, prasanna wrote:
> On 8 March 2013 22:24, Alex Huang <Al...@citrix.com> wrote:
> >
> > I'm going to start off a branch specifically to do BVT on
> > simulator and devCloud so that we can at least have system vms/vrs
> > and business logic tested.
>
> Hey Alex - will you be writing new tests or revamping the simulator in
> the bvt branch? On the devcloud side of things I've kept the
> devcloud-ci (https://github.com/vogxn/devcloud-ci/commits/master) up
> to date with changes reqd for running devcloud workers in a CI
> envrionment. I'm looking into the devcloud-kvm infrastructure some of
> which I ear marked last week.
>
Alex - After some hours with maven I've now removed the cyclic
dependency in the simulator build and it works again. I've pushed my
changes to the bvt branch for the further work on the BVT.
Here are the steps to build, run and test the setup of the simulator:
$ mvn -Pdeveloper clean install
$ mvn -Pdeveloper -pl developer -Ddeploydb
$ mvn -Pdeveloper -pl developer -Ddeploydb-simulator
$ mvn -pl client jetty:run
To deploy an adv. zone and test the server health:
$ mvn -Pdeveloper,marvin -Dmarvin.config=`find . -name simulator.cfg` -pl :cloud-marvin test
I should be able to put this in to a jenkins job next two days. The
health check is failing now because systemVMs won't come up. There is
an NPE in the ImageDataStoreManagerImpl. I'm not too familiar with
this part of code. So someone will have to resolve that.
I also feel we should decouple the simulator from depending on the
data-access of cloudstack. This can put off the performance test
results that the simulator is extensively used for. Perhaps after the
bvt is up and running!
Thanks,
--
Prasanna.,
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by prasanna <ts...@apache.org>.
On 8 March 2013 22:24, Alex Huang <Al...@citrix.com> wrote:
>
> I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
Hey Alex - will you be writing new tests or revamping the simulator in
the bvt branch? On the devcloud side of things I've kept the
devcloud-ci (https://github.com/vogxn/devcloud-ci/commits/master) up
to date with changes reqd for running devcloud workers in a CI
envrionment. I'm looking into the devcloud-kvm infrastructure some of
which I ear marked last week.
>>
>> The person who drives this, particularly with infrastructure, should be the
>> person who plans on doing the work to get this accomplished. In my mind
>> this is a medium term effort - probably a couple of months.
>
> I'm perfectly fine with driving the BVT side of things but like I said, I don't think I can do the infra side of things. Someone else needs to take that over.
>
I would've offered to do this but I have my hands tied with too many
infrastructure issues at the moment:
- The test-infra in our fremont dc (Ahmad and I have been
troubleshooting a strange arpcache issue past week that prevented us
from turning on the tests. We've worked around it now with an
aggressive nmap scan). I've been in touch with some folks at infra
about this and they've more or less agreed to allow us to live on
builds.a.o. I'll follow up again on IRC.
- devcloud-basic (been testing this on jenkins) and advanced with devcloud-kvm
- I'm also going to setup a nexus proxy in front of our builds and
devcloud workers for faster build processing.
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Alex Huang <Al...@citrix.com>.
>
> I'd add:
> Large scale of our code base
> Difficulty of testing at scale without plenty of hardware (which was a problem
> stated during incubation proposal)
I'm going to start off a branch specifically to do BVT on simulator and devCloud so that we can at least have system vms/vrs and business logic tested.
Specific hardware will be different but should be a much smaller part of our code base. Also I believe specific hardware code can be broken but mostly won't affect others. Here the main concern is changes impacting the developer community at large.
>
>
> > We should also be prepared to discuss how other ASF projects deal with
> > this, and how we are different from them (are we bigger, are we moving
> > faster, etc...).
> >
>
>
> Honestly the Hadoop ecosystem has some of the same problems. It might be
> worth talking with some of those folks.
>
> > So if we want to head down this path, which again I think we should
> > (and I understand some of the concerns raised by others... we need to
> > deal with those specific issues), then we need to bring the points
> > above to a conclusion.
> >
> > Who wants to drive this discussion forward?
>
> The person who drives this, particularly with infrastructure, should be the
> person who plans on doing the work to get this accomplished. In my mind
> this is a medium term effort - probably a couple of months.
I'm perfectly fine with driving the BVT side of things but like I said, I don't think I can do the infra side of things. Someone else needs to take that over.
--Alex
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by David Nalley <da...@gnsa.us>.
On Fri, Mar 8, 2013 at 11:27 AM, Chip Childers
<ch...@sungard.com> wrote:
> On Tue, Mar 05, 2013 at 10:42:11AM -0800, Alex Huang wrote:
>> - I again push that we must use Gerrit to test the code before it gets merge into the branch but I'll leave that for someone else to do that.
>
> So the first step to getting Gerrit, is for us to agree to using it and
> to be able to clearly articulate *why*. Without being able to explain
> our issue, we'll be questioned about jumping to a tool-based solution by
> the infra team.
>
> IMO - the problems we are dealing with are (1) velocity of development
> and (2) scale of our contributor base. Agree? Other thoughts?
>
I'd add:
Large scale of our code base
Difficulty of testing at scale without plenty of hardware (which was a
problem stated during incubation proposal)
> We should also be prepared to discuss how other ASF projects deal with
> this, and how we are different from them (are we bigger, are we moving
> faster, etc...).
>
Honestly the Hadoop ecosystem has some of the same problems. It might
be worth talking with some of those folks.
> So if we want to head down this path, which again I think we should (and
> I understand some of the concerns raised by others... we need to deal
> with those specific issues), then we need to bring the points above to a
> conclusion.
>
> Who wants to drive this discussion forward?
The person who drives this, particularly with infrastructure, should
be the person who plans on doing the work to get this accomplished. In
my mind this is a medium term effort - probably a couple of months.
--David
Re: [PROPOSAL] BVT for CloudStack checkins
Posted by Chip Childers <ch...@sungard.com>.
On Tue, Mar 05, 2013 at 10:42:11AM -0800, Alex Huang wrote:
> - I again push that we must use Gerrit to test the code before it gets merge into the branch but I'll leave that for someone else to do that.
So the first step to getting Gerrit, is for us to agree to using it and
to be able to clearly articulate *why*. Without being able to explain
our issue, we'll be questioned about jumping to a tool-based solution by
the infra team.
IMO - the problems we are dealing with are (1) velocity of development
and (2) scale of our contributor base. Agree? Other thoughts?
We should also be prepared to discuss how other ASF projects deal with
this, and how we are different from them (are we bigger, are we moving
faster, etc...).
So if we want to head down this path, which again I think we should (and
I understand some of the concerns raised by others... we need to deal
with those specific issues), then we need to bring the points above to a
conclusion.
Who wants to drive this discussion forward?
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Frank Zhang <Fr...@citrix.com>.
Sorry, pervious mail is off topic of BVT.
+1 to BVT
> -----Original Message-----
> From: Frank Zhang [mailto:Frank.Zhang@citrix.com]
> Sent: Tuesday, March 05, 2013 11:41 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [PROPOSAL] BVT for CloudStack checkins
>
> >> Many types of designs/applications are better tested by automating tests
> at the module/package level.
> >>In my experience, most coding errors are not because the code in a class
> was coded incorrectly but because the coder didn't understand how their
> class was supposed to work with other classes.
>
> If you guys remembered, I have argued with you intensely last month (in a
> lunch time, not in maillist), now I am happy to see you finally find out
> somebody supporting my idea.
> I would not hesitate to say again, the unit tests you guys thinking, talking,
> doing are INPRACTICAL and valueless.
> Testing on method level, mocking everything(current CloudStack doing, the
> idea you guys like) is commonly used in textbook and library project, it's not
> suitable for integration project like CloudStack. If you force people to do so,
> the only thing you get is tons of fake test cases.
>
> Again, I would say loudly that unit test in CloudStack should be in
> COMPONENT level. By Component level I mean, the test cases actual test
> workflow of a component, not to test if a 50 line method correctly returns a
> true or throws an exception.
> You can argue with me again that what I am saying is not unit test but
> integration test. Then I would say UNIT is not only a method or a class, the
> UNIT can be a component that is up to how your project defines it.
>
> Yesterday I heard from Min that our unit test doesn't allow to deployDB data
> because faking some DB data in test IS NOT unit test. The point sounds like a
> joke to me.
>
>
>
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Tuesday, March 05, 2013 11:14 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [PROPOSAL] BVT for CloudStack checkins
> >
> > Recently, I read a topic about, "why automated test keeping
> > failing"(http://programmers.stackexchange.com/questions/185819/why-
> > does-automated-testing-keep-failing-in-my-company/185839#185839), the
> > comment is really resonated with me:
> >
> > "Most unit tests are questionable in value. Since the vast majority of
> > tests seem to be too simple."
> > "It is much harder to write good testable code than just working code."
> > "Many types of designs/applications are better tested by automating
> > tests at the module/package level. In my experience, most coding
> > errors are not because the code in a class was coded incorrectly but
> > because the coder didn't understand how their class was supposed to work
> with other classes."
> >
> > We'd better put our limited resource on BVT test on both simulator and
> > real hardware, at least, our BVT should cover all the hypervisors and
> > all the cloudstack APIs.
> >
> > So +1.
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > > Sent: Tuesday, March 05, 2013 10:42 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: [PROPOSAL] BVT for CloudStack checkins
> > >
> > > Hi All,
> > >
> > > As most of you are aware, the master branch keeps getting broken by
> > > checkins for various reasons. Committers need to be more
> > > responsible about their checkins but I don't think we can depend on that
> happening.
> > > There are various reasons. The most obvious to me is that granting
> > > committership is not based on code competency. (And I don't think
> > > it
> > > should.) Given that we need to build a BVT system for ensure that
> > > checkins do not break the branch.
> > >
> > > Here's my proposal:
> > >
> > > Existing components that we'll use.
> > >
> > > - Citrix has contributed its testing to Apache.
> > >
> > > - Apache CloudStack has already a simulator that's been used for
> scale
> > > testing.
> > >
> > > - Marvin
> > >
> > > - DevCloud-kvm
> > >
> > > Work Proposal:
> > >
> > > - Convert the Citrix testing into three phases:
> > >
> > > o Setup
> > >
> > > o Test
> > >
> > > o Verify
> > >
> > > - Add a Setup and Verify phase for the simulator
> > >
> > > - Add all of the agent commands necessary for the simulator to pass
> > the
> > > testing.
> > >
> > > - Add a Setup and Verify phase for devCloud-kvm
> > >
> > > - Add two more profiles to pom
> > >
> > > o Checkin-test-with-simulator: Runs the testing against the simulator
> > >
> > > o Checkin-test-with-devCloud: Runs the testing against devcloud
> > >
> > > - All of the profiles will attempt to also check the merge list that Chip
> > has
> > > proposed.'
> > >
> > > - We will also change marvin to easily add zones with actual hardware.
> > It
> > > will be based on a data driven document to do the setup.
> > >
> > > For a developer to checkin:
> > >
> > > - S/he must writes marvin tests for their feature and add it to the
> BVT.
> > >
> > > - S/he must run the checkin tests to verify everything works.
> > >
> > > - If the feature contains a hardware/vpx component, simulation
> must
> > be
> > > added.
> > >
> > > At this point, everything is about the developer writing in their
> > > feature branch and merging in.
> > >
> > > On infrastructure side:
> > >
> > > - We'll setup continuous BVT based on the simulator.
> > >
> > > - I again push that we must use Gerrit to test the code before it gets
> > > merge into the branch but I'll leave that for someone else to do that.
> > >
> > > Let me know what you guys think. I'll probably break out a bvt
> > > branch to work on this. Anyone want to join me?
> > >
> > > --Alex
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Frank Zhang <Fr...@citrix.com>.
>> Many types of designs/applications are better tested by automating tests at the module/package level.
>>In my experience, most coding errors are not because the code in a class was coded incorrectly but because the coder didn't understand how their class was supposed to work with other classes.
If you guys remembered, I have argued with you intensely last month (in a lunch time, not in maillist), now I am happy to see you finally find out somebody supporting my idea.
I would not hesitate to say again, the unit tests you guys thinking, talking, doing are INPRACTICAL and valueless.
Testing on method level, mocking everything(current CloudStack doing, the idea you guys like) is commonly used in textbook and library project, it's not suitable for integration project like CloudStack. If you force people to do so,
the only thing you get is tons of fake test cases.
Again, I would say loudly that unit test in CloudStack should be in COMPONENT level. By Component level I mean, the test cases actual test workflow of a component, not to test if a 50 line method correctly returns a true or throws an exception.
You can argue with me again that what I am saying is not unit test but integration test. Then I would say UNIT is not only a method or a class, the UNIT can be a component that is up to how your project defines it.
Yesterday I heard from Min that our unit test doesn't allow to deployDB data because faking some DB data in test IS NOT unit test. The point sounds like a joke to me.
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Tuesday, March 05, 2013 11:14 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [PROPOSAL] BVT for CloudStack checkins
>
> Recently, I read a topic about, "why automated test keeping
> failing"(http://programmers.stackexchange.com/questions/185819/why-
> does-automated-testing-keep-failing-in-my-company/185839#185839), the
> comment is really resonated with me:
>
> "Most unit tests are questionable in value. Since the vast majority of tests
> seem to be too simple."
> "It is much harder to write good testable code than just working code."
> "Many types of designs/applications are better tested by automating tests at
> the module/package level. In my experience, most coding errors are not
> because the code in a class was coded incorrectly but because the coder
> didn't understand how their class was supposed to work with other classes."
>
> We'd better put our limited resource on BVT test on both simulator and real
> hardware, at least, our BVT should cover all the hypervisors and all the
> cloudstack APIs.
>
> So +1.
>
> > -----Original Message-----
> > From: Alex Huang [mailto:Alex.Huang@citrix.com]
> > Sent: Tuesday, March 05, 2013 10:42 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: [PROPOSAL] BVT for CloudStack checkins
> >
> > Hi All,
> >
> > As most of you are aware, the master branch keeps getting broken by
> > checkins for various reasons. Committers need to be more responsible
> > about their checkins but I don't think we can depend on that happening.
> > There are various reasons. The most obvious to me is that granting
> > committership is not based on code competency. (And I don't think it
> > should.) Given that we need to build a BVT system for ensure that
> > checkins do not break the branch.
> >
> > Here's my proposal:
> >
> > Existing components that we'll use.
> >
> > - Citrix has contributed its testing to Apache.
> >
> > - Apache CloudStack has already a simulator that's been used for scale
> > testing.
> >
> > - Marvin
> >
> > - DevCloud-kvm
> >
> > Work Proposal:
> >
> > - Convert the Citrix testing into three phases:
> >
> > o Setup
> >
> > o Test
> >
> > o Verify
> >
> > - Add a Setup and Verify phase for the simulator
> >
> > - Add all of the agent commands necessary for the simulator to pass
> the
> > testing.
> >
> > - Add a Setup and Verify phase for devCloud-kvm
> >
> > - Add two more profiles to pom
> >
> > o Checkin-test-with-simulator: Runs the testing against the simulator
> >
> > o Checkin-test-with-devCloud: Runs the testing against devcloud
> >
> > - All of the profiles will attempt to also check the merge list that Chip
> has
> > proposed.'
> >
> > - We will also change marvin to easily add zones with actual hardware.
> It
> > will be based on a data driven document to do the setup.
> >
> > For a developer to checkin:
> >
> > - S/he must writes marvin tests for their feature and add it to the BVT.
> >
> > - S/he must run the checkin tests to verify everything works.
> >
> > - If the feature contains a hardware/vpx component, simulation must
> be
> > added.
> >
> > At this point, everything is about the developer writing in their
> > feature branch and merging in.
> >
> > On infrastructure side:
> >
> > - We'll setup continuous BVT based on the simulator.
> >
> > - I again push that we must use Gerrit to test the code before it gets
> > merge into the branch but I'll leave that for someone else to do that.
> >
> > Let me know what you guys think. I'll probably break out a bvt branch
> > to work on this. Anyone want to join me?
> >
> > --Alex
RE: [PROPOSAL] BVT for CloudStack checkins
Posted by Edison Su <Ed...@citrix.com>.
Recently, I read a topic about, "why automated test keeping failing"(http://programmers.stackexchange.com/questions/185819/why-does-automated-testing-keep-failing-in-my-company/185839#185839), the comment is really resonated with me:
"Most unit tests are questionable in value. Since the vast majority of tests seem to be too simple."
"It is much harder to write good testable code than just working code."
"Many types of designs/applications are better tested by automating tests at the module/package level. In my experience, most coding errors are not because the code in a class was coded incorrectly but because the coder didn't understand how their class was supposed to work with other classes."
We'd better put our limited resource on BVT test on both simulator and real hardware, at least, our BVT should cover all the hypervisors and all the cloudstack APIs.
So +1.
> -----Original Message-----
> From: Alex Huang [mailto:Alex.Huang@citrix.com]
> Sent: Tuesday, March 05, 2013 10:42 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: [PROPOSAL] BVT for CloudStack checkins
>
> Hi All,
>
> As most of you are aware, the master branch keeps getting broken by
> checkins for various reasons. Committers need to be more responsible
> about their checkins but I don't think we can depend on that happening.
> There are various reasons. The most obvious to me is that granting
> committership is not based on code competency. (And I don't think it
> should.) Given that we need to build a BVT system for ensure that checkins
> do not break the branch.
>
> Here's my proposal:
>
> Existing components that we'll use.
>
> - Citrix has contributed its testing to Apache.
>
> - Apache CloudStack has already a simulator that's been used for scale
> testing.
>
> - Marvin
>
> - DevCloud-kvm
>
> Work Proposal:
>
> - Convert the Citrix testing into three phases:
>
> o Setup
>
> o Test
>
> o Verify
>
> - Add a Setup and Verify phase for the simulator
>
> - Add all of the agent commands necessary for the simulator to pass the
> testing.
>
> - Add a Setup and Verify phase for devCloud-kvm
>
> - Add two more profiles to pom
>
> o Checkin-test-with-simulator: Runs the testing against the simulator
>
> o Checkin-test-with-devCloud: Runs the testing against devcloud
>
> - All of the profiles will attempt to also check the merge list that Chip has
> proposed.'
>
> - We will also change marvin to easily add zones with actual hardware. It
> will be based on a data driven document to do the setup.
>
> For a developer to checkin:
>
> - S/he must writes marvin tests for their feature and add it to the BVT.
>
> - S/he must run the checkin tests to verify everything works.
>
> - If the feature contains a hardware/vpx component, simulation must be
> added.
>
> At this point, everything is about the developer writing in their feature
> branch and merging in.
>
> On infrastructure side:
>
> - We'll setup continuous BVT based on the simulator.
>
> - I again push that we must use Gerrit to test the code before it gets
> merge into the branch but I'll leave that for someone else to do that.
>
> Let me know what you guys think. I'll probably break out a bvt branch to
> work on this. Anyone want to join me?
>
> --Alex