You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Pierre-Luc Dion <pd...@cloudops.com> on 2014/08/20 20:20:25 UTC
[DISCUSSION] CI
I'd like to move this discussion out of the GIT workflow thread.
Do we have any CI strategy in place or a starting point at least?
I've never worked with marvin I from the small understanding that I have
from it, it might not be the right too. I don't know.
I've spend some times recently with test-kitchen and serverspec and found
that quite promising as test platform for "real cloudstack deployment test
scenarios". By using test framework like Rspec the test output is clean to
read for human and test scenarios seams quite simple to write.
I've started a rspec lib for cloudstack to enhance serverspec for
CloudStack tests (cloudstack-rspec). I'm wondering if it would be usable
in future CI system for CloudStack...
Also, having a CI in place for cloudstack is quite of a huge deal if we
want to validate all kind of deployment scenario, networks, hypervisor
version and guest OS version.
For those that have some knowledge on the CI subject in CloudStack please
update the community. I don't feel alone being lost about this CloudStack
topic.
Thanks,
Pierre-Luc
Re: [DISCUSSION] CI
Posted by Leo Simons <LS...@schubergphilis.com>.
Hey folks,
Glad to see all the attention on this topic. Let’s do more testing! :-)
CI that we have internally
--------------------------
At schuberg philis we have a “cloudstack integration test” (CIT) environment. It’s grown and evolved incrementally over time along with our use of cloudstack, so its strongest test coverage is for the things the folks over here have worked on the most.
The current version consists of an assortment of shell and python scripts that use knife-cloudstack, various hackery, and various bash and python to bring up a fresh environment, and then some custom test cases written using the 4.2/4.3 version of marvin to provision a zone and do basic tests. It’s pretty closely tied to our internal systems (hardcoded template UUIDs are just the start of it).
On the upside, this stuff does run fully automatically from our internal jenkins. The basic trick we use is to spin up a pristine environment inside of our existing internal cloudstack-powered cloud, where we have a dedicated zone for it, backed by some dedicated vmware hypervisors (since xen doesn’t nest), into which we nest xen server (kvm experiments have never been completed — it hasn’t been a big priority since we don’t use KVM).
For an impression of what this kind of code looks like, take a look at the workshop scripts hugo put up at
https://github.com/spark404/cs-workshop
that’s using cloudmonkey instead of knife, but the basic idea is similar. It’s not that special, just lots of little bits and gotchas to it. Oh, we do also have an open source version of “our” cloudstack cookbook
https://github.com/schubergphilis/cloudstack-cookbook
though I think that’s not quite the same as what we use.
CI changes I’m working on
-------------------------
I’ve been working on and off to reduce schuberg philis specificness of all the scripts so that we can open source (and contribute here if that makes sense…though if citrix can open source their internal CI tooling, that might be a much better place to start) the end result.
Starting off from Ian’s GSOC-2014 work, we’re now doing the environment provisioning using vagrant and vagrant-cloudstack, with various scripts slowly being replaced by chef cookbooks. This new setup is at the point where we have basic environment provisioning working somewhat robustly, and its decoupled from most of our expectations. By using vagrant and some parameterization, we can now _almost_ bring up “a personal cloud” for anyone doing cloudstack development internally, that is more or less the same as the one used in our CIT environment.
The big lesson learned is that _almost_works_ is miles off from _works reliably enough to trust the test results_. The nature of the thing being built (and rebuilt) is simply that most of the time a test failure is caused by some glitch somewhere other than the actual test. I.e. there might be a failure to apply a port forwarding rule, or several failures followed by eventual success, but success _after_ the test has timed out, and then the whole thing topples over and you have to restart a two-hour long build process.
I would love to learn more about how citrix QA handles that kind of thing? I imagine, basically, with a lot of patience :-)
Anyway, once we have this vagrant-ified stuff up and running for integration needs, in the coming months, we’ll look at aligning it closer to our production cloud infrastructure (which is also chef-managed). This is where open source “CIT” would pay off, since I imagine such a codified how-to-production-cloud is of interest!
Somewhere along the line I am assuming this work can also be fully aligned with broader community setup/tooling but I can’t see that far ahead yet, especially since I don’t know the shape of what’s "out there” :-)
Marvin...
---------
Last week I started trying to get some of the existing marvin-based component tests running on our CIT. Honestly, this has been pretty frustrating and slow-going…I’m not exactly sure why just yet. Testing with python is supposed to be real fun :-).
I hate pointing at things and saying “I know a better way to do that” without fully understanding why they are a certain way. I’d prefer to get a larger chunk of the component tests up and running, first, and blame my own limitations for my frustration until then. So far, I guess marvin (and/or its usage) just doesn’t feel very pythonic (especially when contrasted with, say, cloud monkey…was marvin written before there was cloud monkey?). I’m interested in contributing to improving this, though, when I figure out how!
Regardless of reservations about how convenient or pretty the code is, there’s lots and lots of component tests that work this way, and a lot of people working on those tests every day. I think the only realistic approach is to build on top of that existing work — so we can have _one_ suite of tests, and so that anyone that can work on one test suite can work on all of them. Even if I’d personally prefer serverspec to envassert :)
Strategy
--------
> Do we have any CI strategy in place or a starting point at least?
Here’s my take.
* I think Ian’s GSOC-2014 vagrant-based devcloud3 (perhaps a better name? ;-)) should become the starting point for everyone doing local development and testing. The smoke tests should run and pass against that.
* Automated CI smoke tests should have an environment setup close to / compatible with / reusing that devcloud3 stuff, to avoid “test works locally, fails on CI”. Having the free TravisCI approach available to any developer is great. Some kind of configuration management (i.e. chef) is vital to help spin up resources reliably enough. The jobs being ran should have all the detail necessary to reproduce them in source control, so that you can figure out how to replicate a failure without needing access to the CI system(s) that failed.
* I believe the current split between smoke test w/ simulator and component test w/ target hardware/software is about right. I don’t believe more extensive non-simulator tests with somehow faked/awkward hardware (i.e. some kind of lightweight docker wrapper to act as a hypervisor) is actually very useful, since that way you don’t test the component-specific java code, of which there is a lot.
* The existing marvin-based component test suite runs need to, piece by piece, a step at a time, migrate out of running inside citrix only, to also running on open/public/accessible CI resources. The first place to migrate to is probably jenkins.buildacloud.org. I don’t think TravisCI should be used for this. Apache has buildbot and jenkins, it should be one of those, and since we have jenkins already… :)
* It’s possible already to contribute slave nodes to the buildacloud.org install that have access to the resources needed to run the component tests. For example, I hope we can get (a copy/version of) our own CIT environment hooked up to it at some point, so we can contribute test results there. How to do this probably needs better documentation (at the moment, I know that “ask Hugo” works…but I realize not everyone has the luxury of an on-site Hugo!) and call-to-action.
* buildacloud.org needs to become an ASF-hosted resource. I believe some people are working on that. I don’t know much about builds.apache.org, but looking at how much that’s already doing, I imagine merging the two is out of the question. Ideally, I guess, at some point in the future (2016? :)) the “apache cloudstack jenkins” spins up a cloud into which builds.apache.org can spin up slaves (and apache buildbot for that matter), and that way the “slave running hardware” can get pooled.
* All tests _should_ be run for every commit, but there’s no chance of getting those kinds of resources in place unless the build is refactored into lots of independently testable modules that you can pinpoint commits to. So, tests will be ran as frequently as available resources allow.
** I imagine this means doing only 3-10 full CI runs a day. That means picking the branches to run, i.e. the active trunk (whether you call it master or develop) and active release branches.
** I do imagine there’s room for running a subset of the full suite (i.e. the smoke tests) regularly (if not _every_ commit) on, say, active feature branches.
** For this branch-selection it would be nice to have a branch naming convention that’s easily pattern matched and followed by all...
* I’m interested to see whether the marvin tests can be clearly split into distinct suites that stack on top of each other, i.e. some kind of dependency chain where it is quite clear that one pack of tests reuses outputs/config results from another pack. The deployDatacenter module for example, that could be structured as a set of “stage1” test cases. Down the line I want to be able to take a partially provisioned integration environment, drop a new RPM into it, and rerun just some of the component tests related to the component I’m working on — and I want jenkins to do that for me :-)
* Things that can be broken out into components that can be partially integration tested outside of a “full” environment, should be. For example, we’re putting a framework around the systemvm so it can be tested without having a running management server. I would like to see something similar for plugins, so you can have just the plugin + the thing that it plugs in, and run a suite of tests for just that.
cheers,
Leo
RE: [DISCUSSION] CI
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> Sent: Thursday, August 21, 2014 11:21 AM
> To: dev@cloudstack.apache.org
> Cc: Will Stevens
> Subject: Re: [DISCUSSION] CI
>
> On Thu, Aug 21, 2014 at 1:32 PM, Edison Su <Ed...@citrix.com> wrote:
>
> >
> >
> > > -----Original Message-----
> > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > Sent: Thursday, August 21, 2014 5:06 AM
> > > To: dev@cloudstack.apache.org
> > > Cc: Will Stevens
> > > Subject: Re: [DISCUSSION] CI
> > >
> > > Thanks for you responses,
> > >
> > > The revamp of marvin look promising, I'll have a closer look at it,
> > > even
> > if I'm
> > > not Python fan ;). In our side, we have close to fully automated
> > installation
> > > of CloudStack, XenServer hypervisor and an initial zone creation.
> > > Our
> > missing
> >
> >
> > Will it be open sourced?:)
>
>
> Of course :-) for the Cloudstack configuration part.
> https://github.com/cloudops/cookbook_cloudstack
>
> For the XenServer side it's a bit arch because we basically manage xenserver
> kickstart with a Chef Cookbook that is quite ugly. but could be opensourced.
Internally in Citrix(every time I mentioned my company name in ML, I am so nerves.. ), we almost do
The same thing, cobber/kickstart to install hypervisors, we support KVM/Vmware/Xenserver.
Let me ask the team working it, when can we open source the code.
>
> Zone, service offerings, templates creation, I have a set of scripts not on git
> hub yet (still too ugly) but marven does it too.
>
>
>
> > > part is automated test cases to use it. So we could provide
> > infrastructure and
> > > run tests against it for one or more test scenarios during QA phase
> > > of
> > release.
> > > Hopefully this could be integrated to the CI.
> > Currently http://jenkins.buildacloud.org/, is CloudStack CI portal.
> > If you can provide Infrastructure to integrate with it, that will be
> > awesome!
> >
> > >
> > > Would it make sense to have a sheet, maybe a wiki page that list
> > > all infrastructure test scenarios and their result per version of ACS?
> > > for:
> > > xenserver, hyperv, vmware,kvm
> > > advanced, simplenetworking, vlan/vxlan,gre isolated, networks
> > s3,swift,nfs,
> > > storage plugin
> > >
> > > Unless we have this kind of results somewhere, I don't feel we
> > > really
> > keep
> > > track of what has been tested on each release.
> > It's a good idea. Let me ask the team who wrote these test cases,
> > maybe they have answers.
> >
> > >
> > >
> > >
> > >
> > > *Pierre-Luc DION*
> > > Architecte de Solution Cloud | Cloud Solutions Architect t
> > > 855.652.5683
> > >
> > > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
> > > tw @CloudOps_
> > >
> > >
> > >
> > > On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <ts...@qubole.com>
> > > wrote:
> > >
> > > > I use rspec in Ruby these days and find it get complex to maintain
> > > > pretty quickly. It's good for dealing with REST apis and
> > > > middleware but with Marvin (and python in general) I find
> > > > infrastructure handling easier. Stack traces of BDD code is also a
> > > > little bit hard to troubleshoot and comprehend. I may be biased
> > > > because my Python expertise is better, so you can take this with a pinch
> of salt.
> > > >
> > > > Edison is right however, the setup in marvin is atrociously
> > > > complex and needs to be a lot simpler and intuitive. In most cases
> > > > I don't want to specify tons of args (service offering, disk
> > > > offering, network
> > > > type) to create a simple vM that is part of every test. To this
> > > > end I refactored Marvin to use easily defined factories (in lines
> > > > with Ruby's factory_girl) and also demo-ed some code in the
> > > > branch. Haven't had time to maintain and revamp this honestly. So
> > > > if anyone wants to pick up from there, I'm happy to help.
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refact
> > > or
> > > >
> > > >
> > > >
> > > > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Ed...@citrix.com>
> > wrote:
> > > > >
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > >> Sent: Wednesday, August 20, 2014 11:20 AM
> > > > >> To: dev@cloudstack.apache.org; Will Stevens
> > > > >> Subject: [DISCUSSION] CI
> > > > >>
> > > > >> I'd like to move this discussion out of the GIT workflow thread.
> > > > >>
> > > > >> Do we have any CI strategy in place or a starting point at least?
> > > > > TravisCI will be a starting point for now. It tests against
> > > > > simulator
> > > > only.
> > > > > I am working on a solution to test against KVM(that's most of
> > > > > users are
> > > > concerned about) in a machine I created in
> > > > > Public cloud. It has big memory, super fast disk IO, should be a
> > > > > good
> > > > solution for KVM test. It may take few days to get it work.
> > > > > They are both for simple test, and in small scale.
> > > > >>
> > > > >> I've never worked with marvin I from the small understanding
> > > > >> that I have from it, it might not be the right too. I don't know.
> > > > >>
> > > > >> I've spend some times recently with test-kitchen and serverspec
> > > > >> and
> > > > found
> > > > >> that quite promising as test platform for "real cloudstack
> > > > >> deployment
> > > > test
> > > > >> scenarios". By using test framework like Rspec the test output
> > > > >> is clean
> > > > to
> > > > >> read for human and test scenarios seams quite simple to write.
> > > > >>
> > > > >> I've started a rspec lib for cloudstack to enhance serverspec
> > > > >> for
> > > > CloudStack
> > > > >> tests (cloudstack-rspec). I'm wondering if it would be usable
> > > > >> in
> > > > future CI
> > > > >> system for CloudStack...
> > > > > I am familiar with Specs, like Rspec, but in scala. I agree
> > > > > with you
> > > > that the BDD style test case
> > > > > Is easier to read/write. It's good to have, but I think before
> > > > > we move
> > > > to new test framework, we should figure out the issues
> > > > > We have today in Marvin, so that we can take effort to address
> > > > > them,
> > > > either fixing Marvin, or switch to a new test framework.
> > > > > The issue I have in current Marvin, is that it still needs to
> > > > > write a
> > > > lot of setup code for each test case, which is complicated enough.
> > > > Let's take an Marvin test case as an example:
> > > > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=
> > > > test
> > > >
> > >
> /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad
> > > 38
> > > > 72d5e27342d91;hb=HEAD
> > > > >
> > > > > Like this kind of code in test case setup code:
> > > > >
> > > > > # Get Zone, Domain and templates
> > > > > cls.domain = get_domain(cls.apiclient)
> > > > > cls.zone = get_zone(cls.apiclient,
> > testClient.getZoneForTests())
> > > > > cls.services['mode'] = cls.zone.networktype
> > > > >
> > > > > #If local storage is enabled, alter the offerings to use
> > > > localstorage
> > > > > #this step is needed for devcloud
> > > > > if cls.zone.localstorageenabled == True:
> > > > >
> > > > > cls.services["service_offerings"]["tiny"]["storagetype"]
> > > > > =
> > > > 'local'
> > > > >
> > > > > cls.services["service_offerings"]["small"]["storagetype"] =
> > > > 'local'
> > > > >
> > > > > cls.services["service_offerings"]["medium"]["storagetype"] =
> > > > 'local'
> > > > >
> > > > > this kind of information should be provided by test framework
> > > > automatically, instead of explicitly set in every test case.
> > > > > Do you have other concerns about current Marvin, other than test
> > style?
> > > > >
> > > > >
> > > > >>
> > > > >> Also, having a CI in place for cloudstack is quite of a huge
> > > > >> deal if we
> > > > want to
> > > > >> validate all kind of deployment scenario, networks, hypervisor
> > > > >> version
> > > > and
> > > > >> guest OS version.
> > > > >
> > > > >
> > > > > Can't agree with you more.
> > > > >
> > > > >>
> > > > >>
> > > > >> For those that have some knowledge on the CI subject in
> > > > >> CloudStack
> > > > please
> > > > >> update the community. I don't feel alone being lost about this
> > > > CloudStack
> > > > >> topic.
> > > > >
> > > > > There is an update-to-date write up regarding to Marvin:
> > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-
> > > +Testin
> > > > g+with+Python
> > > > > Marvin does two things, one is to call cloudstack mgt API to add
> > > > resources into CloudStack mgt server, such as hypervisors, primary
> > > > storages, networks, and secondary storages etc.
> > > > > Another one is to run test cases under tools/integration/smoke
> > > > > or any
> > > > other python test cases.
> > > > > It doesn't cover how to deploy hypervisors on real HW, how to
> > > > > setup
> > > > primary storages, secondary storage etc. That's something like
> > > > test-kitchen and serverspec can help, I think.
> > > > >
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >>
> > > > >> Pierre-Luc
> > > >
> > > >
> > > >
> > > > --
> > > > Prasanna.,
> > > >
> >
Re: [DISCUSSION] CI
Posted by Pierre-Luc Dion <pd...@cloudops.com>.
On Thu, Aug 21, 2014 at 1:32 PM, Edison Su <Ed...@citrix.com> wrote:
>
>
> > -----Original Message-----
> > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > Sent: Thursday, August 21, 2014 5:06 AM
> > To: dev@cloudstack.apache.org
> > Cc: Will Stevens
> > Subject: Re: [DISCUSSION] CI
> >
> > Thanks for you responses,
> >
> > The revamp of marvin look promising, I'll have a closer look at it, even
> if I'm
> > not Python fan ;). In our side, we have close to fully automated
> installation
> > of CloudStack, XenServer hypervisor and an initial zone creation. Our
> missing
>
>
> Will it be open sourced?:)
Of course :-) for the Cloudstack configuration part.
https://github.com/cloudops/cookbook_cloudstack
For the XenServer side it's a bit arch because we basically manage
xenserver kickstart with a Chef Cookbook that is quite ugly. but could be
opensourced.
Zone, service offerings, templates creation, I have a set of scripts not on
git hub yet (still too ugly) but marven does it too.
> > part is automated test cases to use it. So we could provide
> infrastructure and
> > run tests against it for one or more test scenarios during QA phase of
> release.
> > Hopefully this could be integrated to the CI.
> Currently http://jenkins.buildacloud.org/, is CloudStack CI portal. If
> you can provide
> Infrastructure to integrate with it, that will be awesome!
>
> >
> > Would it make sense to have a sheet, maybe a wiki page that list all
> > infrastructure test scenarios and their result per version of ACS?
> > for:
> > xenserver, hyperv, vmware,kvm
> > advanced, simplenetworking, vlan/vxlan,gre isolated, networks
> s3,swift,nfs,
> > storage plugin
> >
> > Unless we have this kind of results somewhere, I don't feel we really
> keep
> > track of what has been tested on each release.
> It's a good idea. Let me ask the team who wrote these test cases, maybe
> they have answers.
>
> >
> >
> >
> >
> > *Pierre-Luc DION*
> > Architecte de Solution Cloud | Cloud Solutions Architect t 855.652.5683
> >
> > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> > @CloudOps_
> >
> >
> >
> > On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <ts...@qubole.com>
> > wrote:
> >
> > > I use rspec in Ruby these days and find it get complex to maintain
> > > pretty quickly. It's good for dealing with REST apis and middleware
> > > but with Marvin (and python in general) I find infrastructure handling
> > > easier. Stack traces of BDD code is also a little bit hard to
> > > troubleshoot and comprehend. I may be biased because my Python
> > > expertise is better, so you can take this with a pinch of salt.
> > >
> > > Edison is right however, the setup in marvin is atrociously complex
> > > and needs to be a lot simpler and intuitive. In most cases I don't
> > > want to specify tons of args (service offering, disk offering, network
> > > type) to create a simple vM that is part of every test. To this end I
> > > refactored Marvin to use easily defined factories (in lines with
> > > Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
> > > had time to maintain and revamp this honestly. So if anyone wants to
> > > pick up from there, I'm happy to help.
> > >
> > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
> > >
> > >
> > >
> > > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Ed...@citrix.com>
> wrote:
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > >> Sent: Wednesday, August 20, 2014 11:20 AM
> > > >> To: dev@cloudstack.apache.org; Will Stevens
> > > >> Subject: [DISCUSSION] CI
> > > >>
> > > >> I'd like to move this discussion out of the GIT workflow thread.
> > > >>
> > > >> Do we have any CI strategy in place or a starting point at least?
> > > > TravisCI will be a starting point for now. It tests against
> > > > simulator
> > > only.
> > > > I am working on a solution to test against KVM(that's most of users
> > > > are
> > > concerned about) in a machine I created in
> > > > Public cloud. It has big memory, super fast disk IO, should be a
> > > > good
> > > solution for KVM test. It may take few days to get it work.
> > > > They are both for simple test, and in small scale.
> > > >>
> > > >> I've never worked with marvin I from the small understanding that I
> > > >> have from it, it might not be the right too. I don't know.
> > > >>
> > > >> I've spend some times recently with test-kitchen and serverspec and
> > > found
> > > >> that quite promising as test platform for "real cloudstack
> > > >> deployment
> > > test
> > > >> scenarios". By using test framework like Rspec the test output is
> > > >> clean
> > > to
> > > >> read for human and test scenarios seams quite simple to write.
> > > >>
> > > >> I've started a rspec lib for cloudstack to enhance serverspec for
> > > CloudStack
> > > >> tests (cloudstack-rspec). I'm wondering if it would be usable in
> > > future CI
> > > >> system for CloudStack...
> > > > I am familiar with Specs, like Rspec, but in scala. I agree with
> > > > you
> > > that the BDD style test case
> > > > Is easier to read/write. It's good to have, but I think before we
> > > > move
> > > to new test framework, we should figure out the issues
> > > > We have today in Marvin, so that we can take effort to address them,
> > > either fixing Marvin, or switch to a new test framework.
> > > > The issue I have in current Marvin, is that it still needs to write
> > > > a
> > > lot of setup code for each test case, which is complicated enough.
> > > Let's take an Marvin test case as an example:
> > > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test
> > >
> > /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad38
> > > 72d5e27342d91;hb=HEAD
> > > >
> > > > Like this kind of code in test case setup code:
> > > >
> > > > # Get Zone, Domain and templates
> > > > cls.domain = get_domain(cls.apiclient)
> > > > cls.zone = get_zone(cls.apiclient,
> testClient.getZoneForTests())
> > > > cls.services['mode'] = cls.zone.networktype
> > > >
> > > > #If local storage is enabled, alter the offerings to use
> > > localstorage
> > > > #this step is needed for devcloud
> > > > if cls.zone.localstorageenabled == True:
> > > > cls.services["service_offerings"]["tiny"]["storagetype"]
> > > > =
> > > 'local'
> > > >
> > > > cls.services["service_offerings"]["small"]["storagetype"] =
> > > 'local'
> > > >
> > > > cls.services["service_offerings"]["medium"]["storagetype"] =
> > > 'local'
> > > >
> > > > this kind of information should be provided by test framework
> > > automatically, instead of explicitly set in every test case.
> > > > Do you have other concerns about current Marvin, other than test
> style?
> > > >
> > > >
> > > >>
> > > >> Also, having a CI in place for cloudstack is quite of a huge deal
> > > >> if we
> > > want to
> > > >> validate all kind of deployment scenario, networks, hypervisor
> > > >> version
> > > and
> > > >> guest OS version.
> > > >
> > > >
> > > > Can't agree with you more.
> > > >
> > > >>
> > > >>
> > > >> For those that have some knowledge on the CI subject in CloudStack
> > > please
> > > >> update the community. I don't feel alone being lost about this
> > > CloudStack
> > > >> topic.
> > > >
> > > > There is an update-to-date write up regarding to Marvin:
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-
> > +Testin
> > > g+with+Python
> > > > Marvin does two things, one is to call cloudstack mgt API to add
> > > resources into CloudStack mgt server, such as hypervisors, primary
> > > storages, networks, and secondary storages etc.
> > > > Another one is to run test cases under tools/integration/smoke or
> > > > any
> > > other python test cases.
> > > > It doesn't cover how to deploy hypervisors on real HW, how to setup
> > > primary storages, secondary storage etc. That's something like
> > > test-kitchen and serverspec can help, I think.
> > > >
> > > >>
> > > >> Thanks,
> > > >>
> > > >>
> > > >> Pierre-Luc
> > >
> > >
> > >
> > > --
> > > Prasanna.,
> > >
>
RE: [DISCUSSION] CI
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> Sent: Thursday, August 21, 2014 5:06 AM
> To: dev@cloudstack.apache.org
> Cc: Will Stevens
> Subject: Re: [DISCUSSION] CI
>
> Thanks for you responses,
>
> The revamp of marvin look promising, I'll have a closer look at it, even if I'm
> not Python fan ;). In our side, we have close to fully automated installation
> of CloudStack, XenServer hypervisor and an initial zone creation. Our missing
Will it be open sourced?:)
> part is automated test cases to use it. So we could provide infrastructure and
> run tests against it for one or more test scenarios during QA phase of release.
> Hopefully this could be integrated to the CI.
Currently http://jenkins.buildacloud.org/, is CloudStack CI portal. If you can provide
Infrastructure to integrate with it, that will be awesome!
>
> Would it make sense to have a sheet, maybe a wiki page that list all
> infrastructure test scenarios and their result per version of ACS?
> for:
> xenserver, hyperv, vmware,kvm
> advanced, simplenetworking, vlan/vxlan,gre isolated, networks s3,swift,nfs,
> storage plugin
>
> Unless we have this kind of results somewhere, I don't feel we really keep
> track of what has been tested on each release.
It's a good idea. Let me ask the team who wrote these test cases, maybe they have answers.
>
>
>
>
> *Pierre-Luc DION*
> Architecte de Solution Cloud | Cloud Solutions Architect t 855.652.5683
>
> *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> @CloudOps_
>
>
>
> On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <ts...@qubole.com>
> wrote:
>
> > I use rspec in Ruby these days and find it get complex to maintain
> > pretty quickly. It's good for dealing with REST apis and middleware
> > but with Marvin (and python in general) I find infrastructure handling
> > easier. Stack traces of BDD code is also a little bit hard to
> > troubleshoot and comprehend. I may be biased because my Python
> > expertise is better, so you can take this with a pinch of salt.
> >
> > Edison is right however, the setup in marvin is atrociously complex
> > and needs to be a lot simpler and intuitive. In most cases I don't
> > want to specify tons of args (service offering, disk offering, network
> > type) to create a simple vM that is part of every test. To this end I
> > refactored Marvin to use easily defined factories (in lines with
> > Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
> > had time to maintain and revamp this honestly. So if anyone wants to
> > pick up from there, I'm happy to help.
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
> >
> >
> >
> > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Ed...@citrix.com> wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > >> Sent: Wednesday, August 20, 2014 11:20 AM
> > >> To: dev@cloudstack.apache.org; Will Stevens
> > >> Subject: [DISCUSSION] CI
> > >>
> > >> I'd like to move this discussion out of the GIT workflow thread.
> > >>
> > >> Do we have any CI strategy in place or a starting point at least?
> > > TravisCI will be a starting point for now. It tests against
> > > simulator
> > only.
> > > I am working on a solution to test against KVM(that's most of users
> > > are
> > concerned about) in a machine I created in
> > > Public cloud. It has big memory, super fast disk IO, should be a
> > > good
> > solution for KVM test. It may take few days to get it work.
> > > They are both for simple test, and in small scale.
> > >>
> > >> I've never worked with marvin I from the small understanding that I
> > >> have from it, it might not be the right too. I don't know.
> > >>
> > >> I've spend some times recently with test-kitchen and serverspec and
> > found
> > >> that quite promising as test platform for "real cloudstack
> > >> deployment
> > test
> > >> scenarios". By using test framework like Rspec the test output is
> > >> clean
> > to
> > >> read for human and test scenarios seams quite simple to write.
> > >>
> > >> I've started a rspec lib for cloudstack to enhance serverspec for
> > CloudStack
> > >> tests (cloudstack-rspec). I'm wondering if it would be usable in
> > future CI
> > >> system for CloudStack...
> > > I am familiar with Specs, like Rspec, but in scala. I agree with
> > > you
> > that the BDD style test case
> > > Is easier to read/write. It's good to have, but I think before we
> > > move
> > to new test framework, we should figure out the issues
> > > We have today in Marvin, so that we can take effort to address them,
> > either fixing Marvin, or switch to a new test framework.
> > > The issue I have in current Marvin, is that it still needs to write
> > > a
> > lot of setup code for each test case, which is complicated enough.
> > Let's take an Marvin test case as an example:
> > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test
> >
> /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad38
> > 72d5e27342d91;hb=HEAD
> > >
> > > Like this kind of code in test case setup code:
> > >
> > > # Get Zone, Domain and templates
> > > cls.domain = get_domain(cls.apiclient)
> > > cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
> > > cls.services['mode'] = cls.zone.networktype
> > >
> > > #If local storage is enabled, alter the offerings to use
> > localstorage
> > > #this step is needed for devcloud
> > > if cls.zone.localstorageenabled == True:
> > > cls.services["service_offerings"]["tiny"]["storagetype"]
> > > =
> > 'local'
> > >
> > > cls.services["service_offerings"]["small"]["storagetype"] =
> > 'local'
> > >
> > > cls.services["service_offerings"]["medium"]["storagetype"] =
> > 'local'
> > >
> > > this kind of information should be provided by test framework
> > automatically, instead of explicitly set in every test case.
> > > Do you have other concerns about current Marvin, other than test style?
> > >
> > >
> > >>
> > >> Also, having a CI in place for cloudstack is quite of a huge deal
> > >> if we
> > want to
> > >> validate all kind of deployment scenario, networks, hypervisor
> > >> version
> > and
> > >> guest OS version.
> > >
> > >
> > > Can't agree with you more.
> > >
> > >>
> > >>
> > >> For those that have some knowledge on the CI subject in CloudStack
> > please
> > >> update the community. I don't feel alone being lost about this
> > CloudStack
> > >> topic.
> > >
> > > There is an update-to-date write up regarding to Marvin:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-
> +Testin
> > g+with+Python
> > > Marvin does two things, one is to call cloudstack mgt API to add
> > resources into CloudStack mgt server, such as hypervisors, primary
> > storages, networks, and secondary storages etc.
> > > Another one is to run test cases under tools/integration/smoke or
> > > any
> > other python test cases.
> > > It doesn't cover how to deploy hypervisors on real HW, how to setup
> > primary storages, secondary storage etc. That's something like
> > test-kitchen and serverspec can help, I think.
> > >
> > >>
> > >> Thanks,
> > >>
> > >>
> > >> Pierre-Luc
> >
> >
> >
> > --
> > Prasanna.,
> >
Re: [DISCUSSION] CI
Posted by Pierre-Luc Dion <pd...@cloudops.com>.
Thanks for you responses,
The revamp of marvin look promising, I'll have a closer look at it, even if
I'm not Python fan ;). In our side, we have close to fully automated
installation of CloudStack, XenServer hypervisor and an initial zone
creation. Our missing part is automated test cases to use it. So we could
provide infrastructure and run tests against it for one or more test
scenarios during QA phase of release. Hopefully this could be integrated to
the CI.
Would it make sense to have a sheet, maybe a wiki page that list all
infrastructure test scenarios and their result per version of ACS?
for:
xenserver, hyperv, vmware,kvm
advanced, simplenetworking, vlan/vxlan,gre isolated, networks
s3,swift,nfs, storage plugin
Unless we have this kind of results somewhere, I don't feel we really keep
track of what has been tested on each release.
*Pierre-Luc DION*
Architecte de Solution Cloud | Cloud Solutions Architect
t 855.652.5683
*CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <ts...@qubole.com> wrote:
> I use rspec in Ruby these days and find it get complex to maintain
> pretty quickly. It's good for dealing with REST apis and middleware
> but with Marvin (and python in general) I find infrastructure handling
> easier. Stack traces of BDD code is also a little bit hard to
> troubleshoot and comprehend. I may be biased because my Python
> expertise is better, so you can take this with a pinch of salt.
>
> Edison is right however, the setup in marvin is atrociously complex
> and needs to be a lot simpler and intuitive. In most cases I don't
> want to specify tons of args (service offering, disk offering, network
> type) to create a simple vM that is part of every test. To this end I
> refactored Marvin to use easily defined factories (in lines with
> Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
> had time to maintain and revamp this honestly. So if anyone wants to
> pick up from there, I'm happy to help.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
>
>
>
> On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Ed...@citrix.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> >> Sent: Wednesday, August 20, 2014 11:20 AM
> >> To: dev@cloudstack.apache.org; Will Stevens
> >> Subject: [DISCUSSION] CI
> >>
> >> I'd like to move this discussion out of the GIT workflow thread.
> >>
> >> Do we have any CI strategy in place or a starting point at least?
> > TravisCI will be a starting point for now. It tests against simulator
> only.
> > I am working on a solution to test against KVM(that's most of users are
> concerned about) in a machine I created in
> > Public cloud. It has big memory, super fast disk IO, should be a good
> solution for KVM test. It may take few days to get it work.
> > They are both for simple test, and in small scale.
> >>
> >> I've never worked with marvin I from the small understanding that I have
> >> from it, it might not be the right too. I don't know.
> >>
> >> I've spend some times recently with test-kitchen and serverspec and
> found
> >> that quite promising as test platform for "real cloudstack deployment
> test
> >> scenarios". By using test framework like Rspec the test output is clean
> to
> >> read for human and test scenarios seams quite simple to write.
> >>
> >> I've started a rspec lib for cloudstack to enhance serverspec for
> CloudStack
> >> tests (cloudstack-rspec). I'm wondering if it would be usable in
> future CI
> >> system for CloudStack...
> > I am familiar with Specs, like Rspec, but in scala. I agree with you
> that the BDD style test case
> > Is easier to read/write. It's good to have, but I think before we move
> to new test framework, we should figure out the issues
> > We have today in Marvin, so that we can take effort to address them,
> either fixing Marvin, or switch to a new test framework.
> > The issue I have in current Marvin, is that it still needs to write a
> lot of setup code for each test case, which is complicated enough. Let's
> take an Marvin test case as an example:
> https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test/integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad3872d5e27342d91;hb=HEAD
> >
> > Like this kind of code in test case setup code:
> >
> > # Get Zone, Domain and templates
> > cls.domain = get_domain(cls.apiclient)
> > cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
> > cls.services['mode'] = cls.zone.networktype
> >
> > #If local storage is enabled, alter the offerings to use
> localstorage
> > #this step is needed for devcloud
> > if cls.zone.localstorageenabled == True:
> > cls.services["service_offerings"]["tiny"]["storagetype"] =
> 'local'
> > cls.services["service_offerings"]["small"]["storagetype"] =
> 'local'
> > cls.services["service_offerings"]["medium"]["storagetype"] =
> 'local'
> >
> > this kind of information should be provided by test framework
> automatically, instead of explicitly set in every test case.
> > Do you have other concerns about current Marvin, other than test style?
> >
> >
> >>
> >> Also, having a CI in place for cloudstack is quite of a huge deal if we
> want to
> >> validate all kind of deployment scenario, networks, hypervisor version
> and
> >> guest OS version.
> >
> >
> > Can't agree with you more.
> >
> >>
> >>
> >> For those that have some knowledge on the CI subject in CloudStack
> please
> >> update the community. I don't feel alone being lost about this
> CloudStack
> >> topic.
> >
> > There is an update-to-date write up regarding to Marvin:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
> > Marvin does two things, one is to call cloudstack mgt API to add
> resources into CloudStack mgt server, such as hypervisors, primary
> storages, networks, and secondary storages etc.
> > Another one is to run test cases under tools/integration/smoke or any
> other python test cases.
> > It doesn't cover how to deploy hypervisors on real HW, how to setup
> primary storages, secondary storage etc. That's something like test-kitchen
> and serverspec can help, I think.
> >
> >>
> >> Thanks,
> >>
> >>
> >> Pierre-Luc
>
>
>
> --
> Prasanna.,
>
Re: [DISCUSSION] CI
Posted by Prasanna Santhanam <ts...@qubole.com>.
I use rspec in Ruby these days and find it get complex to maintain
pretty quickly. It's good for dealing with REST apis and middleware
but with Marvin (and python in general) I find infrastructure handling
easier. Stack traces of BDD code is also a little bit hard to
troubleshoot and comprehend. I may be biased because my Python
expertise is better, so you can take this with a pinch of salt.
Edison is right however, the setup in marvin is atrociously complex
and needs to be a lot simpler and intuitive. In most cases I don't
want to specify tons of args (service offering, disk offering, network
type) to create a simple vM that is part of every test. To this end I
refactored Marvin to use easily defined factories (in lines with
Ruby's factory_girl) and also demo-ed some code in the branch. Haven't
had time to maintain and revamp this honestly. So if anyone wants to
pick up from there, I'm happy to help.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor
On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <Ed...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
>> Sent: Wednesday, August 20, 2014 11:20 AM
>> To: dev@cloudstack.apache.org; Will Stevens
>> Subject: [DISCUSSION] CI
>>
>> I'd like to move this discussion out of the GIT workflow thread.
>>
>> Do we have any CI strategy in place or a starting point at least?
> TravisCI will be a starting point for now. It tests against simulator only.
> I am working on a solution to test against KVM(that's most of users are concerned about) in a machine I created in
> Public cloud. It has big memory, super fast disk IO, should be a good solution for KVM test. It may take few days to get it work.
> They are both for simple test, and in small scale.
>>
>> I've never worked with marvin I from the small understanding that I have
>> from it, it might not be the right too. I don't know.
>>
>> I've spend some times recently with test-kitchen and serverspec and found
>> that quite promising as test platform for "real cloudstack deployment test
>> scenarios". By using test framework like Rspec the test output is clean to
>> read for human and test scenarios seams quite simple to write.
>>
>> I've started a rspec lib for cloudstack to enhance serverspec for CloudStack
>> tests (cloudstack-rspec). I'm wondering if it would be usable in future CI
>> system for CloudStack...
> I am familiar with Specs, like Rspec, but in scala. I agree with you that the BDD style test case
> Is easier to read/write. It's good to have, but I think before we move to new test framework, we should figure out the issues
> We have today in Marvin, so that we can take effort to address them, either fixing Marvin, or switch to a new test framework.
> The issue I have in current Marvin, is that it still needs to write a lot of setup code for each test case, which is complicated enough. Let's take an Marvin test case as an example: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test/integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad3872d5e27342d91;hb=HEAD
>
> Like this kind of code in test case setup code:
>
> # Get Zone, Domain and templates
> cls.domain = get_domain(cls.apiclient)
> cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
> cls.services['mode'] = cls.zone.networktype
>
> #If local storage is enabled, alter the offerings to use localstorage
> #this step is needed for devcloud
> if cls.zone.localstorageenabled == True:
> cls.services["service_offerings"]["tiny"]["storagetype"] = 'local'
> cls.services["service_offerings"]["small"]["storagetype"] = 'local'
> cls.services["service_offerings"]["medium"]["storagetype"] = 'local'
>
> this kind of information should be provided by test framework automatically, instead of explicitly set in every test case.
> Do you have other concerns about current Marvin, other than test style?
>
>
>>
>> Also, having a CI in place for cloudstack is quite of a huge deal if we want to
>> validate all kind of deployment scenario, networks, hypervisor version and
>> guest OS version.
>
>
> Can't agree with you more.
>
>>
>>
>> For those that have some knowledge on the CI subject in CloudStack please
>> update the community. I don't feel alone being lost about this CloudStack
>> topic.
>
> There is an update-to-date write up regarding to Marvin: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
> Marvin does two things, one is to call cloudstack mgt API to add resources into CloudStack mgt server, such as hypervisors, primary storages, networks, and secondary storages etc.
> Another one is to run test cases under tools/integration/smoke or any other python test cases.
> It doesn't cover how to deploy hypervisors on real HW, how to setup primary storages, secondary storage etc. That's something like test-kitchen and serverspec can help, I think.
>
>>
>> Thanks,
>>
>>
>> Pierre-Luc
--
Prasanna.,
RE: [DISCUSSION] CI
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> Sent: Wednesday, August 20, 2014 11:20 AM
> To: dev@cloudstack.apache.org; Will Stevens
> Subject: [DISCUSSION] CI
>
> I'd like to move this discussion out of the GIT workflow thread.
>
> Do we have any CI strategy in place or a starting point at least?
TravisCI will be a starting point for now. It tests against simulator only.
I am working on a solution to test against KVM(that's most of users are concerned about) in a machine I created in
Public cloud. It has big memory, super fast disk IO, should be a good solution for KVM test. It may take few days to get it work.
They are both for simple test, and in small scale.
>
> I've never worked with marvin I from the small understanding that I have
> from it, it might not be the right too. I don't know.
>
> I've spend some times recently with test-kitchen and serverspec and found
> that quite promising as test platform for "real cloudstack deployment test
> scenarios". By using test framework like Rspec the test output is clean to
> read for human and test scenarios seams quite simple to write.
>
> I've started a rspec lib for cloudstack to enhance serverspec for CloudStack
> tests (cloudstack-rspec). I'm wondering if it would be usable in future CI
> system for CloudStack...
I am familiar with Specs, like Rspec, but in scala. I agree with you that the BDD style test case
Is easier to read/write. It's good to have, but I think before we move to new test framework, we should figure out the issues
We have today in Marvin, so that we can take effort to address them, either fixing Marvin, or switch to a new test framework.
The issue I have in current Marvin, is that it still needs to write a lot of setup code for each test case, which is complicated enough. Let's take an Marvin test case as an example: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test/integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad3872d5e27342d91;hb=HEAD
Like this kind of code in test case setup code:
# Get Zone, Domain and templates
cls.domain = get_domain(cls.apiclient)
cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
cls.services['mode'] = cls.zone.networktype
#If local storage is enabled, alter the offerings to use localstorage
#this step is needed for devcloud
if cls.zone.localstorageenabled == True:
cls.services["service_offerings"]["tiny"]["storagetype"] = 'local'
cls.services["service_offerings"]["small"]["storagetype"] = 'local'
cls.services["service_offerings"]["medium"]["storagetype"] = 'local'
this kind of information should be provided by test framework automatically, instead of explicitly set in every test case.
Do you have other concerns about current Marvin, other than test style?
>
> Also, having a CI in place for cloudstack is quite of a huge deal if we want to
> validate all kind of deployment scenario, networks, hypervisor version and
> guest OS version.
Can't agree with you more.
>
>
> For those that have some knowledge on the CI subject in CloudStack please
> update the community. I don't feel alone being lost about this CloudStack
> topic.
There is an update-to-date write up regarding to Marvin: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
Marvin does two things, one is to call cloudstack mgt API to add resources into CloudStack mgt server, such as hypervisors, primary storages, networks, and secondary storages etc.
Another one is to run test cases under tools/integration/smoke or any other python test cases.
It doesn't cover how to deploy hypervisors on real HW, how to setup primary storages, secondary storage etc. That's something like test-kitchen and serverspec can help, I think.
>
> Thanks,
>
>
> Pierre-Luc