Nightly tests

[Ahmad Emneina <Ah...@citrix.com>]

I'd like to call peoples attention to
http://jenkins.cloudstack.org/job/nightly-smoke-kvm/1/console, what were
looking at here is the culmination of some seriously awesome work. In
brief, this is an automated setup that deploys the latest cloudstack (in a
vm) and provisions 'fresh' hypervisors running on bare metal. Then marvin
configures cloudstack and drives the testing! This is all integrated with
and runs as a jenkins job. The gang responsible for this, Rohit, Edison
and Prasanna... managed to scrape time in their lives, between all the
code contribution/bug fixing, and got this up and running. Kudos, congrats
and a sincere thank you!

This is also a call to the community at large to help expand the
functional/automated test suites. Have a look in
<source_tree>/test/integration/ and peruse the README's.

-- 
Æ

Re: Nightly tests

[prasanna <ts...@apache.org>]

On 10 November 2012 03:34, David Nalley <da...@gnsa.us> wrote:
> On Fri, Nov 9, 2012 at 8:45 AM, Prasanna Santhanam
> <pr...@citrix.com> wrote:
>> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>>
>>> Since the module level fixtures are independent of each other. I will
>>> set them to execute concurrently.  Over the next week - I will add
>>> more test suites to this setup. Right now it runs 73 tests for build
>>> verification.
>>
>> Quick Update:
>> Reduced the time of run from 11000s to 3000s for the tests. Tests are
>> re-enabled now and will run each context suite in parallel.
>>
>
> That's awesome!!!
>
> Is the box that runs this (smoketest-kvm, smoketest-jenkins) a jenkins
> slave we can push from builds.a.o? Maybe that makes sense to make one
> of the first jobs we tackle since you are actively working on it.
>
> --David

Yes I think so. We run the current slave from behind Citrix firewall
in headless mode. So it shouldn't be a problem to point it to
builds.a.o once we move the jobs there.

Re: Nightly tests

[prasanna <sr...@gmail.com>]

On 10 November 2012 03:34, David Nalley <da...@gnsa.us> wrote:
> On Fri, Nov 9, 2012 at 8:45 AM, Prasanna Santhanam
> <pr...@citrix.com> wrote:
>> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>>
>>> Since the module level fixtures are independent of each other. I will
>>> set them to execute concurrently.  Over the next week - I will add
>>> more test suites to this setup. Right now it runs 73 tests for build
>>> verification.
>>
>> Quick Update:
>> Reduced the time of run from 11000s to 3000s for the tests. Tests are
>> re-enabled now and will run each context suite in parallel.
>>
>
> That's awesome!!!
>
> Is the box that runs this (smoketest-kvm, smoketest-jenkins) a jenkins
> slave we can push from builds.a.o? Maybe that makes sense to make one
> of the first jobs we tackle since you are actively working on it.
>
> --David

Yes I think so. We run the current slave from behind Citrix firewall
in headless mode. So it shouldn't be a problem to point it to
builds.a.o once we move the jobs there.

-- 
Prasanna.,

Re: Nightly tests

[David Nalley <da...@gnsa.us>]

On Fri, Nov 9, 2012 at 8:45 AM, Prasanna Santhanam
<pr...@citrix.com> wrote:
> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>
>> Since the module level fixtures are independent of each other. I will
>> set them to execute concurrently.  Over the next week - I will add
>> more test suites to this setup. Right now it runs 73 tests for build
>> verification.
>
> Quick Update:
> Reduced the time of run from 11000s to 3000s for the tests. Tests are
> re-enabled now and will run each context suite in parallel.
>

That's awesome!!!

Is the box that runs this (smoketest-kvm, smoketest-jenkins) a jenkins
slave we can push from builds.a.o? Maybe that makes sense to make one
of the first jobs we tackle since you are actively working on it.

--David

Re: Nightly tests

[Ahmad Emneina <Ah...@citrix.com>]

This sounds incredibly valuable, and what you propose with devcloud sounds like platinum. 

Ahmad

On Nov 9, 2012, at 6:18 AM, "Rohit Yadav" <ro...@citrix.com> wrote:

> Awesome! Keep us posted.
> ________________________________________
> From: Prasanna Santhanam [prasanna.santhanam@citrix.com]
> Sent: Friday, November 09, 2012 7:24 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: Nightly tests
> 
> On Fri, Nov 09, 2012 at 07:15:47PM +0530, Prasanna Santhanam wrote:
>> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>> 
>>> Since the module level fixtures are independent of each other. I will
>>> set them to execute concurrently.  Over the next week - I will add
>>> more test suites to this setup. Right now it runs 73 tests for build
>>> verification.
>> 
>> Quick Update:
>> Reduced the time of run from 11000s to 3000s for the tests. Tests are
>> re-enabled now and will run each context suite in parallel.
> 
> Next I'm going to take up running a subset of tests from devcloud so
> it can be run quickly enough to run per-checkin. Will report back how
> it goes.
> 
> --
> Prasanna.,

RE: Nightly tests

[Rohit Yadav <ro...@citrix.com>]

Awesome! Keep us posted.
________________________________________
From: Prasanna Santhanam [prasanna.santhanam@citrix.com]
Sent: Friday, November 09, 2012 7:24 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: Nightly tests

On Fri, Nov 09, 2012 at 07:15:47PM +0530, Prasanna Santhanam wrote:
> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>
> > Since the module level fixtures are independent of each other. I will
> > set them to execute concurrently.  Over the next week - I will add
> > more test suites to this setup. Right now it runs 73 tests for build
> > verification.
>
> Quick Update:
> Reduced the time of run from 11000s to 3000s for the tests. Tests are
> re-enabled now and will run each context suite in parallel.
>

Next I'm going to take up running a subset of tests from devcloud so
it can be run quickly enough to run per-checkin. Will report back how
it goes.

--
Prasanna.,

Re: Nightly tests

[Prasanna Santhanam <pr...@citrix.com>]

On Fri, Nov 09, 2012 at 07:15:47PM +0530, Prasanna Santhanam wrote:
> On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
>  
> > Since the module level fixtures are independent of each other. I will
> > set them to execute concurrently.  Over the next week - I will add
> > more test suites to this setup. Right now it runs 73 tests for build
> > verification.
> 
> Quick Update:
> Reduced the time of run from 11000s to 3000s for the tests. Tests are
> re-enabled now and will run each context suite in parallel.
> 

Next I'm going to take up running a subset of tests from devcloud so
it can be run quickly enough to run per-checkin. Will report back how
it goes.

-- 
Prasanna.,

Re: Nightly tests

[Prasanna Santhanam <pr...@citrix.com>]

On Fri, Nov 02, 2012 at 06:14:34PM +0530, Prasanna Santhanam wrote:
 
> Since the module level fixtures are independent of each other. I will
> set them to execute concurrently.  Over the next week - I will add
> more test suites to this setup. Right now it runs 73 tests for build
> verification.

Quick Update:
Reduced the time of run from 11000s to 3000s for the tests. Tests are
re-enabled now and will run each context suite in parallel.

Thanks,

-- 
Prasanna.,

Re: Nightly tests

[Prasanna Santhanam <pr...@citrix.com>]

On Fri, Nov 02, 2012 at 01:48:35PM -0400, Chiradeep Vittal wrote:
> +1 to this awesomeness.
> - where is this test case repo so we can see the 73 tests?

The tests are within incubator-cloudstack/test/integration/smoke.
There are more tests in test/integration/component. Currently, the
test infrastructure will look for everything that is a cloudstack
testcase under these directories and launch them against our internal
deployment.

> - what is the size of this infrastructure?

Infrastructure is small right now. We've got 4 hosts spread across two
zones (3/1) to qualify inter-cluster, inter-pod, inter-zone scenarios.

It's up to Marvin's configuration file[1] on how we want to distribute
the hosts we have in to the kind of deployment that one is planning.
And since it is a simplified json representation of the deployment it
can be maintained easily.

> - can someone replicate this?

Not immediately. But we'd like for it to be replicated. We've tried to
use DNS names for all machines so as not to tie things down to
ipaddresses. We have some cleanup to untangle some of the
environmental dependencies. So we're writing down every service,
process, and infrastructure component into repeatable puppet recipes
[2] so there's not too much overhead setting all this up.  

[1] https://github.com/vogxn/cloud-autodeploy/blob/acs-infra-test/kvm.cfg
[2] https://github.com/vogxn/acs-infra-test

Thanks,

-- 
Prasanna.,

Re: Nightly tests

[Chiradeep Vittal <Ch...@citrix.com>]

+1 to this awesomeness.
- where is this test case repo so we can see the 73 tests?
- what is the size of this infrastructure?
- can someone replicate this?

On 11/2/12 5:44 AM, "Prasanna Santhanam" <pr...@citrix.com>
wrote:

>On Wed, Oct 31, 2012 at 11:14:18PM +0530, Ahmad Emneina wrote:
>> I'd like to call peoples attention to
>> http://jenkins.cloudstack.org/job/nightly-smoke-kvm/1/console, what were
>> looking at here is the culmination of some seriously awesome work. In
>> brief, this is an automated setup that deploys the latest cloudstack
>>(in a
>> vm) and provisions 'fresh' hypervisors running on bare metal. Then
>>marvin
>> configures cloudstack and drives the testing! This is all integrated
>>with
>> and runs as a jenkins job. The gang responsible for this, Rohit, Edison
>> and Prasanna... managed to scrape time in their lives, between all the
>> code contribution/bug fixing, and got this up and running. Kudos,
>>congrats
>> and a sincere thank you!
>> 
>> This is also a call to the community at large to help expand the
>> functional/automated test suites. Have a look in
>> <source_tree>/test/integration/ and peruse the README's.
>
>(There is a jenkins failure right now that is preventing the results
>from showing in graph. I'm looking into fixing that.)
>
>The tests now run according to the following schedule:
>
>KVM - 0030h, 1230h
>Xenserver - 0630h, 1830h
>
>The setup will configure Advanced Zone with either hypervisor in each
>of its runs.
>
>Since the module level fixtures are independent of each other. I will
>set them to execute concurrently.  Over the next week - I will add
>more test suites to this setup. Right now it runs 73 tests for build
>verification.
>
>Any tests that one checks-in to the repo will be automatically picked
>and run against the environments.
>
>I'm also writing down more detailed explanation of how this works, the
>scripts, puppet recipes, kickstart files and how we plan on expanding
>it. Hope in the near future we can replicate the same across other
>locations and convert the tests into our own version of the Chaos
>Monkey :)
>
>
>Thanks,
>-- 
>Prasanna.,

Re: Nightly tests

[Prasanna Santhanam <pr...@citrix.com>]

On Wed, Oct 31, 2012 at 11:14:18PM +0530, Ahmad Emneina wrote:
> I'd like to call peoples attention to
> http://jenkins.cloudstack.org/job/nightly-smoke-kvm/1/console, what were
> looking at here is the culmination of some seriously awesome work. In
> brief, this is an automated setup that deploys the latest cloudstack (in a
> vm) and provisions 'fresh' hypervisors running on bare metal. Then marvin
> configures cloudstack and drives the testing! This is all integrated with
> and runs as a jenkins job. The gang responsible for this, Rohit, Edison
> and Prasanna... managed to scrape time in their lives, between all the
> code contribution/bug fixing, and got this up and running. Kudos, congrats
> and a sincere thank you!
> 
> This is also a call to the community at large to help expand the
> functional/automated test suites. Have a look in
> <source_tree>/test/integration/ and peruse the README's.

(There is a jenkins failure right now that is preventing the results
from showing in graph. I'm looking into fixing that.)

The tests now run according to the following schedule:

KVM - 0030h, 1230h 
Xenserver - 0630h, 1830h

The setup will configure Advanced Zone with either hypervisor in each
of its runs.

Since the module level fixtures are independent of each other. I will
set them to execute concurrently.  Over the next week - I will add
more test suites to this setup. Right now it runs 73 tests for build
verification.

Any tests that one checks-in to the repo will be automatically picked
and run against the environments.

I'm also writing down more detailed explanation of how this works, the
scripts, puppet recipes, kickstart files and how we plan on expanding
it. Hope in the near future we can replicate the same across other
locations and convert the tests into our own version of the Chaos
Monkey :)


Thanks,
-- 
Prasanna.,