RE: Important Pending Items

[Paul Angus <pa...@shapeblue.com>]

Hi Will,

It will be 100% open source. The $dayjob has blocked the team from working on it recently. I'd like to get some of our ideas into the CloudStack wiki this week before I go on leave for a week.
Rohit has created a repo for us: https://github.com/shapeblue/Trillian
(which looks a bit sad at the moment - I hope to push our Ansible roles for building mgmt. servers and Marvin boxes into it this week also)

Internally we're planning a two week sprint on it starting 14th March. Having to dip in and out of this effort is making it very difficult to get anywhere

I think the biggest part of the work will be to get Marvin up to scratch, the log output from the tests is pretty bad.

We very much want this to be a community 'thing', and we're happy to work with everyone/anyone to get there.

Any ideas on the most effective way to have a number of people on this - I'm all ears.
In the meantime I'll try to make good on my promises to get our ideas visible to everyone...




Paul Angus
VP Technology   ,       ShapeBlue


t:      @cloudyangus<te...@cloudyangus>

e:      paul.angus@shapeblue.com<ma...@shapeblue.com>        |      w:      www.shapeblue.com<http://www.shapeblue.com>





-----Original Message-----
From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On Behalf Of Will Stevens
Sent: Monday, February 29, 2016 9:52 PM
To: dev@cloudstack.apache.org
Subject: Re: Important Pending Items

@Paul and @Bharat, thank you for the quality write-ups on this topic.

re:Paul's Details:
I share a lot of your thoughts on this. The hypervisor in hypervisor setup with the ability to test multiple environments is also core to my thinking. I can't find any details on Trillian, other than what is in this thread. Is it open source and freely available? This is being developed at ShapeBlue?

re:Bharat's Details:
This seems like it is the most likely candidate for being able to offer value in the short term. Thank you for the work documenting it and making the source available. I still need to review the code. This approach will take quite a bit of work get setup initially, but once setup, it looks like it will do the trick. By using only freely available software makes it more accessible than some options which have been suggested in the past.

My thoughts on the topic:
- I think it is important that the CI can test all the different hypervisors and any hardware integrations available in the environment.
- Ideally the CI tool would be easily distributed and installed at multiple sites. Every company who depends on ACS will have their own configuration, hardware and mission critical features. If the CI can be distributed (as independent installs), it will reduce the load on any specific site and it will enable companies who have specific hardware in their direct path to success the ability to validate their use case.
- I think a standardized CI output is also important. Ideally the summary of the CI run would be posted back to the PR thread. I think it is also important to push the detailed logs to a central location so others can review them. Ideally, links to the detailed logs would also be pushed to the PR thread.
- I think there should be multiple tiers of testing. Basic simulator testing and full runs. We may be able to setup simulator testing using a Docker container. I think @pdion891 has put some work into this that I may be able to build on.
- I think it is probably too ambitious to do CI on every PR before merge, but ideally we would We may have to only CI the master branch for now and revert any PR that causes it to fail.

I have been digging into this quite a bit today. I will continue working on this as it is very important to me for the 4.9 release. I would like to setup a CI environment locally as well to help validate and contribute to the effort. I have some hardware I can throw at this, so I will be working to get something setup to start testing ASAP.

If I can work off either (or both) of your work, I would appreciate it.

Cheers,

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_

On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bh...@citrix.com>
wrote:

> Hi Paul,
>
> The goals and functionality of the CI on which we are working are the
> same (please review the FS<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> trator+and+test+runner+to+enable+continuos+integration>
> ), The difference being the method of implementation. We are not using
> nested virtualisation to create test environments. The work on CI
> began long back but unfortunately due to other priorities we could not
> work on a continuous basis and bring it to completion.
>
> I agree that we should come to a consensus on what the community wants
> from the CI. IMHO the basic and immediate requirement would be to have
> a stable system to test the PRs and post results. Once we have this,
> we can talk about providing test setups to the community and next
> steps. I think the availability of hardware has been a big hurdle
> (even if we use virtualisation).
>
> Instead of working independently on the same thing let us work on this
> together. I will share what we have currently, we can use it, or parts
> of it (some of it is there at
> https://github.com/bvbharatk/cloudstack-automation, but it is not
> organised as in cannot be run out of the box). While virtualisation is
> good from the ease of orchestration and resource utilisation point of
> view, it has other problems like compatibility/dependencies issues.
> Also it might tie the CI to a particular Hypervisor like ESXi as you have pointed out.
>
> So let us discuss on how to proceed further, and get things rolling.
>
> Thanks,
> Bharat.
>
>
> On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com<mailto:
> paul.angus@shapeblue.com>> wrote:
>
> Hi Bharat,
>
> Early last week I opened a discussion thread around the aims of
> Trillian which we had begun working on as a CI/CD platform for the
> community hardware (as well as individual use). Please could you
> elaborate on where there is/isn't cross-over? In terms of
> functionality and goals rather than method. It would be great to get
> community consensus around what we want from our centralised CI/CD.
>
>
> Project: Trillian
> We have been working on PoC of a CI environment design which will:
> * Provide fast build or rebuild of environments for testing.
> * Enable multiple independent concurrent builds
> * Be available on-demand through automation or individual request.
> * Be capable of fully utilising all available hardware
> * Flexible enough to be used to build super-realistic development
> environments.
> We intend to contribute and maintain our work within the Apache repos.
> However, we are currently building the POC, figuring out the
> requirements (and quirks) of the individual pieces, before pushing
> something concrete for to the community to review.
>
> We envision that Trillian would cater for a number of use cases:
> 1. CloudStack community integration testing of master against multiple
> deployment scenarios (using ASF infra) 2. CloudStack community
> integration testing of PRs against multiple deployment scenarios
> (using ASF infra) 3. Organisations/individuals running the full suites
> of tests available in Marvin against any physical environment they
> have.
> 4. Organisations/individuals deploying and running the full suites of
> tests available in Marvin against virtualised infrastructures which
> can be deployed by Marvin.
> As we intend Trillian to test multiple environments concurrently, we
> use nested virtualization on ESXi hosts (our testing has shown that
> this is the only hypervisor which can support the nested
> virtualisation of all other hypervisors with reasonable performance).
> We use Ansible to deploy and configure all aspects of the build as
> this will greatly lower the barrier to entry for independent testers.
> We use CloudStack to provision the management server and virtualised
> (nested) hosts on the physical hosts. We are creating Ansible
> playbooks and roles which can:
> 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules
> - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute
> hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
> (inc. installing the relevant CloudStack agent where
> required)
> 3. Install required ACS packages on management server 4. Configure a
> zone (including adding the compute hosts) via Marvin.
> 5. Run the required Marvin tests.
> 6. Return the results
> We may need to propose enhancements to Marvin in order to sync the
> configuration of hosts with the configuration used by Marvin.
>
> Using virtualised test environments, we can have multiple test
> scenarios running concurrently. To do this we have found that it is
> necessary to create pools or ranges of VLANs and IP addresses and
> allocate them to environments. So for any given physical environment
> which will be used for testing in, we take the total range(s) of IPs
> and VLANs available and carve them into non-overlapping chunks
> suitable for concurrent use as mgmt, public and guest networks. These
> are stored in a MariaDB database. When a range is being used in a
> testing environment, that range is marked as 'inuse' in the database.
> When creating a test environment, Trillian looks in the database for
> the next available VLAN range, the next available public IP range and
> so on. The returned values are used to populate a Marvin cfg file
> which in turn will be used to both build the environment and when
> running the Marvin testing. When the virtualised infra is cleaned up,
> the database will be updated to reflect that the used ranges are available again.
> This initiative has only recently been started, and as stated earlier
> we are currently figuring out the requirements (and quirks) of the
> individual pieces and looking for the most suitable wrapper to glue it all together.
> Also I have found that Marvin requires a little work to make the
> output more meaningful/readable (especially in the case of errors and
> exceptions) and to make it a little more intelligent about the tests
> it can/can't run based on the chosen infrastructure components. I have
> also found unreachable or very slow ISO and template paths hardcoded
> into Marvin or individual tests.
> We plan to enhance tests to address these issues and also reduce
> runtimes where possible.
>
>
> <http://www.shapeblue.com/>
> Paul Angus
> VP Technology , ShapeBlue
>
>
> d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> | m: +44 7711 418784<tel:+44%207711%20418784>
>
> e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> www.shapeblue.com<http://www.shapeblue.com/>
>
> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
>
>
>
>
>
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated
> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> is a company incorporated in Brasil and is operated under license from
> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> Republic of South Africa and is traded under license from Shape Blue
> Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related
> companies. If you are not the intended recipient of this email, you
> must neither take any action based upon its contents, nor copy or show
> it to anyone. Please contact the sender if you believe you have received this email in error.
>
>
>
>
> -----Original Message-----
> From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> Sent: Thursday, February 11, 2016 9:57 AM
> To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> Subject: Re: Important Pending Items
>
> Hi Sebastien,
>
> As Raja said, we are actively working on it and we will share the code
> on github soon.
>
> Thanks,
> Bharat.
>
>
> > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
> <ma...@citrix.com>> wrote:
> >
> > Hi Sebastien,
> >
> > On item (3) - the BVTs have been running consistently (with same
> passrates) and haven't had the time to automate the process of
> reporting to an external site.
> > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> > working
> on getting it run automatically and consistently.
> > Hope to see it ready soon.
> >
> > Best,
> > Raja
> > -----Original Message-----
> > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > Sent: Wednesday, February 10, 2016 2:29 PM
> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > Subject: Important Pending Items
> >
> > Morning folks,
> >
> > We have several crucial pending items, that we need to resolve
> > before
> moving on:
> >
> > 1- We need an RM for master ( just saw some commits there that
> > should be
> reverted or merged properly in other branches).
> >
> > 2- We need to automate writing release notes, pushing/tagging new
> > docs
> when release come out and announcing releases on website. Currently
> neither
> 4.7 nor 4.8 have been announced.
> >
> > 3- CI is still almost inexistent
> >
> > -Sebastien
>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: Important Pending Items

[Will Stevens <wi...@gmail.com>]

I would love that.  Thank you.  I have done a very high level review of the
code and I like what I see.

Are my current assumptions correct?

- It uses KVM as its base hypervisor and all environments are spun up and
tested on it?
- It currently supports testing KVM and Xen?
- It currently supports both Basic and Advanced networking?

It looks like you have other things supported as well, like; OVS, NSX,
openvswitch, etc...

Looking forward to talking with you.  :)

Cheers,

On Tue, Mar 1, 2016 at 4:29 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:

> Hi Will,
>
> We used this to test hundreds of PRs:
> https://github.com/schubergphilis/MCT-shared/. I can talk you through it
> sometime next week or so if you want.
>
> Regards,
> Remi
>
>
> On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will Stevens" <
> williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
>
> >Thanks Paul.  Great to hear you are planning to make everything available
> >and you are willing to take contributions from the community.
> >
> >I am currently trying to get up to speed on everything that has been done
> >and all the different pieces in play.  I would like to have a CI in place
> >(even if it is only locally on my hardware) for testing PRs for 4.9.  My
> >focus right now is solving the CI problem, so I am interested in any
> >developments on this front.
> >
> >We have Travis doing smoke tests on every PR right now.  Travis is free
> for
> >open source projects.  Is there a reason we are not doing more extensive
> >tests with Travis since it is already integrated into github?  Like maybe
> >running the simulator on every PR?  I think this could be a good first
> step
> >to weed out some of the PRs that are likely to fail a full CI run.
> >
> >Cheers,
> >
> >*Will STEVENS*
> >Lead Developer
> >
> >*CloudOps* *| *Cloud Solutions Experts
> >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >w cloudops.com *|* tw @CloudOps_
> >
> >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
> >
> >> Hi Will,
> >>
> >> It will be 100% open source. The $dayjob has blocked the team from
> working
> >> on it recently. I'd like to get some of our ideas into the CloudStack
> wiki
> >> this week before I go on leave for a week.
> >> Rohit has created a repo for us: https://github.com/shapeblue/Trillian
> >> (which looks a bit sad at the moment - I hope to push our Ansible roles
> >> for building mgmt. servers and Marvin boxes into it this week also)
> >>
> >> Internally we're planning a two week sprint on it starting 14th March.
> >> Having to dip in and out of this effort is making it very difficult to
> get
> >> anywhere
> >>
> >> I think the biggest part of the work will be to get Marvin up to
> scratch,
> >> the log output from the tests is pretty bad.
> >>
> >> We very much want this to be a community 'thing', and we're happy to
> work
> >> with everyone/anyone to get there.
> >>
> >> Any ideas on the most effective way to have a number of people on this -
> >> I'm all ears.
> >> In the meantime I'll try to make good on my promises to get our ideas
> >> visible to everyone...
> >>
> >>
> >>
> >>
> >> Paul Angus
> >> VP Technology   ,       ShapeBlue
> >>
> >>
> >> t:      @cloudyangus<te...@cloudyangus>
> >>
> >> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
> >> |      w:      www.shapeblue.com<http://www.shapeblue.com>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> >> Behalf Of Will Stevens
> >> Sent: Monday, February 29, 2016 9:52 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Important Pending Items
> >>
> >> @Paul and @Bharat, thank you for the quality write-ups on this topic.
> >>
> >> re:Paul's Details:
> >> I share a lot of your thoughts on this. The hypervisor in hypervisor
> setup
> >> with the ability to test multiple environments is also core to my
> thinking.
> >> I can't find any details on Trillian, other than what is in this
> thread. Is
> >> it open source and freely available? This is being developed at
> ShapeBlue?
> >>
> >> re:Bharat's Details:
> >> This seems like it is the most likely candidate for being able to offer
> >> value in the short term. Thank you for the work documenting it and
> making
> >> the source available. I still need to review the code. This approach
> will
> >> take quite a bit of work get setup initially, but once setup, it looks
> like
> >> it will do the trick. By using only freely available software makes it
> more
> >> accessible than some options which have been suggested in the past.
> >>
> >> My thoughts on the topic:
> >> - I think it is important that the CI can test all the different
> >> hypervisors and any hardware integrations available in the environment.
> >> - Ideally the CI tool would be easily distributed and installed at
> >> multiple sites. Every company who depends on ACS will have their own
> >> configuration, hardware and mission critical features. If the CI can be
> >> distributed (as independent installs), it will reduce the load on any
> >> specific site and it will enable companies who have specific hardware in
> >> their direct path to success the ability to validate their use case.
> >> - I think a standardized CI output is also important. Ideally the
> summary
> >> of the CI run would be posted back to the PR thread. I think it is also
> >> important to push the detailed logs to a central location so others can
> >> review them. Ideally, links to the detailed logs would also be pushed to
> >> the PR thread.
> >> - I think there should be multiple tiers of testing. Basic simulator
> >> testing and full runs. We may be able to setup simulator testing using a
> >> Docker container. I think @pdion891 has put some work into this that I
> may
> >> be able to build on.
> >> - I think it is probably too ambitious to do CI on every PR before
> merge,
> >> but ideally we would We may have to only CI the master branch for now
> and
> >> revert any PR that causes it to fail.
> >>
> >> I have been digging into this quite a bit today. I will continue working
> >> on this as it is very important to me for the 4.9 release. I would like
> to
> >> setup a CI environment locally as well to help validate and contribute
> to
> >> the effort. I have some hardware I can throw at this, so I will be
> working
> >> to get something setup to start testing ASAP.
> >>
> >> If I can work off either (or both) of your work, I would appreciate it.
> >>
> >> Cheers,
> >>
> >> *Will STEVENS*
> >> Lead Developer
> >>
> >> *CloudOps* *| *Cloud Solutions Experts
> >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> >> @CloudOps_
> >>
> >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bharat.kumar@citrix.com
> >
> >> wrote:
> >>
> >> > Hi Paul,
> >> >
> >> > The goals and functionality of the CI on which we are working are the
> >> > same (please review the FS<
> >> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> >> > trator+and+test+runner+to+enable+continuos+integration>
> >> > ), The difference being the method of implementation. We are not using
> >> > nested virtualisation to create test environments. The work on CI
> >> > began long back but unfortunately due to other priorities we could not
> >> > work on a continuous basis and bring it to completion.
> >> >
> >> > I agree that we should come to a consensus on what the community wants
> >> > from the CI. IMHO the basic and immediate requirement would be to have
> >> > a stable system to test the PRs and post results. Once we have this,
> >> > we can talk about providing test setups to the community and next
> >> > steps. I think the availability of hardware has been a big hurdle
> >> > (even if we use virtualisation).
> >> >
> >> > Instead of working independently on the same thing let us work on this
> >> > together. I will share what we have currently, we can use it, or parts
> >> > of it (some of it is there at
> >> > https://github.com/bvbharatk/cloudstack-automation, but it is not
> >> > organised as in cannot be run out of the box). While virtualisation is
> >> > good from the ease of orchestration and resource utilisation point of
> >> > view, it has other problems like compatibility/dependencies issues.
> >> > Also it might tie the CI to a particular Hypervisor like ESXi as you
> >> have pointed out.
> >> >
> >> > So let us discuss on how to proceed further, and get things rolling.
> >> >
> >> > Thanks,
> >> > Bharat.
> >> >
> >> >
> >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com
> <mailto:
> >> > paul.angus@shapeblue.com>> wrote:
> >> >
> >> > Hi Bharat,
> >> >
> >> > Early last week I opened a discussion thread around the aims of
> >> > Trillian which we had begun working on as a CI/CD platform for the
> >> > community hardware (as well as individual use). Please could you
> >> > elaborate on where there is/isn't cross-over? In terms of
> >> > functionality and goals rather than method. It would be great to get
> >> > community consensus around what we want from our centralised CI/CD.
> >> >
> >> >
> >> > Project: Trillian
> >> > We have been working on PoC of a CI environment design which will:
> >> > * Provide fast build or rebuild of environments for testing.
> >> > * Enable multiple independent concurrent builds
> >> > * Be available on-demand through automation or individual request.
> >> > * Be capable of fully utilising all available hardware
> >> > * Flexible enough to be used to build super-realistic development
> >> > environments.
> >> > We intend to contribute and maintain our work within the Apache repos.
> >> > However, we are currently building the POC, figuring out the
> >> > requirements (and quirks) of the individual pieces, before pushing
> >> > something concrete for to the community to review.
> >> >
> >> > We envision that Trillian would cater for a number of use cases:
> >> > 1. CloudStack community integration testing of master against multiple
> >> > deployment scenarios (using ASF infra) 2. CloudStack community
> >> > integration testing of PRs against multiple deployment scenarios
> >> > (using ASF infra) 3. Organisations/individuals running the full suites
> >> > of tests available in Marvin against any physical environment they
> >> > have.
> >> > 4. Organisations/individuals deploying and running the full suites of
> >> > tests available in Marvin against virtualised infrastructures which
> >> > can be deployed by Marvin.
> >> > As we intend Trillian to test multiple environments concurrently, we
> >> > use nested virtualization on ESXi hosts (our testing has shown that
> >> > this is the only hypervisor which can support the nested
> >> > virtualisation of all other hypervisors with reasonable performance).
> >> > We use Ansible to deploy and configure all aspects of the build as
> >> > this will greatly lower the barrier to entry for independent testers.
> >> > We use CloudStack to provision the management server and virtualised
> >> > (nested) hosts on the physical hosts. We are creating Ansible
> >> > playbooks and roles which can:
> >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules
> >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute
> >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
> >> > (inc. installing the relevant CloudStack agent where
> >> > required)
> >> > 3. Install required ACS packages on management server 4. Configure a
> >> > zone (including adding the compute hosts) via Marvin.
> >> > 5. Run the required Marvin tests.
> >> > 6. Return the results
> >> > We may need to propose enhancements to Marvin in order to sync the
> >> > configuration of hosts with the configuration used by Marvin.
> >> >
> >> > Using virtualised test environments, we can have multiple test
> >> > scenarios running concurrently. To do this we have found that it is
> >> > necessary to create pools or ranges of VLANs and IP addresses and
> >> > allocate them to environments. So for any given physical environment
> >> > which will be used for testing in, we take the total range(s) of IPs
> >> > and VLANs available and carve them into non-overlapping chunks
> >> > suitable for concurrent use as mgmt, public and guest networks. These
> >> > are stored in a MariaDB database. When a range is being used in a
> >> > testing environment, that range is marked as 'inuse' in the database.
> >> > When creating a test environment, Trillian looks in the database for
> >> > the next available VLAN range, the next available public IP range and
> >> > so on. The returned values are used to populate a Marvin cfg file
> >> > which in turn will be used to both build the environment and when
> >> > running the Marvin testing. When the virtualised infra is cleaned up,
> >> > the database will be updated to reflect that the used ranges are
> >> available again.
> >> > This initiative has only recently been started, and as stated earlier
> >> > we are currently figuring out the requirements (and quirks) of the
> >> > individual pieces and looking for the most suitable wrapper to glue it
> >> all together.
> >> > Also I have found that Marvin requires a little work to make the
> >> > output more meaningful/readable (especially in the case of errors and
> >> > exceptions) and to make it a little more intelligent about the tests
> >> > it can/can't run based on the chosen infrastructure components. I have
> >> > also found unreachable or very slow ISO and template paths hardcoded
> >> > into Marvin or individual tests.
> >> > We plan to enhance tests to address these issues and also reduce
> >> > runtimes where possible.
> >> >
> >> >
> >> > <http://www.shapeblue.com/>
> >> > Paul Angus
> >> > VP Technology , ShapeBlue
> >> >
> >> >
> >> > d: +44 203 617 0528 | s: +44 203 603 0540
> >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> >> > | m: +44 7711 418784<tel:+44%207711%20418784>
> >> >
> >> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> >> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> >> > www.shapeblue.com<http://www.shapeblue.com/>
> >> >
> >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> >> > Services India LLP is a company incorporated in India and is operated
> >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> >> > is a company incorporated in Brasil and is operated under license from
> >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> >> > Republic of South Africa and is traded under license from Shape Blue
> >> > Ltd. ShapeBlue is a registered trademark.
> >> > This email and any attachments to it may be confidential and are
> >> > intended solely for the use of the individual to whom it is addressed.
> >> > Any views or opinions expressed are solely those of the author and do
> >> > not necessarily represent those of Shape Blue Ltd or related
> >> > companies. If you are not the intended recipient of this email, you
> >> > must neither take any action based upon its contents, nor copy or show
> >> > it to anyone. Please contact the sender if you believe you have
> received
> >> this email in error.
> >> >
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> >> > Sent: Thursday, February 11, 2016 9:57 AM
> >> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >> > Subject: Re: Important Pending Items
> >> >
> >> > Hi Sebastien,
> >> >
> >> > As Raja said, we are actively working on it and we will share the code
> >> > on github soon.
> >> >
> >> > Thanks,
> >> > Bharat.
> >> >
> >> >
> >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
> >> > <ma...@citrix.com>> wrote:
> >> > >
> >> > > Hi Sebastien,
> >> > >
> >> > > On item (3) - the BVTs have been running consistently (with same
> >> > passrates) and haven't had the time to automate the process of
> >> > reporting to an external site.
> >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> >> > > working
> >> > on getting it run automatically and consistently.
> >> > > Hope to see it ready soon.
> >> > >
> >> > > Best,
> >> > > Raja
> >> > > -----Original Message-----
> >> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> > > Sent: Wednesday, February 10, 2016 2:29 PM
> >> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >> > > Subject: Important Pending Items
> >> > >
> >> > > Morning folks,
> >> > >
> >> > > We have several crucial pending items, that we need to resolve
> >> > > before
> >> > moving on:
> >> > >
> >> > > 1- We need an RM for master ( just saw some commits there that
> >> > > should be
> >> > reverted or merged properly in other branches).
> >> > >
> >> > > 2- We need to automate writing release notes, pushing/tagging new
> >> > > docs
> >> > when release come out and announcing releases on website. Currently
> >> > neither
> >> > 4.7 nor 4.8 have been announced.
> >> > >
> >> > > 3- CI is still almost inexistent
> >> > >
> >> > > -Sebastien
> >> >
> >> > Find out more about ShapeBlue and our range of CloudStack related
> >> services:
> >> > IaaS Cloud Design & Build<
> >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> >> > IaaS deployment framework<http://shapeblue.com/csforge/>
> >> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> >> > CloudStack Software Engineering<
> >> > http://shapeblue.com/cloudstack-software-engineering/>
> >> > CloudStack Infrastructure Support<
> >> > http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> >> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> >> >
> >> >
> >> Find out more about ShapeBlue and our range of CloudStack related
> services:
> >> IaaS Cloud Design & Build<
> >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> >> IaaS deployment framework<http://shapeblue.com/csforge/>
> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> >> CloudStack Software Engineering<
> >> http://shapeblue.com/cloudstack-software-engineering/>
> >> CloudStack Infrastructure Support<
> >> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> >>
>

Re: Important Pending Items

[Will Stevens <ws...@cloudops.com>]

Thanks Sanjeev, I did not realize that the whole simulator was tested.  Ok
that is good...  :)

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Wed, Mar 2, 2016 at 12:15 AM, Sanjeev N <sa...@apache.org> wrote:

> Hi Steve,
>
> Currently travis runs all simulator tests(marvin tests which does not
> require any hardware, tagged with requierd_hardware=false) on every PR. If
> any test fails it updates the PR and marks it red. IMO apart from 2 LGTMs
> travis test pass is also mandatory for PR merge.
>
> Thanks,
> Sanjeev N
> Chief Product Engineer@Accelerite
>
> On Wed, Mar 2, 2016 at 3:13 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Awesome, thanks Remi.  I will review this.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Mar 1, 2016 at 4:29 PM, Remi Bergsma <
> RBergsma@schubergphilis.com>
> > wrote:
> >
> > > Hi Will,
> > >
> > > We used this to test hundreds of PRs:
> > > https://github.com/schubergphilis/MCT-shared/. I can talk you through
> it
> > > sometime next week or so if you want.
> > >
> > > Regards,
> > > Remi
> > >
> > >
> > > On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will
> Stevens"
> > <
> > > williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> > >
> > > >Thanks Paul.  Great to hear you are planning to make everything
> > available
> > > >and you are willing to take contributions from the community.
> > > >
> > > >I am currently trying to get up to speed on everything that has been
> > done
> > > >and all the different pieces in play.  I would like to have a CI in
> > place
> > > >(even if it is only locally on my hardware) for testing PRs for 4.9.
> My
> > > >focus right now is solving the CI problem, so I am interested in any
> > > >developments on this front.
> > > >
> > > >We have Travis doing smoke tests on every PR right now.  Travis is
> free
> > > for
> > > >open source projects.  Is there a reason we are not doing more
> extensive
> > > >tests with Travis since it is already integrated into github?  Like
> > maybe
> > > >running the simulator on every PR?  I think this could be a good first
> > > step
> > > >to weed out some of the PRs that are likely to fail a full CI run.
> > > >
> > > >Cheers,
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > >*CloudOps* *| *Cloud Solutions Experts
> > > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > >w cloudops.com *|* tw @CloudOps_
> > > >
> > > >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com>
> > > wrote:
> > > >
> > > >> Hi Will,
> > > >>
> > > >> It will be 100% open source. The $dayjob has blocked the team from
> > > working
> > > >> on it recently. I'd like to get some of our ideas into the
> CloudStack
> > > wiki
> > > >> this week before I go on leave for a week.
> > > >> Rohit has created a repo for us:
> > https://github.com/shapeblue/Trillian
> > > >> (which looks a bit sad at the moment - I hope to push our Ansible
> > roles
> > > >> for building mgmt. servers and Marvin boxes into it this week also)
> > > >>
> > > >> Internally we're planning a two week sprint on it starting 14th
> March.
> > > >> Having to dip in and out of this effort is making it very difficult
> to
> > > get
> > > >> anywhere
> > > >>
> > > >> I think the biggest part of the work will be to get Marvin up to
> > > scratch,
> > > >> the log output from the tests is pretty bad.
> > > >>
> > > >> We very much want this to be a community 'thing', and we're happy to
> > > work
> > > >> with everyone/anyone to get there.
> > > >>
> > > >> Any ideas on the most effective way to have a number of people on
> > this -
> > > >> I'm all ears.
> > > >> In the meantime I'll try to make good on my promises to get our
> ideas
> > > >> visible to everyone...
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Paul Angus
> > > >> VP Technology   ,       ShapeBlue
> > > >>
> > > >>
> > > >> t:      @cloudyangus<te...@cloudyangus>
> > > >>
> > > >> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
> > > >> |      w:      www.shapeblue.com<http://www.shapeblue.com>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > > >> Behalf Of Will Stevens
> > > >> Sent: Monday, February 29, 2016 9:52 PM
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: Re: Important Pending Items
> > > >>
> > > >> @Paul and @Bharat, thank you for the quality write-ups on this
> topic.
> > > >>
> > > >> re:Paul's Details:
> > > >> I share a lot of your thoughts on this. The hypervisor in hypervisor
> > > setup
> > > >> with the ability to test multiple environments is also core to my
> > > thinking.
> > > >> I can't find any details on Trillian, other than what is in this
> > > thread. Is
> > > >> it open source and freely available? This is being developed at
> > > ShapeBlue?
> > > >>
> > > >> re:Bharat's Details:
> > > >> This seems like it is the most likely candidate for being able to
> > offer
> > > >> value in the short term. Thank you for the work documenting it and
> > > making
> > > >> the source available. I still need to review the code. This approach
> > > will
> > > >> take quite a bit of work get setup initially, but once setup, it
> looks
> > > like
> > > >> it will do the trick. By using only freely available software makes
> it
> > > more
> > > >> accessible than some options which have been suggested in the past.
> > > >>
> > > >> My thoughts on the topic:
> > > >> - I think it is important that the CI can test all the different
> > > >> hypervisors and any hardware integrations available in the
> > environment.
> > > >> - Ideally the CI tool would be easily distributed and installed at
> > > >> multiple sites. Every company who depends on ACS will have their own
> > > >> configuration, hardware and mission critical features. If the CI can
> > be
> > > >> distributed (as independent installs), it will reduce the load on
> any
> > > >> specific site and it will enable companies who have specific
> hardware
> > in
> > > >> their direct path to success the ability to validate their use case.
> > > >> - I think a standardized CI output is also important. Ideally the
> > > summary
> > > >> of the CI run would be posted back to the PR thread. I think it is
> > also
> > > >> important to push the detailed logs to a central location so others
> > can
> > > >> review them. Ideally, links to the detailed logs would also be
> pushed
> > to
> > > >> the PR thread.
> > > >> - I think there should be multiple tiers of testing. Basic simulator
> > > >> testing and full runs. We may be able to setup simulator testing
> > using a
> > > >> Docker container. I think @pdion891 has put some work into this
> that I
> > > may
> > > >> be able to build on.
> > > >> - I think it is probably too ambitious to do CI on every PR before
> > > merge,
> > > >> but ideally we would We may have to only CI the master branch for
> now
> > > and
> > > >> revert any PR that causes it to fail.
> > > >>
> > > >> I have been digging into this quite a bit today. I will continue
> > working
> > > >> on this as it is very important to me for the 4.9 release. I would
> > like
> > > to
> > > >> setup a CI environment locally as well to help validate and
> contribute
> > > to
> > > >> the effort. I have some hardware I can throw at this, so I will be
> > > working
> > > >> to get something setup to start testing ASAP.
> > > >>
> > > >> If I can work off either (or both) of your work, I would appreciate
> > it.
> > > >>
> > > >> Cheers,
> > > >>
> > > >> *Will STEVENS*
> > > >> Lead Developer
> > > >>
> > > >> *CloudOps* *| *Cloud Solutions Experts
> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|*
> tw
> > > >> @CloudOps_
> > > >>
> > > >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <
> > bharat.kumar@citrix.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Hi Paul,
> > > >> >
> > > >> > The goals and functionality of the CI on which we are working are
> > the
> > > >> > same (please review the FS<
> > > >> >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> > > >> > trator+and+test+runner+to+enable+continuos+integration>
> > > >> > ), The difference being the method of implementation. We are not
> > using
> > > >> > nested virtualisation to create test environments. The work on CI
> > > >> > began long back but unfortunately due to other priorities we could
> > not
> > > >> > work on a continuous basis and bring it to completion.
> > > >> >
> > > >> > I agree that we should come to a consensus on what the community
> > wants
> > > >> > from the CI. IMHO the basic and immediate requirement would be to
> > have
> > > >> > a stable system to test the PRs and post results. Once we have
> this,
> > > >> > we can talk about providing test setups to the community and next
> > > >> > steps. I think the availability of hardware has been a big hurdle
> > > >> > (even if we use virtualisation).
> > > >> >
> > > >> > Instead of working independently on the same thing let us work on
> > this
> > > >> > together. I will share what we have currently, we can use it, or
> > parts
> > > >> > of it (some of it is there at
> > > >> > https://github.com/bvbharatk/cloudstack-automation, but it is not
> > > >> > organised as in cannot be run out of the box). While
> virtualisation
> > is
> > > >> > good from the ease of orchestration and resource utilisation point
> > of
> > > >> > view, it has other problems like compatibility/dependencies
> issues.
> > > >> > Also it might tie the CI to a particular Hypervisor like ESXi as
> you
> > > >> have pointed out.
> > > >> >
> > > >> > So let us discuss on how to proceed further, and get things
> rolling.
> > > >> >
> > > >> > Thanks,
> > > >> > Bharat.
> > > >> >
> > > >> >
> > > >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com
> > > <mailto:
> > > >> > paul.angus@shapeblue.com>> wrote:
> > > >> >
> > > >> > Hi Bharat,
> > > >> >
> > > >> > Early last week I opened a discussion thread around the aims of
> > > >> > Trillian which we had begun working on as a CI/CD platform for the
> > > >> > community hardware (as well as individual use). Please could you
> > > >> > elaborate on where there is/isn't cross-over? In terms of
> > > >> > functionality and goals rather than method. It would be great to
> get
> > > >> > community consensus around what we want from our centralised
> CI/CD.
> > > >> >
> > > >> >
> > > >> > Project: Trillian
> > > >> > We have been working on PoC of a CI environment design which will:
> > > >> > * Provide fast build or rebuild of environments for testing.
> > > >> > * Enable multiple independent concurrent builds
> > > >> > * Be available on-demand through automation or individual request.
> > > >> > * Be capable of fully utilising all available hardware
> > > >> > * Flexible enough to be used to build super-realistic development
> > > >> > environments.
> > > >> > We intend to contribute and maintain our work within the Apache
> > repos.
> > > >> > However, we are currently building the POC, figuring out the
> > > >> > requirements (and quirks) of the individual pieces, before pushing
> > > >> > something concrete for to the community to review.
> > > >> >
> > > >> > We envision that Trillian would cater for a number of use cases:
> > > >> > 1. CloudStack community integration testing of master against
> > multiple
> > > >> > deployment scenarios (using ASF infra) 2. CloudStack community
> > > >> > integration testing of PRs against multiple deployment scenarios
> > > >> > (using ASF infra) 3. Organisations/individuals running the full
> > suites
> > > >> > of tests available in Marvin against any physical environment they
> > > >> > have.
> > > >> > 4. Organisations/individuals deploying and running the full suites
> > of
> > > >> > tests available in Marvin against virtualised infrastructures
> which
> > > >> > can be deployed by Marvin.
> > > >> > As we intend Trillian to test multiple environments concurrently,
> we
> > > >> > use nested virtualization on ESXi hosts (our testing has shown
> that
> > > >> > this is the only hypervisor which can support the nested
> > > >> > virtualisation of all other hypervisors with reasonable
> > performance).
> > > >> > We use Ansible to deploy and configure all aspects of the build as
> > > >> > this will greatly lower the barrier to entry for independent
> > testers.
> > > >> > We use CloudStack to provision the management server and
> virtualised
> > > >> > (nested) hosts on the physical hosts. We are creating Ansible
> > > >> > playbooks and roles which can:
> > > >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack
> > modules
> > > >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of
> > compute
> > > >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure
> hosts
> > > >> > (inc. installing the relevant CloudStack agent where
> > > >> > required)
> > > >> > 3. Install required ACS packages on management server 4.
> Configure a
> > > >> > zone (including adding the compute hosts) via Marvin.
> > > >> > 5. Run the required Marvin tests.
> > > >> > 6. Return the results
> > > >> > We may need to propose enhancements to Marvin in order to sync the
> > > >> > configuration of hosts with the configuration used by Marvin.
> > > >> >
> > > >> > Using virtualised test environments, we can have multiple test
> > > >> > scenarios running concurrently. To do this we have found that it
> is
> > > >> > necessary to create pools or ranges of VLANs and IP addresses and
> > > >> > allocate them to environments. So for any given physical
> environment
> > > >> > which will be used for testing in, we take the total range(s) of
> IPs
> > > >> > and VLANs available and carve them into non-overlapping chunks
> > > >> > suitable for concurrent use as mgmt, public and guest networks.
> > These
> > > >> > are stored in a MariaDB database. When a range is being used in a
> > > >> > testing environment, that range is marked as 'inuse' in the
> > database.
> > > >> > When creating a test environment, Trillian looks in the database
> for
> > > >> > the next available VLAN range, the next available public IP range
> > and
> > > >> > so on. The returned values are used to populate a Marvin cfg file
> > > >> > which in turn will be used to both build the environment and when
> > > >> > running the Marvin testing. When the virtualised infra is cleaned
> > up,
> > > >> > the database will be updated to reflect that the used ranges are
> > > >> available again.
> > > >> > This initiative has only recently been started, and as stated
> > earlier
> > > >> > we are currently figuring out the requirements (and quirks) of the
> > > >> > individual pieces and looking for the most suitable wrapper to
> glue
> > it
> > > >> all together.
> > > >> > Also I have found that Marvin requires a little work to make the
> > > >> > output more meaningful/readable (especially in the case of errors
> > and
> > > >> > exceptions) and to make it a little more intelligent about the
> tests
> > > >> > it can/can't run based on the chosen infrastructure components. I
> > have
> > > >> > also found unreachable or very slow ISO and template paths
> hardcoded
> > > >> > into Marvin or individual tests.
> > > >> > We plan to enhance tests to address these issues and also reduce
> > > >> > runtimes where possible.
> > > >> >
> > > >> >
> > > >> > <http://www.shapeblue.com/>
> > > >> > Paul Angus
> > > >> > VP Technology , ShapeBlue
> > > >> >
> > > >> >
> > > >> > d: +44 203 617 0528 | s: +44 203 603 0540
> > > >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> > > >> > | m: +44 7711 418784<tel:+44%207711%20418784>
> > > >> >
> > > >> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> > > >> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> > > >> > www.shapeblue.com<http://www.shapeblue.com/>
> > > >> >
> > > >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Shape Blue Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > >> > Services India LLP is a company incorporated in India and is
> > operated
> > > >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > Ltda
> > > >> > is a company incorporated in Brasil and is operated under license
> > from
> > > >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
> The
> > > >> > Republic of South Africa and is traded under license from Shape
> Blue
> > > >> > Ltd. ShapeBlue is a registered trademark.
> > > >> > This email and any attachments to it may be confidential and are
> > > >> > intended solely for the use of the individual to whom it is
> > addressed.
> > > >> > Any views or opinions expressed are solely those of the author and
> > do
> > > >> > not necessarily represent those of Shape Blue Ltd or related
> > > >> > companies. If you are not the intended recipient of this email,
> you
> > > >> > must neither take any action based upon its contents, nor copy or
> > show
> > > >> > it to anyone. Please contact the sender if you believe you have
> > > received
> > > >> this email in error.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> > > >> > Sent: Thursday, February 11, 2016 9:57 AM
> > > >> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > > >> > Subject: Re: Important Pending Items
> > > >> >
> > > >> > Hi Sebastien,
> > > >> >
> > > >> > As Raja said, we are actively working on it and we will share the
> > code
> > > >> > on github soon.
> > > >> >
> > > >> > Thanks,
> > > >> > Bharat.
> > > >> >
> > > >> >
> > > >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <
> raja.pullela@citrix.com
> > > >> > <ma...@citrix.com>> wrote:
> > > >> > >
> > > >> > > Hi Sebastien,
> > > >> > >
> > > >> > > On item (3) - the BVTs have been running consistently (with same
> > > >> > passrates) and haven't had the time to automate the process of
> > > >> > reporting to an external site.
> > > >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> > > >> > > working
> > > >> > on getting it run automatically and consistently.
> > > >> > > Hope to see it ready soon.
> > > >> > >
> > > >> > > Best,
> > > >> > > Raja
> > > >> > > -----Original Message-----
> > > >> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > > >> > > Sent: Wednesday, February 10, 2016 2:29 PM
> > > >> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > > >> > > Subject: Important Pending Items
> > > >> > >
> > > >> > > Morning folks,
> > > >> > >
> > > >> > > We have several crucial pending items, that we need to resolve
> > > >> > > before
> > > >> > moving on:
> > > >> > >
> > > >> > > 1- We need an RM for master ( just saw some commits there that
> > > >> > > should be
> > > >> > reverted or merged properly in other branches).
> > > >> > >
> > > >> > > 2- We need to automate writing release notes, pushing/tagging
> new
> > > >> > > docs
> > > >> > when release come out and announcing releases on website.
> Currently
> > > >> > neither
> > > >> > 4.7 nor 4.8 have been announced.
> > > >> > >
> > > >> > > 3- CI is still almost inexistent
> > > >> > >
> > > >> > > -Sebastien
> > > >> >
> > > >> > Find out more about ShapeBlue and our range of CloudStack related
> > > >> services:
> > > >> > IaaS Cloud Design & Build<
> > > >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > rapid
> > > >> > IaaS deployment framework<http://shapeblue.com/csforge/>
> > > >> > CloudStack Consulting<
> http://shapeblue.com/cloudstack-consultancy/>
> > |
> > > >> > CloudStack Software Engineering<
> > > >> > http://shapeblue.com/cloudstack-software-engineering/>
> > > >> > CloudStack Infrastructure Support<
> > > >> > http://shapeblue.com/cloudstack-infrastructure-support/> |
> > CloudStack
> > > >> > Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/
> > >
> > > >> >
> > > >> >
> > > >> Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > >> IaaS Cloud Design & Build<
> > > >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > > >> IaaS deployment framework<http://shapeblue.com/csforge/>
> > > >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> |
> > > >> CloudStack Software Engineering<
> > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > >> CloudStack Infrastructure Support<
> > > >> http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > > >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/
> >
> > > >>
> > >
> >
>

Re: Important Pending Items

[Sanjeev N <sa...@apache.org>]

Hi Steve,

Currently travis runs all simulator tests(marvin tests which does not
require any hardware, tagged with requierd_hardware=false) on every PR. If
any test fails it updates the PR and marks it red. IMO apart from 2 LGTMs
travis test pass is also mandatory for PR merge.

Thanks,
Sanjeev N
Chief Product Engineer@Accelerite

On Wed, Mar 2, 2016 at 3:13 AM, Will Stevens <ws...@cloudops.com> wrote:

> Awesome, thanks Remi.  I will review this.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, Mar 1, 2016 at 4:29 PM, Remi Bergsma <RB...@schubergphilis.com>
> wrote:
>
> > Hi Will,
> >
> > We used this to test hundreds of PRs:
> > https://github.com/schubergphilis/MCT-shared/. I can talk you through it
> > sometime next week or so if you want.
> >
> > Regards,
> > Remi
> >
> >
> > On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will Stevens"
> <
> > williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
> >
> > >Thanks Paul.  Great to hear you are planning to make everything
> available
> > >and you are willing to take contributions from the community.
> > >
> > >I am currently trying to get up to speed on everything that has been
> done
> > >and all the different pieces in play.  I would like to have a CI in
> place
> > >(even if it is only locally on my hardware) for testing PRs for 4.9.  My
> > >focus right now is solving the CI problem, so I am interested in any
> > >developments on this front.
> > >
> > >We have Travis doing smoke tests on every PR right now.  Travis is free
> > for
> > >open source projects.  Is there a reason we are not doing more extensive
> > >tests with Travis since it is already integrated into github?  Like
> maybe
> > >running the simulator on every PR?  I think this could be a good first
> > step
> > >to weed out some of the PRs that are likely to fail a full CI run.
> > >
> > >Cheers,
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > >*CloudOps* *| *Cloud Solutions Experts
> > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > >w cloudops.com *|* tw @CloudOps_
> > >
> > >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> > >
> > >> Hi Will,
> > >>
> > >> It will be 100% open source. The $dayjob has blocked the team from
> > working
> > >> on it recently. I'd like to get some of our ideas into the CloudStack
> > wiki
> > >> this week before I go on leave for a week.
> > >> Rohit has created a repo for us:
> https://github.com/shapeblue/Trillian
> > >> (which looks a bit sad at the moment - I hope to push our Ansible
> roles
> > >> for building mgmt. servers and Marvin boxes into it this week also)
> > >>
> > >> Internally we're planning a two week sprint on it starting 14th March.
> > >> Having to dip in and out of this effort is making it very difficult to
> > get
> > >> anywhere
> > >>
> > >> I think the biggest part of the work will be to get Marvin up to
> > scratch,
> > >> the log output from the tests is pretty bad.
> > >>
> > >> We very much want this to be a community 'thing', and we're happy to
> > work
> > >> with everyone/anyone to get there.
> > >>
> > >> Any ideas on the most effective way to have a number of people on
> this -
> > >> I'm all ears.
> > >> In the meantime I'll try to make good on my promises to get our ideas
> > >> visible to everyone...
> > >>
> > >>
> > >>
> > >>
> > >> Paul Angus
> > >> VP Technology   ,       ShapeBlue
> > >>
> > >>
> > >> t:      @cloudyangus<te...@cloudyangus>
> > >>
> > >> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
> > >> |      w:      www.shapeblue.com<http://www.shapeblue.com>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> > >> Behalf Of Will Stevens
> > >> Sent: Monday, February 29, 2016 9:52 PM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: Re: Important Pending Items
> > >>
> > >> @Paul and @Bharat, thank you for the quality write-ups on this topic.
> > >>
> > >> re:Paul's Details:
> > >> I share a lot of your thoughts on this. The hypervisor in hypervisor
> > setup
> > >> with the ability to test multiple environments is also core to my
> > thinking.
> > >> I can't find any details on Trillian, other than what is in this
> > thread. Is
> > >> it open source and freely available? This is being developed at
> > ShapeBlue?
> > >>
> > >> re:Bharat's Details:
> > >> This seems like it is the most likely candidate for being able to
> offer
> > >> value in the short term. Thank you for the work documenting it and
> > making
> > >> the source available. I still need to review the code. This approach
> > will
> > >> take quite a bit of work get setup initially, but once setup, it looks
> > like
> > >> it will do the trick. By using only freely available software makes it
> > more
> > >> accessible than some options which have been suggested in the past.
> > >>
> > >> My thoughts on the topic:
> > >> - I think it is important that the CI can test all the different
> > >> hypervisors and any hardware integrations available in the
> environment.
> > >> - Ideally the CI tool would be easily distributed and installed at
> > >> multiple sites. Every company who depends on ACS will have their own
> > >> configuration, hardware and mission critical features. If the CI can
> be
> > >> distributed (as independent installs), it will reduce the load on any
> > >> specific site and it will enable companies who have specific hardware
> in
> > >> their direct path to success the ability to validate their use case.
> > >> - I think a standardized CI output is also important. Ideally the
> > summary
> > >> of the CI run would be posted back to the PR thread. I think it is
> also
> > >> important to push the detailed logs to a central location so others
> can
> > >> review them. Ideally, links to the detailed logs would also be pushed
> to
> > >> the PR thread.
> > >> - I think there should be multiple tiers of testing. Basic simulator
> > >> testing and full runs. We may be able to setup simulator testing
> using a
> > >> Docker container. I think @pdion891 has put some work into this that I
> > may
> > >> be able to build on.
> > >> - I think it is probably too ambitious to do CI on every PR before
> > merge,
> > >> but ideally we would We may have to only CI the master branch for now
> > and
> > >> revert any PR that causes it to fail.
> > >>
> > >> I have been digging into this quite a bit today. I will continue
> working
> > >> on this as it is very important to me for the 4.9 release. I would
> like
> > to
> > >> setup a CI environment locally as well to help validate and contribute
> > to
> > >> the effort. I have some hardware I can throw at this, so I will be
> > working
> > >> to get something setup to start testing ASAP.
> > >>
> > >> If I can work off either (or both) of your work, I would appreciate
> it.
> > >>
> > >> Cheers,
> > >>
> > >> *Will STEVENS*
> > >> Lead Developer
> > >>
> > >> *CloudOps* *| *Cloud Solutions Experts
> > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> > >> @CloudOps_
> > >>
> > >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <
> bharat.kumar@citrix.com
> > >
> > >> wrote:
> > >>
> > >> > Hi Paul,
> > >> >
> > >> > The goals and functionality of the CI on which we are working are
> the
> > >> > same (please review the FS<
> > >> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> > >> > trator+and+test+runner+to+enable+continuos+integration>
> > >> > ), The difference being the method of implementation. We are not
> using
> > >> > nested virtualisation to create test environments. The work on CI
> > >> > began long back but unfortunately due to other priorities we could
> not
> > >> > work on a continuous basis and bring it to completion.
> > >> >
> > >> > I agree that we should come to a consensus on what the community
> wants
> > >> > from the CI. IMHO the basic and immediate requirement would be to
> have
> > >> > a stable system to test the PRs and post results. Once we have this,
> > >> > we can talk about providing test setups to the community and next
> > >> > steps. I think the availability of hardware has been a big hurdle
> > >> > (even if we use virtualisation).
> > >> >
> > >> > Instead of working independently on the same thing let us work on
> this
> > >> > together. I will share what we have currently, we can use it, or
> parts
> > >> > of it (some of it is there at
> > >> > https://github.com/bvbharatk/cloudstack-automation, but it is not
> > >> > organised as in cannot be run out of the box). While virtualisation
> is
> > >> > good from the ease of orchestration and resource utilisation point
> of
> > >> > view, it has other problems like compatibility/dependencies issues.
> > >> > Also it might tie the CI to a particular Hypervisor like ESXi as you
> > >> have pointed out.
> > >> >
> > >> > So let us discuss on how to proceed further, and get things rolling.
> > >> >
> > >> > Thanks,
> > >> > Bharat.
> > >> >
> > >> >
> > >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com
> > <mailto:
> > >> > paul.angus@shapeblue.com>> wrote:
> > >> >
> > >> > Hi Bharat,
> > >> >
> > >> > Early last week I opened a discussion thread around the aims of
> > >> > Trillian which we had begun working on as a CI/CD platform for the
> > >> > community hardware (as well as individual use). Please could you
> > >> > elaborate on where there is/isn't cross-over? In terms of
> > >> > functionality and goals rather than method. It would be great to get
> > >> > community consensus around what we want from our centralised CI/CD.
> > >> >
> > >> >
> > >> > Project: Trillian
> > >> > We have been working on PoC of a CI environment design which will:
> > >> > * Provide fast build or rebuild of environments for testing.
> > >> > * Enable multiple independent concurrent builds
> > >> > * Be available on-demand through automation or individual request.
> > >> > * Be capable of fully utilising all available hardware
> > >> > * Flexible enough to be used to build super-realistic development
> > >> > environments.
> > >> > We intend to contribute and maintain our work within the Apache
> repos.
> > >> > However, we are currently building the POC, figuring out the
> > >> > requirements (and quirks) of the individual pieces, before pushing
> > >> > something concrete for to the community to review.
> > >> >
> > >> > We envision that Trillian would cater for a number of use cases:
> > >> > 1. CloudStack community integration testing of master against
> multiple
> > >> > deployment scenarios (using ASF infra) 2. CloudStack community
> > >> > integration testing of PRs against multiple deployment scenarios
> > >> > (using ASF infra) 3. Organisations/individuals running the full
> suites
> > >> > of tests available in Marvin against any physical environment they
> > >> > have.
> > >> > 4. Organisations/individuals deploying and running the full suites
> of
> > >> > tests available in Marvin against virtualised infrastructures which
> > >> > can be deployed by Marvin.
> > >> > As we intend Trillian to test multiple environments concurrently, we
> > >> > use nested virtualization on ESXi hosts (our testing has shown that
> > >> > this is the only hypervisor which can support the nested
> > >> > virtualisation of all other hypervisors with reasonable
> performance).
> > >> > We use Ansible to deploy and configure all aspects of the build as
> > >> > this will greatly lower the barrier to entry for independent
> testers.
> > >> > We use CloudStack to provision the management server and virtualised
> > >> > (nested) hosts on the physical hosts. We are creating Ansible
> > >> > playbooks and roles which can:
> > >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack
> modules
> > >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of
> compute
> > >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
> > >> > (inc. installing the relevant CloudStack agent where
> > >> > required)
> > >> > 3. Install required ACS packages on management server 4. Configure a
> > >> > zone (including adding the compute hosts) via Marvin.
> > >> > 5. Run the required Marvin tests.
> > >> > 6. Return the results
> > >> > We may need to propose enhancements to Marvin in order to sync the
> > >> > configuration of hosts with the configuration used by Marvin.
> > >> >
> > >> > Using virtualised test environments, we can have multiple test
> > >> > scenarios running concurrently. To do this we have found that it is
> > >> > necessary to create pools or ranges of VLANs and IP addresses and
> > >> > allocate them to environments. So for any given physical environment
> > >> > which will be used for testing in, we take the total range(s) of IPs
> > >> > and VLANs available and carve them into non-overlapping chunks
> > >> > suitable for concurrent use as mgmt, public and guest networks.
> These
> > >> > are stored in a MariaDB database. When a range is being used in a
> > >> > testing environment, that range is marked as 'inuse' in the
> database.
> > >> > When creating a test environment, Trillian looks in the database for
> > >> > the next available VLAN range, the next available public IP range
> and
> > >> > so on. The returned values are used to populate a Marvin cfg file
> > >> > which in turn will be used to both build the environment and when
> > >> > running the Marvin testing. When the virtualised infra is cleaned
> up,
> > >> > the database will be updated to reflect that the used ranges are
> > >> available again.
> > >> > This initiative has only recently been started, and as stated
> earlier
> > >> > we are currently figuring out the requirements (and quirks) of the
> > >> > individual pieces and looking for the most suitable wrapper to glue
> it
> > >> all together.
> > >> > Also I have found that Marvin requires a little work to make the
> > >> > output more meaningful/readable (especially in the case of errors
> and
> > >> > exceptions) and to make it a little more intelligent about the tests
> > >> > it can/can't run based on the chosen infrastructure components. I
> have
> > >> > also found unreachable or very slow ISO and template paths hardcoded
> > >> > into Marvin or individual tests.
> > >> > We plan to enhance tests to address these issues and also reduce
> > >> > runtimes where possible.
> > >> >
> > >> >
> > >> > <http://www.shapeblue.com/>
> > >> > Paul Angus
> > >> > VP Technology , ShapeBlue
> > >> >
> > >> >
> > >> > d: +44 203 617 0528 | s: +44 203 603 0540
> > >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> > >> > | m: +44 7711 418784<tel:+44%207711%20418784>
> > >> >
> > >> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> > >> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> > >> > www.shapeblue.com<http://www.shapeblue.com/>
> > >> >
> > >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Shape Blue Ltd is a company incorporated in England & Wales.
> ShapeBlue
> > >> > Services India LLP is a company incorporated in India and is
> operated
> > >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> Ltda
> > >> > is a company incorporated in Brasil and is operated under license
> from
> > >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > >> > Republic of South Africa and is traded under license from Shape Blue
> > >> > Ltd. ShapeBlue is a registered trademark.
> > >> > This email and any attachments to it may be confidential and are
> > >> > intended solely for the use of the individual to whom it is
> addressed.
> > >> > Any views or opinions expressed are solely those of the author and
> do
> > >> > not necessarily represent those of Shape Blue Ltd or related
> > >> > companies. If you are not the intended recipient of this email, you
> > >> > must neither take any action based upon its contents, nor copy or
> show
> > >> > it to anyone. Please contact the sender if you believe you have
> > received
> > >> this email in error.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > -----Original Message-----
> > >> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> > >> > Sent: Thursday, February 11, 2016 9:57 AM
> > >> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > >> > Subject: Re: Important Pending Items
> > >> >
> > >> > Hi Sebastien,
> > >> >
> > >> > As Raja said, we are actively working on it and we will share the
> code
> > >> > on github soon.
> > >> >
> > >> > Thanks,
> > >> > Bharat.
> > >> >
> > >> >
> > >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
> > >> > <ma...@citrix.com>> wrote:
> > >> > >
> > >> > > Hi Sebastien,
> > >> > >
> > >> > > On item (3) - the BVTs have been running consistently (with same
> > >> > passrates) and haven't had the time to automate the process of
> > >> > reporting to an external site.
> > >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> > >> > > working
> > >> > on getting it run automatically and consistently.
> > >> > > Hope to see it ready soon.
> > >> > >
> > >> > > Best,
> > >> > > Raja
> > >> > > -----Original Message-----
> > >> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > >> > > Sent: Wednesday, February 10, 2016 2:29 PM
> > >> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > >> > > Subject: Important Pending Items
> > >> > >
> > >> > > Morning folks,
> > >> > >
> > >> > > We have several crucial pending items, that we need to resolve
> > >> > > before
> > >> > moving on:
> > >> > >
> > >> > > 1- We need an RM for master ( just saw some commits there that
> > >> > > should be
> > >> > reverted or merged properly in other branches).
> > >> > >
> > >> > > 2- We need to automate writing release notes, pushing/tagging new
> > >> > > docs
> > >> > when release come out and announcing releases on website. Currently
> > >> > neither
> > >> > 4.7 nor 4.8 have been announced.
> > >> > >
> > >> > > 3- CI is still almost inexistent
> > >> > >
> > >> > > -Sebastien
> > >> >
> > >> > Find out more about ShapeBlue and our range of CloudStack related
> > >> services:
> > >> > IaaS Cloud Design & Build<
> > >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > >> > IaaS deployment framework<http://shapeblue.com/csforge/>
> > >> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> |
> > >> > CloudStack Software Engineering<
> > >> > http://shapeblue.com/cloudstack-software-engineering/>
> > >> > CloudStack Infrastructure Support<
> > >> > http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > >> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/
> >
> > >> >
> > >> >
> > >> Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > >> IaaS Cloud Design & Build<
> > >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > >> IaaS deployment framework<http://shapeblue.com/csforge/>
> > >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> > >> CloudStack Software Engineering<
> > >> http://shapeblue.com/cloudstack-software-engineering/>
> > >> CloudStack Infrastructure Support<
> > >> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> > >>
> >
>

Re: Important Pending Items

[Will Stevens <ws...@cloudops.com>]

Awesome, thanks Remi.  I will review this.

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Tue, Mar 1, 2016 at 4:29 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:

> Hi Will,
>
> We used this to test hundreds of PRs:
> https://github.com/schubergphilis/MCT-shared/. I can talk you through it
> sometime next week or so if you want.
>
> Regards,
> Remi
>
>
> On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will Stevens" <
> williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
>
> >Thanks Paul.  Great to hear you are planning to make everything available
> >and you are willing to take contributions from the community.
> >
> >I am currently trying to get up to speed on everything that has been done
> >and all the different pieces in play.  I would like to have a CI in place
> >(even if it is only locally on my hardware) for testing PRs for 4.9.  My
> >focus right now is solving the CI problem, so I am interested in any
> >developments on this front.
> >
> >We have Travis doing smoke tests on every PR right now.  Travis is free
> for
> >open source projects.  Is there a reason we are not doing more extensive
> >tests with Travis since it is already integrated into github?  Like maybe
> >running the simulator on every PR?  I think this could be a good first
> step
> >to weed out some of the PRs that are likely to fail a full CI run.
> >
> >Cheers,
> >
> >*Will STEVENS*
> >Lead Developer
> >
> >*CloudOps* *| *Cloud Solutions Experts
> >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >w cloudops.com *|* tw @CloudOps_
> >
> >On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
> >
> >> Hi Will,
> >>
> >> It will be 100% open source. The $dayjob has blocked the team from
> working
> >> on it recently. I'd like to get some of our ideas into the CloudStack
> wiki
> >> this week before I go on leave for a week.
> >> Rohit has created a repo for us: https://github.com/shapeblue/Trillian
> >> (which looks a bit sad at the moment - I hope to push our Ansible roles
> >> for building mgmt. servers and Marvin boxes into it this week also)
> >>
> >> Internally we're planning a two week sprint on it starting 14th March.
> >> Having to dip in and out of this effort is making it very difficult to
> get
> >> anywhere
> >>
> >> I think the biggest part of the work will be to get Marvin up to
> scratch,
> >> the log output from the tests is pretty bad.
> >>
> >> We very much want this to be a community 'thing', and we're happy to
> work
> >> with everyone/anyone to get there.
> >>
> >> Any ideas on the most effective way to have a number of people on this -
> >> I'm all ears.
> >> In the meantime I'll try to make good on my promises to get our ideas
> >> visible to everyone...
> >>
> >>
> >>
> >>
> >> Paul Angus
> >> VP Technology   ,       ShapeBlue
> >>
> >>
> >> t:      @cloudyangus<te...@cloudyangus>
> >>
> >> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
> >> |      w:      www.shapeblue.com<http://www.shapeblue.com>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> >> Behalf Of Will Stevens
> >> Sent: Monday, February 29, 2016 9:52 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Important Pending Items
> >>
> >> @Paul and @Bharat, thank you for the quality write-ups on this topic.
> >>
> >> re:Paul's Details:
> >> I share a lot of your thoughts on this. The hypervisor in hypervisor
> setup
> >> with the ability to test multiple environments is also core to my
> thinking.
> >> I can't find any details on Trillian, other than what is in this
> thread. Is
> >> it open source and freely available? This is being developed at
> ShapeBlue?
> >>
> >> re:Bharat's Details:
> >> This seems like it is the most likely candidate for being able to offer
> >> value in the short term. Thank you for the work documenting it and
> making
> >> the source available. I still need to review the code. This approach
> will
> >> take quite a bit of work get setup initially, but once setup, it looks
> like
> >> it will do the trick. By using only freely available software makes it
> more
> >> accessible than some options which have been suggested in the past.
> >>
> >> My thoughts on the topic:
> >> - I think it is important that the CI can test all the different
> >> hypervisors and any hardware integrations available in the environment.
> >> - Ideally the CI tool would be easily distributed and installed at
> >> multiple sites. Every company who depends on ACS will have their own
> >> configuration, hardware and mission critical features. If the CI can be
> >> distributed (as independent installs), it will reduce the load on any
> >> specific site and it will enable companies who have specific hardware in
> >> their direct path to success the ability to validate their use case.
> >> - I think a standardized CI output is also important. Ideally the
> summary
> >> of the CI run would be posted back to the PR thread. I think it is also
> >> important to push the detailed logs to a central location so others can
> >> review them. Ideally, links to the detailed logs would also be pushed to
> >> the PR thread.
> >> - I think there should be multiple tiers of testing. Basic simulator
> >> testing and full runs. We may be able to setup simulator testing using a
> >> Docker container. I think @pdion891 has put some work into this that I
> may
> >> be able to build on.
> >> - I think it is probably too ambitious to do CI on every PR before
> merge,
> >> but ideally we would We may have to only CI the master branch for now
> and
> >> revert any PR that causes it to fail.
> >>
> >> I have been digging into this quite a bit today. I will continue working
> >> on this as it is very important to me for the 4.9 release. I would like
> to
> >> setup a CI environment locally as well to help validate and contribute
> to
> >> the effort. I have some hardware I can throw at this, so I will be
> working
> >> to get something setup to start testing ASAP.
> >>
> >> If I can work off either (or both) of your work, I would appreciate it.
> >>
> >> Cheers,
> >>
> >> *Will STEVENS*
> >> Lead Developer
> >>
> >> *CloudOps* *| *Cloud Solutions Experts
> >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> >> @CloudOps_
> >>
> >> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bharat.kumar@citrix.com
> >
> >> wrote:
> >>
> >> > Hi Paul,
> >> >
> >> > The goals and functionality of the CI on which we are working are the
> >> > same (please review the FS<
> >> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> >> > trator+and+test+runner+to+enable+continuos+integration>
> >> > ), The difference being the method of implementation. We are not using
> >> > nested virtualisation to create test environments. The work on CI
> >> > began long back but unfortunately due to other priorities we could not
> >> > work on a continuous basis and bring it to completion.
> >> >
> >> > I agree that we should come to a consensus on what the community wants
> >> > from the CI. IMHO the basic and immediate requirement would be to have
> >> > a stable system to test the PRs and post results. Once we have this,
> >> > we can talk about providing test setups to the community and next
> >> > steps. I think the availability of hardware has been a big hurdle
> >> > (even if we use virtualisation).
> >> >
> >> > Instead of working independently on the same thing let us work on this
> >> > together. I will share what we have currently, we can use it, or parts
> >> > of it (some of it is there at
> >> > https://github.com/bvbharatk/cloudstack-automation, but it is not
> >> > organised as in cannot be run out of the box). While virtualisation is
> >> > good from the ease of orchestration and resource utilisation point of
> >> > view, it has other problems like compatibility/dependencies issues.
> >> > Also it might tie the CI to a particular Hypervisor like ESXi as you
> >> have pointed out.
> >> >
> >> > So let us discuss on how to proceed further, and get things rolling.
> >> >
> >> > Thanks,
> >> > Bharat.
> >> >
> >> >
> >> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com
> <mailto:
> >> > paul.angus@shapeblue.com>> wrote:
> >> >
> >> > Hi Bharat,
> >> >
> >> > Early last week I opened a discussion thread around the aims of
> >> > Trillian which we had begun working on as a CI/CD platform for the
> >> > community hardware (as well as individual use). Please could you
> >> > elaborate on where there is/isn't cross-over? In terms of
> >> > functionality and goals rather than method. It would be great to get
> >> > community consensus around what we want from our centralised CI/CD.
> >> >
> >> >
> >> > Project: Trillian
> >> > We have been working on PoC of a CI environment design which will:
> >> > * Provide fast build or rebuild of environments for testing.
> >> > * Enable multiple independent concurrent builds
> >> > * Be available on-demand through automation or individual request.
> >> > * Be capable of fully utilising all available hardware
> >> > * Flexible enough to be used to build super-realistic development
> >> > environments.
> >> > We intend to contribute and maintain our work within the Apache repos.
> >> > However, we are currently building the POC, figuring out the
> >> > requirements (and quirks) of the individual pieces, before pushing
> >> > something concrete for to the community to review.
> >> >
> >> > We envision that Trillian would cater for a number of use cases:
> >> > 1. CloudStack community integration testing of master against multiple
> >> > deployment scenarios (using ASF infra) 2. CloudStack community
> >> > integration testing of PRs against multiple deployment scenarios
> >> > (using ASF infra) 3. Organisations/individuals running the full suites
> >> > of tests available in Marvin against any physical environment they
> >> > have.
> >> > 4. Organisations/individuals deploying and running the full suites of
> >> > tests available in Marvin against virtualised infrastructures which
> >> > can be deployed by Marvin.
> >> > As we intend Trillian to test multiple environments concurrently, we
> >> > use nested virtualization on ESXi hosts (our testing has shown that
> >> > this is the only hypervisor which can support the nested
> >> > virtualisation of all other hypervisors with reasonable performance).
> >> > We use Ansible to deploy and configure all aspects of the build as
> >> > this will greatly lower the barrier to entry for independent testers.
> >> > We use CloudStack to provision the management server and virtualised
> >> > (nested) hosts on the physical hosts. We are creating Ansible
> >> > playbooks and roles which can:
> >> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules
> >> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute
> >> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
> >> > (inc. installing the relevant CloudStack agent where
> >> > required)
> >> > 3. Install required ACS packages on management server 4. Configure a
> >> > zone (including adding the compute hosts) via Marvin.
> >> > 5. Run the required Marvin tests.
> >> > 6. Return the results
> >> > We may need to propose enhancements to Marvin in order to sync the
> >> > configuration of hosts with the configuration used by Marvin.
> >> >
> >> > Using virtualised test environments, we can have multiple test
> >> > scenarios running concurrently. To do this we have found that it is
> >> > necessary to create pools or ranges of VLANs and IP addresses and
> >> > allocate them to environments. So for any given physical environment
> >> > which will be used for testing in, we take the total range(s) of IPs
> >> > and VLANs available and carve them into non-overlapping chunks
> >> > suitable for concurrent use as mgmt, public and guest networks. These
> >> > are stored in a MariaDB database. When a range is being used in a
> >> > testing environment, that range is marked as 'inuse' in the database.
> >> > When creating a test environment, Trillian looks in the database for
> >> > the next available VLAN range, the next available public IP range and
> >> > so on. The returned values are used to populate a Marvin cfg file
> >> > which in turn will be used to both build the environment and when
> >> > running the Marvin testing. When the virtualised infra is cleaned up,
> >> > the database will be updated to reflect that the used ranges are
> >> available again.
> >> > This initiative has only recently been started, and as stated earlier
> >> > we are currently figuring out the requirements (and quirks) of the
> >> > individual pieces and looking for the most suitable wrapper to glue it
> >> all together.
> >> > Also I have found that Marvin requires a little work to make the
> >> > output more meaningful/readable (especially in the case of errors and
> >> > exceptions) and to make it a little more intelligent about the tests
> >> > it can/can't run based on the chosen infrastructure components. I have
> >> > also found unreachable or very slow ISO and template paths hardcoded
> >> > into Marvin or individual tests.
> >> > We plan to enhance tests to address these issues and also reduce
> >> > runtimes where possible.
> >> >
> >> >
> >> > <http://www.shapeblue.com/>
> >> > Paul Angus
> >> > VP Technology , ShapeBlue
> >> >
> >> >
> >> > d: +44 203 617 0528 | s: +44 203 603 0540
> >> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> >> > | m: +44 7711 418784<tel:+44%207711%20418784>
> >> >
> >> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> >> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> >> > www.shapeblue.com<http://www.shapeblue.com/>
> >> >
> >> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> >> > Services India LLP is a company incorporated in India and is operated
> >> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> >> > is a company incorporated in Brasil and is operated under license from
> >> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> >> > Republic of South Africa and is traded under license from Shape Blue
> >> > Ltd. ShapeBlue is a registered trademark.
> >> > This email and any attachments to it may be confidential and are
> >> > intended solely for the use of the individual to whom it is addressed.
> >> > Any views or opinions expressed are solely those of the author and do
> >> > not necessarily represent those of Shape Blue Ltd or related
> >> > companies. If you are not the intended recipient of this email, you
> >> > must neither take any action based upon its contents, nor copy or show
> >> > it to anyone. Please contact the sender if you believe you have
> received
> >> this email in error.
> >> >
> >> >
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> >> > Sent: Thursday, February 11, 2016 9:57 AM
> >> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >> > Subject: Re: Important Pending Items
> >> >
> >> > Hi Sebastien,
> >> >
> >> > As Raja said, we are actively working on it and we will share the code
> >> > on github soon.
> >> >
> >> > Thanks,
> >> > Bharat.
> >> >
> >> >
> >> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
> >> > <ma...@citrix.com>> wrote:
> >> > >
> >> > > Hi Sebastien,
> >> > >
> >> > > On item (3) - the BVTs have been running consistently (with same
> >> > passrates) and haven't had the time to automate the process of
> >> > reporting to an external site.
> >> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> >> > > working
> >> > on getting it run automatically and consistently.
> >> > > Hope to see it ready soon.
> >> > >
> >> > > Best,
> >> > > Raja
> >> > > -----Original Message-----
> >> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> > > Sent: Wednesday, February 10, 2016 2:29 PM
> >> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >> > > Subject: Important Pending Items
> >> > >
> >> > > Morning folks,
> >> > >
> >> > > We have several crucial pending items, that we need to resolve
> >> > > before
> >> > moving on:
> >> > >
> >> > > 1- We need an RM for master ( just saw some commits there that
> >> > > should be
> >> > reverted or merged properly in other branches).
> >> > >
> >> > > 2- We need to automate writing release notes, pushing/tagging new
> >> > > docs
> >> > when release come out and announcing releases on website. Currently
> >> > neither
> >> > 4.7 nor 4.8 have been announced.
> >> > >
> >> > > 3- CI is still almost inexistent
> >> > >
> >> > > -Sebastien
> >> >
> >> > Find out more about ShapeBlue and our range of CloudStack related
> >> services:
> >> > IaaS Cloud Design & Build<
> >> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> >> > IaaS deployment framework<http://shapeblue.com/csforge/>
> >> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> >> > CloudStack Software Engineering<
> >> > http://shapeblue.com/cloudstack-software-engineering/>
> >> > CloudStack Infrastructure Support<
> >> > http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> >> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> >> >
> >> >
> >> Find out more about ShapeBlue and our range of CloudStack related
> services:
> >> IaaS Cloud Design & Build<
> >> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> >> IaaS deployment framework<http://shapeblue.com/csforge/>
> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> >> CloudStack Software Engineering<
> >> http://shapeblue.com/cloudstack-software-engineering/>
> >> CloudStack Infrastructure Support<
> >> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> >> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> >>
>

Re: Important Pending Items

[Remi Bergsma <RB...@schubergphilis.com>]

Hi Will,

We used this to test hundreds of PRs: https://github.com/schubergphilis/MCT-shared/. I can talk you through it sometime next week or so if you want.

Regards,
Remi


On 01/03/16 21:50, "williamstevens@gmail.com on behalf of Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:

>Thanks Paul.  Great to hear you are planning to make everything available
>and you are willing to take contributions from the community.
>
>I am currently trying to get up to speed on everything that has been done
>and all the different pieces in play.  I would like to have a CI in place
>(even if it is only locally on my hardware) for testing PRs for 4.9.  My
>focus right now is solving the CI problem, so I am interested in any
>developments on this front.
>
>We have Travis doing smoke tests on every PR right now.  Travis is free for
>open source projects.  Is there a reason we are not doing more extensive
>tests with Travis since it is already integrated into github?  Like maybe
>running the simulator on every PR?  I think this could be a good first step
>to weed out some of the PRs that are likely to fail a full CI run.
>
>Cheers,
>
>*Will STEVENS*
>Lead Developer
>
>*CloudOps* *| *Cloud Solutions Experts
>420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>w cloudops.com *|* tw @CloudOps_
>
>On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com> wrote:
>
>> Hi Will,
>>
>> It will be 100% open source. The $dayjob has blocked the team from working
>> on it recently. I'd like to get some of our ideas into the CloudStack wiki
>> this week before I go on leave for a week.
>> Rohit has created a repo for us: https://github.com/shapeblue/Trillian
>> (which looks a bit sad at the moment - I hope to push our Ansible roles
>> for building mgmt. servers and Marvin boxes into it this week also)
>>
>> Internally we're planning a two week sprint on it starting 14th March.
>> Having to dip in and out of this effort is making it very difficult to get
>> anywhere
>>
>> I think the biggest part of the work will be to get Marvin up to scratch,
>> the log output from the tests is pretty bad.
>>
>> We very much want this to be a community 'thing', and we're happy to work
>> with everyone/anyone to get there.
>>
>> Any ideas on the most effective way to have a number of people on this -
>> I'm all ears.
>> In the meantime I'll try to make good on my promises to get our ideas
>> visible to everyone...
>>
>>
>>
>>
>> Paul Angus
>> VP Technology   ,       ShapeBlue
>>
>>
>> t:      @cloudyangus<te...@cloudyangus>
>>
>> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
>> |      w:      www.shapeblue.com<http://www.shapeblue.com>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
>> Behalf Of Will Stevens
>> Sent: Monday, February 29, 2016 9:52 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Important Pending Items
>>
>> @Paul and @Bharat, thank you for the quality write-ups on this topic.
>>
>> re:Paul's Details:
>> I share a lot of your thoughts on this. The hypervisor in hypervisor setup
>> with the ability to test multiple environments is also core to my thinking.
>> I can't find any details on Trillian, other than what is in this thread. Is
>> it open source and freely available? This is being developed at ShapeBlue?
>>
>> re:Bharat's Details:
>> This seems like it is the most likely candidate for being able to offer
>> value in the short term. Thank you for the work documenting it and making
>> the source available. I still need to review the code. This approach will
>> take quite a bit of work get setup initially, but once setup, it looks like
>> it will do the trick. By using only freely available software makes it more
>> accessible than some options which have been suggested in the past.
>>
>> My thoughts on the topic:
>> - I think it is important that the CI can test all the different
>> hypervisors and any hardware integrations available in the environment.
>> - Ideally the CI tool would be easily distributed and installed at
>> multiple sites. Every company who depends on ACS will have their own
>> configuration, hardware and mission critical features. If the CI can be
>> distributed (as independent installs), it will reduce the load on any
>> specific site and it will enable companies who have specific hardware in
>> their direct path to success the ability to validate their use case.
>> - I think a standardized CI output is also important. Ideally the summary
>> of the CI run would be posted back to the PR thread. I think it is also
>> important to push the detailed logs to a central location so others can
>> review them. Ideally, links to the detailed logs would also be pushed to
>> the PR thread.
>> - I think there should be multiple tiers of testing. Basic simulator
>> testing and full runs. We may be able to setup simulator testing using a
>> Docker container. I think @pdion891 has put some work into this that I may
>> be able to build on.
>> - I think it is probably too ambitious to do CI on every PR before merge,
>> but ideally we would We may have to only CI the master branch for now and
>> revert any PR that causes it to fail.
>>
>> I have been digging into this quite a bit today. I will continue working
>> on this as it is very important to me for the 4.9 release. I would like to
>> setup a CI environment locally as well to help validate and contribute to
>> the effort. I have some hardware I can throw at this, so I will be working
>> to get something setup to start testing ASAP.
>>
>> If I can work off either (or both) of your work, I would appreciate it.
>>
>> Cheers,
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
>> @CloudOps_
>>
>> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bh...@citrix.com>
>> wrote:
>>
>> > Hi Paul,
>> >
>> > The goals and functionality of the CI on which we are working are the
>> > same (please review the FS<
>> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
>> > trator+and+test+runner+to+enable+continuos+integration>
>> > ), The difference being the method of implementation. We are not using
>> > nested virtualisation to create test environments. The work on CI
>> > began long back but unfortunately due to other priorities we could not
>> > work on a continuous basis and bring it to completion.
>> >
>> > I agree that we should come to a consensus on what the community wants
>> > from the CI. IMHO the basic and immediate requirement would be to have
>> > a stable system to test the PRs and post results. Once we have this,
>> > we can talk about providing test setups to the community and next
>> > steps. I think the availability of hardware has been a big hurdle
>> > (even if we use virtualisation).
>> >
>> > Instead of working independently on the same thing let us work on this
>> > together. I will share what we have currently, we can use it, or parts
>> > of it (some of it is there at
>> > https://github.com/bvbharatk/cloudstack-automation, but it is not
>> > organised as in cannot be run out of the box). While virtualisation is
>> > good from the ease of orchestration and resource utilisation point of
>> > view, it has other problems like compatibility/dependencies issues.
>> > Also it might tie the CI to a particular Hypervisor like ESXi as you
>> have pointed out.
>> >
>> > So let us discuss on how to proceed further, and get things rolling.
>> >
>> > Thanks,
>> > Bharat.
>> >
>> >
>> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com<mailto:
>> > paul.angus@shapeblue.com>> wrote:
>> >
>> > Hi Bharat,
>> >
>> > Early last week I opened a discussion thread around the aims of
>> > Trillian which we had begun working on as a CI/CD platform for the
>> > community hardware (as well as individual use). Please could you
>> > elaborate on where there is/isn't cross-over? In terms of
>> > functionality and goals rather than method. It would be great to get
>> > community consensus around what we want from our centralised CI/CD.
>> >
>> >
>> > Project: Trillian
>> > We have been working on PoC of a CI environment design which will:
>> > * Provide fast build or rebuild of environments for testing.
>> > * Enable multiple independent concurrent builds
>> > * Be available on-demand through automation or individual request.
>> > * Be capable of fully utilising all available hardware
>> > * Flexible enough to be used to build super-realistic development
>> > environments.
>> > We intend to contribute and maintain our work within the Apache repos.
>> > However, we are currently building the POC, figuring out the
>> > requirements (and quirks) of the individual pieces, before pushing
>> > something concrete for to the community to review.
>> >
>> > We envision that Trillian would cater for a number of use cases:
>> > 1. CloudStack community integration testing of master against multiple
>> > deployment scenarios (using ASF infra) 2. CloudStack community
>> > integration testing of PRs against multiple deployment scenarios
>> > (using ASF infra) 3. Organisations/individuals running the full suites
>> > of tests available in Marvin against any physical environment they
>> > have.
>> > 4. Organisations/individuals deploying and running the full suites of
>> > tests available in Marvin against virtualised infrastructures which
>> > can be deployed by Marvin.
>> > As we intend Trillian to test multiple environments concurrently, we
>> > use nested virtualization on ESXi hosts (our testing has shown that
>> > this is the only hypervisor which can support the nested
>> > virtualisation of all other hypervisors with reasonable performance).
>> > We use Ansible to deploy and configure all aspects of the build as
>> > this will greatly lower the barrier to entry for independent testers.
>> > We use CloudStack to provision the management server and virtualised
>> > (nested) hosts on the physical hosts. We are creating Ansible
>> > playbooks and roles which can:
>> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules
>> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute
>> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
>> > (inc. installing the relevant CloudStack agent where
>> > required)
>> > 3. Install required ACS packages on management server 4. Configure a
>> > zone (including adding the compute hosts) via Marvin.
>> > 5. Run the required Marvin tests.
>> > 6. Return the results
>> > We may need to propose enhancements to Marvin in order to sync the
>> > configuration of hosts with the configuration used by Marvin.
>> >
>> > Using virtualised test environments, we can have multiple test
>> > scenarios running concurrently. To do this we have found that it is
>> > necessary to create pools or ranges of VLANs and IP addresses and
>> > allocate them to environments. So for any given physical environment
>> > which will be used for testing in, we take the total range(s) of IPs
>> > and VLANs available and carve them into non-overlapping chunks
>> > suitable for concurrent use as mgmt, public and guest networks. These
>> > are stored in a MariaDB database. When a range is being used in a
>> > testing environment, that range is marked as 'inuse' in the database.
>> > When creating a test environment, Trillian looks in the database for
>> > the next available VLAN range, the next available public IP range and
>> > so on. The returned values are used to populate a Marvin cfg file
>> > which in turn will be used to both build the environment and when
>> > running the Marvin testing. When the virtualised infra is cleaned up,
>> > the database will be updated to reflect that the used ranges are
>> available again.
>> > This initiative has only recently been started, and as stated earlier
>> > we are currently figuring out the requirements (and quirks) of the
>> > individual pieces and looking for the most suitable wrapper to glue it
>> all together.
>> > Also I have found that Marvin requires a little work to make the
>> > output more meaningful/readable (especially in the case of errors and
>> > exceptions) and to make it a little more intelligent about the tests
>> > it can/can't run based on the chosen infrastructure components. I have
>> > also found unreachable or very slow ISO and template paths hardcoded
>> > into Marvin or individual tests.
>> > We plan to enhance tests to address these issues and also reduce
>> > runtimes where possible.
>> >
>> >
>> > <http://www.shapeblue.com/>
>> > Paul Angus
>> > VP Technology , ShapeBlue
>> >
>> >
>> > d: +44 203 617 0528 | s: +44 203 603 0540
>> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
>> > | m: +44 7711 418784<tel:+44%207711%20418784>
>> >
>> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
>> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
>> > www.shapeblue.com<http://www.shapeblue.com/>
>> >
>> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
>> >
>> >
>> >
>> >
>> >
>> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>> > Services India LLP is a company incorporated in India and is operated
>> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
>> > is a company incorporated in Brasil and is operated under license from
>> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
>> > Republic of South Africa and is traded under license from Shape Blue
>> > Ltd. ShapeBlue is a registered trademark.
>> > This email and any attachments to it may be confidential and are
>> > intended solely for the use of the individual to whom it is addressed.
>> > Any views or opinions expressed are solely those of the author and do
>> > not necessarily represent those of Shape Blue Ltd or related
>> > companies. If you are not the intended recipient of this email, you
>> > must neither take any action based upon its contents, nor copy or show
>> > it to anyone. Please contact the sender if you believe you have received
>> this email in error.
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
>> > Sent: Thursday, February 11, 2016 9:57 AM
>> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> > Subject: Re: Important Pending Items
>> >
>> > Hi Sebastien,
>> >
>> > As Raja said, we are actively working on it and we will share the code
>> > on github soon.
>> >
>> > Thanks,
>> > Bharat.
>> >
>> >
>> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
>> > <ma...@citrix.com>> wrote:
>> > >
>> > > Hi Sebastien,
>> > >
>> > > On item (3) - the BVTs have been running consistently (with same
>> > passrates) and haven't had the time to automate the process of
>> > reporting to an external site.
>> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
>> > > working
>> > on getting it run automatically and consistently.
>> > > Hope to see it ready soon.
>> > >
>> > > Best,
>> > > Raja
>> > > -----Original Message-----
>> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
>> > > Sent: Wednesday, February 10, 2016 2:29 PM
>> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> > > Subject: Important Pending Items
>> > >
>> > > Morning folks,
>> > >
>> > > We have several crucial pending items, that we need to resolve
>> > > before
>> > moving on:
>> > >
>> > > 1- We need an RM for master ( just saw some commits there that
>> > > should be
>> > reverted or merged properly in other branches).
>> > >
>> > > 2- We need to automate writing release notes, pushing/tagging new
>> > > docs
>> > when release come out and announcing releases on website. Currently
>> > neither
>> > 4.7 nor 4.8 have been announced.
>> > >
>> > > 3- CI is still almost inexistent
>> > >
>> > > -Sebastien
>> >
>> > Find out more about ShapeBlue and our range of CloudStack related
>> services:
>> > IaaS Cloud Design & Build<
>> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>> > IaaS deployment framework<http://shapeblue.com/csforge/>
>> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>> > CloudStack Software Engineering<
>> > http://shapeblue.com/cloudstack-software-engineering/>
>> > CloudStack Infrastructure Support<
>> > http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>> >
>> >
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build<
>> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>> IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>> CloudStack Software Engineering<
>> http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<
>> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>

Re: Important Pending Items

[Will Stevens <ws...@cloudops.com>]

Thanks Paul.  Great to hear you are planning to make everything available
and you are willing to take contributions from the community.

I am currently trying to get up to speed on everything that has been done
and all the different pieces in play.  I would like to have a CI in place
(even if it is only locally on my hardware) for testing PRs for 4.9.  My
focus right now is solving the CI problem, so I am interested in any
developments on this front.

We have Travis doing smoke tests on every PR right now.  Travis is free for
open source projects.  Is there a reason we are not doing more extensive
tests with Travis since it is already integrated into github?  Like maybe
running the simulator on every PR?  I think this could be a good first step
to weed out some of the PRs that are likely to fail a full CI run.

Cheers,

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Tue, Mar 1, 2016 at 3:40 PM, Paul Angus <pa...@shapeblue.com> wrote:

> Hi Will,
>
> It will be 100% open source. The $dayjob has blocked the team from working
> on it recently. I'd like to get some of our ideas into the CloudStack wiki
> this week before I go on leave for a week.
> Rohit has created a repo for us: https://github.com/shapeblue/Trillian
> (which looks a bit sad at the moment - I hope to push our Ansible roles
> for building mgmt. servers and Marvin boxes into it this week also)
>
> Internally we're planning a two week sprint on it starting 14th March.
> Having to dip in and out of this effort is making it very difficult to get
> anywhere
>
> I think the biggest part of the work will be to get Marvin up to scratch,
> the log output from the tests is pretty bad.
>
> We very much want this to be a community 'thing', and we're happy to work
> with everyone/anyone to get there.
>
> Any ideas on the most effective way to have a number of people on this -
> I'm all ears.
> In the meantime I'll try to make good on my promises to get our ideas
> visible to everyone...
>
>
>
>
> Paul Angus
> VP Technology   ,       ShapeBlue
>
>
> t:      @cloudyangus<te...@cloudyangus>
>
> e:      paul.angus@shapeblue.com<ma...@shapeblue.com>
> |      w:      www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
> -----Original Message-----
> From: williamstevens@gmail.com [mailto:williamstevens@gmail.com] On
> Behalf Of Will Stevens
> Sent: Monday, February 29, 2016 9:52 PM
> To: dev@cloudstack.apache.org
> Subject: Re: Important Pending Items
>
> @Paul and @Bharat, thank you for the quality write-ups on this topic.
>
> re:Paul's Details:
> I share a lot of your thoughts on this. The hypervisor in hypervisor setup
> with the ability to test multiple environments is also core to my thinking.
> I can't find any details on Trillian, other than what is in this thread. Is
> it open source and freely available? This is being developed at ShapeBlue?
>
> re:Bharat's Details:
> This seems like it is the most likely candidate for being able to offer
> value in the short term. Thank you for the work documenting it and making
> the source available. I still need to review the code. This approach will
> take quite a bit of work get setup initially, but once setup, it looks like
> it will do the trick. By using only freely available software makes it more
> accessible than some options which have been suggested in the past.
>
> My thoughts on the topic:
> - I think it is important that the CI can test all the different
> hypervisors and any hardware integrations available in the environment.
> - Ideally the CI tool would be easily distributed and installed at
> multiple sites. Every company who depends on ACS will have their own
> configuration, hardware and mission critical features. If the CI can be
> distributed (as independent installs), it will reduce the load on any
> specific site and it will enable companies who have specific hardware in
> their direct path to success the ability to validate their use case.
> - I think a standardized CI output is also important. Ideally the summary
> of the CI run would be posted back to the PR thread. I think it is also
> important to push the detailed logs to a central location so others can
> review them. Ideally, links to the detailed logs would also be pushed to
> the PR thread.
> - I think there should be multiple tiers of testing. Basic simulator
> testing and full runs. We may be able to setup simulator testing using a
> Docker container. I think @pdion891 has put some work into this that I may
> be able to build on.
> - I think it is probably too ambitious to do CI on every PR before merge,
> but ideally we would We may have to only CI the master branch for now and
> revert any PR that causes it to fail.
>
> I have been digging into this quite a bit today. I will continue working
> on this as it is very important to me for the 4.9 release. I would like to
> setup a CI environment locally as well to help validate and contribute to
> the effort. I have some hardware I can throw at this, so I will be working
> to get something setup to start testing ASAP.
>
> If I can work off either (or both) of your work, I would appreciate it.
>
> Cheers,
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> @CloudOps_
>
> On Fri, Feb 12, 2016 at 12:49 AM, Bharat Kumar <bh...@citrix.com>
> wrote:
>
> > Hi Paul,
> >
> > The goals and functionality of the CI on which we are working are the
> > same (please review the FS<
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Test+bed+orches
> > trator+and+test+runner+to+enable+continuos+integration>
> > ), The difference being the method of implementation. We are not using
> > nested virtualisation to create test environments. The work on CI
> > began long back but unfortunately due to other priorities we could not
> > work on a continuous basis and bring it to completion.
> >
> > I agree that we should come to a consensus on what the community wants
> > from the CI. IMHO the basic and immediate requirement would be to have
> > a stable system to test the PRs and post results. Once we have this,
> > we can talk about providing test setups to the community and next
> > steps. I think the availability of hardware has been a big hurdle
> > (even if we use virtualisation).
> >
> > Instead of working independently on the same thing let us work on this
> > together. I will share what we have currently, we can use it, or parts
> > of it (some of it is there at
> > https://github.com/bvbharatk/cloudstack-automation, but it is not
> > organised as in cannot be run out of the box). While virtualisation is
> > good from the ease of orchestration and resource utilisation point of
> > view, it has other problems like compatibility/dependencies issues.
> > Also it might tie the CI to a particular Hypervisor like ESXi as you
> have pointed out.
> >
> > So let us discuss on how to proceed further, and get things rolling.
> >
> > Thanks,
> > Bharat.
> >
> >
> > On 11-Feb-2016, at 9:44 PM, Paul Angus <paul.angus@shapeblue.com<mailto:
> > paul.angus@shapeblue.com>> wrote:
> >
> > Hi Bharat,
> >
> > Early last week I opened a discussion thread around the aims of
> > Trillian which we had begun working on as a CI/CD platform for the
> > community hardware (as well as individual use). Please could you
> > elaborate on where there is/isn't cross-over? In terms of
> > functionality and goals rather than method. It would be great to get
> > community consensus around what we want from our centralised CI/CD.
> >
> >
> > Project: Trillian
> > We have been working on PoC of a CI environment design which will:
> > * Provide fast build or rebuild of environments for testing.
> > * Enable multiple independent concurrent builds
> > * Be available on-demand through automation or individual request.
> > * Be capable of fully utilising all available hardware
> > * Flexible enough to be used to build super-realistic development
> > environments.
> > We intend to contribute and maintain our work within the Apache repos.
> > However, we are currently building the POC, figuring out the
> > requirements (and quirks) of the individual pieces, before pushing
> > something concrete for to the community to review.
> >
> > We envision that Trillian would cater for a number of use cases:
> > 1. CloudStack community integration testing of master against multiple
> > deployment scenarios (using ASF infra) 2. CloudStack community
> > integration testing of PRs against multiple deployment scenarios
> > (using ASF infra) 3. Organisations/individuals running the full suites
> > of tests available in Marvin against any physical environment they
> > have.
> > 4. Organisations/individuals deploying and running the full suites of
> > tests available in Marvin against virtualised infrastructures which
> > can be deployed by Marvin.
> > As we intend Trillian to test multiple environments concurrently, we
> > use nested virtualization on ESXi hosts (our testing has shown that
> > this is the only hypervisor which can support the nested
> > virtualisation of all other hypervisors with reasonable performance).
> > We use Ansible to deploy and configure all aspects of the build as
> > this will greatly lower the barrier to entry for independent testers.
> > We use CloudStack to provision the management server and virtualised
> > (nested) hosts on the physical hosts. We are creating Ansible
> > playbooks and roles which can:
> > 1. Create guest instances using Rene's Ansible 2.0 CloudStack modules
> > - a Marvin VM, a Mgmt Server (CentOS or Ubuntu), any number of compute
> > hosts (KVM, vSphere or XenServer. Hyper-V later) 2. Configure hosts
> > (inc. installing the relevant CloudStack agent where
> > required)
> > 3. Install required ACS packages on management server 4. Configure a
> > zone (including adding the compute hosts) via Marvin.
> > 5. Run the required Marvin tests.
> > 6. Return the results
> > We may need to propose enhancements to Marvin in order to sync the
> > configuration of hosts with the configuration used by Marvin.
> >
> > Using virtualised test environments, we can have multiple test
> > scenarios running concurrently. To do this we have found that it is
> > necessary to create pools or ranges of VLANs and IP addresses and
> > allocate them to environments. So for any given physical environment
> > which will be used for testing in, we take the total range(s) of IPs
> > and VLANs available and carve them into non-overlapping chunks
> > suitable for concurrent use as mgmt, public and guest networks. These
> > are stored in a MariaDB database. When a range is being used in a
> > testing environment, that range is marked as 'inuse' in the database.
> > When creating a test environment, Trillian looks in the database for
> > the next available VLAN range, the next available public IP range and
> > so on. The returned values are used to populate a Marvin cfg file
> > which in turn will be used to both build the environment and when
> > running the Marvin testing. When the virtualised infra is cleaned up,
> > the database will be updated to reflect that the used ranges are
> available again.
> > This initiative has only recently been started, and as stated earlier
> > we are currently figuring out the requirements (and quirks) of the
> > individual pieces and looking for the most suitable wrapper to glue it
> all together.
> > Also I have found that Marvin requires a little work to make the
> > output more meaningful/readable (especially in the case of errors and
> > exceptions) and to make it a little more intelligent about the tests
> > it can/can't run based on the chosen infrastructure components. I have
> > also found unreachable or very slow ISO and template paths hardcoded
> > into Marvin or individual tests.
> > We plan to enhance tests to address these issues and also reduce
> > runtimes where possible.
> >
> >
> > <http://www.shapeblue.com/>
> > Paul Angus
> > VP Technology , ShapeBlue
> >
> >
> > d: +44 203 617 0528 | s: +44 203 603 0540
> <tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>
> > | m: +44 7711 418784<tel:+44%207711%20418784>
> >
> > e: paul.angus@shapeblue.com | t: @cloudyangus<mailto:
> > paul.angus@shapeblue.com%20|%20t:%20@cloudyangus> | w:
> > www.shapeblue.com<http://www.shapeblue.com/>
> >
> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> >
> >
> >
> >
> >
> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > Services India LLP is a company incorporated in India and is operated
> > under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > is a company incorporated in Brasil and is operated under license from
> > Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > Republic of South Africa and is traded under license from Shape Blue
> > Ltd. ShapeBlue is a registered trademark.
> > This email and any attachments to it may be confidential and are
> > intended solely for the use of the individual to whom it is addressed.
> > Any views or opinions expressed are solely those of the author and do
> > not necessarily represent those of Shape Blue Ltd or related
> > companies. If you are not the intended recipient of this email, you
> > must neither take any action based upon its contents, nor copy or show
> > it to anyone. Please contact the sender if you believe you have received
> this email in error.
> >
> >
> >
> >
> > -----Original Message-----
> > From: Bharat Kumar [mailto:bharat.kumar@citrix.com]
> > Sent: Thursday, February 11, 2016 9:57 AM
> > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > Subject: Re: Important Pending Items
> >
> > Hi Sebastien,
> >
> > As Raja said, we are actively working on it and we will share the code
> > on github soon.
> >
> > Thanks,
> > Bharat.
> >
> >
> > > On 11-Feb-2016, at 2:41 PM, Raja Pullela <raja.pullela@citrix.com
> > <ma...@citrix.com>> wrote:
> > >
> > > Hi Sebastien,
> > >
> > > On item (3) - the BVTs have been running consistently (with same
> > passrates) and haven't had the time to automate the process of
> > reporting to an external site.
> > > Bharat and Sanjeev are working on CI - running BVTs/Regression,
> > > working
> > on getting it run automatically and consistently.
> > > Hope to see it ready soon.
> > >
> > > Best,
> > > Raja
> > > -----Original Message-----
> > > From: Sebastien Goasguen [mailto:runseb@gmail.com]
> > > Sent: Wednesday, February 10, 2016 2:29 PM
> > > To: dev@cloudstack.apache.org<ma...@cloudstack.apache.org>
> > > Subject: Important Pending Items
> > >
> > > Morning folks,
> > >
> > > We have several crucial pending items, that we need to resolve
> > > before
> > moving on:
> > >
> > > 1- We need an RM for master ( just saw some commits there that
> > > should be
> > reverted or merged properly in other branches).
> > >
> > > 2- We need to automate writing release notes, pushing/tagging new
> > > docs
> > when release come out and announcing releases on website. Currently
> > neither
> > 4.7 nor 4.8 have been announced.
> > >
> > > 3- CI is still almost inexistent
> > >
> > > -Sebastien
> >
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build<
> > http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework<http://shapeblue.com/csforge/>
> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack Software Engineering<
> > http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support<
> > http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
> >
> >
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>