Testing Apache Cloud Stack versions with an (existing) reference environment and CI/CD test automation - how to?

[pe...@zv.fraunhofer.de]

Hi there,

@Dan & all dealing with testing and test automation in the Apache CloudStack context:

the concern driving the motivation for this message is a long term goal to be able to quickly validate upcoming Apache Cloud Stack versions as well as other changes in the complex heterogeneous cloud environment in terms of interoperability with our environment where versions of single components will change over time as well:

- Hypervisors: KVM and vmware vCenter
- Storage backend: NetApp

Image the following scenarios:

- a new version of ACS is installed as a test instance
- how to make sure that for example fixed bugs apply also in our environment while previous set of features is stable? => run acceptance tests

What I've learnt so far about what is feasible and available in this context:

- we use our custom set of scripts for functional testing of typical user scenarios [1]
- there is a mixed testing setup consisting of generators for test environment and also an integration testing plan, addressing and enabling new developers to build and test local environments [2]

The following concerns appear though to be yet ambiguous:

- a. it seems like there is no CI/CD automation for what is described in [2] aka MonkeyBox, additionally there is a recommendation not to use nested virtualization but rather bare metal test environment?
- b. instructions in [2] and also [3] refer to a test plan name "Marvin" which is as I've understood simultaneously obsolete; if so, what is the current test plan to follow?
- c. as it seems, there is no clear instruction/scenario how to install (in a CI/CD environment say using a standard Docker image) and run an up-to-date test plan against an existing ACS endpoint, that is without the environment generator step?

To have some focus and a problem to solve, it would be great to have a conversation/exchange with the community towards the question "c" above towards elaborating a to "to-go" test plan setup for easy interoperability sanity checks upon changes in the software and hardware environment components. Also understanding the current situation for the questions "a" and "b" would be great.

[1] https://gitlab.cc-asp.fraunhofer.de/fcs-public/apache-cs-extensions/tree/master/csextensions
[2] https://github.com/rhtyd/monkeybox 
[3] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Testing

--
Peter Muryshkin
Fraunhofer-Gesellschaft e.V.

Re: Testing Apache Cloud Stack versions with an (existing) reference environment and CI/CD test automation - how to?

[rv...@privaz.io.INVALID]

Hi Peter,

I have been working on automated ACS with ansible. Trying to achieve full automation from metal to service.

I have setup gitlab to run CI/CD of each commit of my work. It deploys an ACS instance in order to test.

It is a bit tricky to setup, but once done, it works as one should expect. 

With each commit installing an ACS and ansible running tests on it...etc.

I am happy to share my work if that may help.

Regards,
Rafael

On Thu, 2020-11-19 09:43 AM, peter.muryshkin@zv.fraunhofer.de wrote:
> 
Hi there,
> 
> @Dan & all dealing with testing and test automation in the Apache CloudStack context:
> 
> the concern driving the motivation for this message is a long term goal to be able to quickly validate upcoming Apache Cloud Stack versions as well as other changes in the complex heterogeneous cloud environment in terms of interoperability with our environment where versions of single components will change over time as well:
> 
> - Hypervisors: KVM and vmware vCenter
> - Storage backend: NetApp
> 
> Image the following scenarios:
> 
> - a new version of ACS is installed as a test instance
> - how to make sure that for example fixed bugs apply also in our environment while previous set of features is stable? => run acceptance tests
> 
> What I've learnt so far about what is feasible and available in this context:
> 
> - we use our custom set of scripts for functional testing of typical user scenarios [1]
> - there is a mixed testing setup consisting of generators for test environment and also an integration testing plan, addressing and enabling new developers to build and test local environments [2]
> 
> The following concerns appear though to be yet ambiguous:
> 
> - a. it seems like there is no CI/CD automation for what is described in [2] aka MonkeyBox, additionally there is a recommendation not to use nested virtualization but rather bare metal test environment?
> - b. instructions in [2] and also [3] refer to a test plan name "Marvin" which is as I've understood simultaneously obsolete; if so, what is the current test plan to follow?
> - c. as it seems, there is no clear instruction/scenario how to install (in a CI/CD environment say using a standard Docker image) and run an up-to-date test plan against an existing ACS endpoint, that is without the environment generator step?
> 
> To have some focus and a problem to solve, it would be great to have a conversation/exchange with the community towards the question "c" above towards elaborating a to "to-go" test plan setup for easy interoperability sanity checks upon changes in the software and hardware environment components. Also understanding the current situation for the questions "a" and "b" would be great.
> 
> [1] https://gitlab.cc-asp.fraunhofer.de/fcs-public/apache-cs-extensions/tree/master/csextensions
> [2] https://github.com/rhtyd/monkeybox 
> [3] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Testing
> 
> --
> Peter Muryshkin
> Fraunhofer-Gesellschaft e.V.
> 
> 
> 
>