RE: [DISCUSS] Marvin tests interaction

[pa...@apache.org]

I've got 6 cents to offer here;

2c - It could be good to add @peter's style of tests and indeed those
specific tests into the testing regime.

2c - Every new test is being added to the 'smoke test' pile.  Really, the
smoke tests 'scope' should be clearly (re)defined, and anything that doesn't
fit moved out to a 'more in-depth tests' set.  
       I'd have a third category for 'external integration tests', ie Veeam,
SolidFire, Juniper SRX, DPDK - anything that is going to need some kind of
additional setup/hardware.

2c - The existing 'component tests' set needs a whole load of love.


Kind regards

Paul Angus


-----Original Message-----
From: peter.muryshkin@zv.fraunhofer.de <pe...@zv.fraunhofer.de> 
Sent: Wednesday, March 31, 2021 12:39 PM
To: dev@cloudstack.apache.org; nicolas.vazquez@shapeblue.com
Subject: AW: [DISCUSS] Marvin tests interaction

Hi there Nicolas,

why is actually interaction important in this context? I read that you want
to enable developers to quickly run and check the results.
For my own scenario I had a requirement to have a set of functional tests
only seeing ACS as a blackbox to have a "sanity check" of the service, where
Marvin is then an overkill as it runs integration tests looking deeply into
the database etc.
So it was indeed important to have results fast. My appoach has been to
write a set of Python tests using the exoscale/cs library for most common
API usage scenarios, and then I run them in our GitLab instance using the
dynamic parallel CI/CD jobs feature, one separate job for each test, that is
the test runtime complexity is O(1) and not O(n).

The "UI" I can then interact with is GitLab CI/CD pipeline dashboard which
allows to quickly identify failed tests.

Here is the link to the codebase.
https://gitlab.cc-asp.fraunhofer.de/fcs-public/apache-cs-extensions

kind regards
Peter
________________________________________
Von: Nicolas Vazquez <Ni...@shapeblue.com>
Gesendet: Montag, 29. März 2021 04:50:44
An: dev@cloudstack.apache.org
Betreff: [DISCUSS] Marvin tests interaction

Hi,

I would like to propose an idea to improve the interaction with the marvin
tests through the management server. This could be useful for development
and test environments in which tests could be easily started, configured and
their results monitored through the UI.

This could be achieved by creating a new service in charge of the execution
of the tests and sending results back to the management server, so it can
display them. A more detailed description:
https://github.com/apache/cloudstack/issues/4799

I would like to hear your thoughts and ideas about it. Would you find this
useful?


Regards,

Nicolas Vazquez

Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue