End to end test framework for Stratos

[Udara Liyanage <ud...@wso2.com>]

Hi Lakmal and Devs,

Me and Chamila had a offline talk about having a end to end testing framework
for Stratos. The high level idea is we expose a framework kind of thing
where setup, teardown and tests methods can be written by tester.
We can wrapup Stratos REST API and expose APIs such as deploy policies ,
partitions and cartridges etc. So a tester can call those high level APIs
and write the test.


Additionally IAAS configurations can be provided so when tests are executed
Stratos would be up and configured and instances would be spin up in
configured IaaS.

We discussed what to uses underlying (docker, vagrant etc), yet could not
come up with a 100% clear picture.

As an startup, simple use case such as subscribing to a cartridge and check
wheterh instance got spin up and cluster get activated , can be automated

@Chamila please add your comments too.

Please share your thoughts.



Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware

web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897

Re: End to end test framework for Stratos

[Chamila De Alwis <ch...@wso2.com>]

Hi,


*Functional testing*

*The choice of IaaS* - We discussed the possibility of Vagrant, Docker and
OpenStack (with or without Docker provider) and decided we needed to
include both an IaaS and a container manager to cover a majority of the
code paths.  Once the Kubernetes setup is stable we can go ahead on a more
detailed design on how to use Kubernetes + Docker to enable container based
testing. For OpenStack we are still unsure of whether to provide an
OpenStack Vagrant box or use a common resource such as the OpenStack
deployment we have for Stratos. Alternatively we could use EC2 as our IaaS.


*Enabling a framework* - What we have in mind is a setup where the test
author writes *setup*, *test*, and *teardown* methods to (ex:) configure
stratos and define catridges, subscribe to cartridges and carry out tests,
and unsubscribe from cartridges, respectively in a test case automation
(for example, with Python). The testing framework will call these methods
in their order and also provide resourceful APIs to interact with the
Stratos installation, IaaS provider, and the container manager.


*Component Testing*

We considered the possibility of testing each component separately, and
this can also be achieved too. The test author can define the input data
and the expected outcome (or a series of input data and their respective
expected outcomes) as JSON files and start the execution of a test run,
which will provide him/her with the final results. We are still in the
process of researching on this structure for all the components, however we
strongly feel there is the possibility.


*Infrastructure*

We think the test infrastructure (ex: Vagrant boxes with Stratos +
OpenStack, Stratos + Docker, Stratos + EC2, etc) would be possible to be
distributed so that each individual can run the framework on his/her own.
However we could also go for a centralized infrastructure where the IaaS
and the container management is setup as a service to be used by any test
run. This could provide us with a centralized reporting on test success,
code coverage etc.


​
*Use of Python*
As far as I know, the Py.Test framework for Python allows defining a test
declarative language that can be used to express input and expected output
values in YAML format[1]. This needs to be further investigated before
consolidating, but IMO there is a good potential for Stratos to make use of
this to come up with input (policies, events etc) and output (mostly
events) values.

WDYT?

[1] - http://pytest.org/latest/example/nonpython.html


Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com



On Fri, Oct 31, 2014 at 3:26 PM, Udara Liyanage <ud...@wso2.com> wrote:

> Hi Lakmal and Devs,
>
> Me and Chamila had a offline talk about having a end to end testing framework
> for Stratos. The high level idea is we expose a framework kind of thing
> where setup, teardown and tests methods can be written by tester.
> We can wrapup Stratos REST API and expose APIs such as deploy policies ,
> partitions and cartridges etc. So a tester can call those high level APIs
> and write the test.
>
>
> Additionally IAAS configurations can be provided so when tests are
> executed Stratos would be up and configured and instances would be spin up
> in configured IaaS.
>
> We discussed what to uses underlying (docker, vagrant etc), yet could not
> come up with a 100% clear picture.
>
> As an startup, simple use case such as subscribing to a cartridge and
> check wheterh instance got spin up and cluster get activated , can be
> automated
>
> @Chamila please add your comments too.
>
> Please share your thoughts.
>
>
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>