Declarative Creation of Test Input for AutoScaler policy

[Alessio Gambi <al...@gmail.com>]

Hi all,

I am doing a (research) work about the use of a declarative approach to define test inputs for the testing of AutoScaler-like policies,
and I apply this to the brooklyn code.

I would like ear your takes over it and opinions on the idea. Feedbacks, suggestions for improvements and wishes/requests from the
community are welcome as well, and definitively will help me improving the work. That soon might be released as well.

A short summary of the idea is below:

================================================================
To test scaling policies a common pattern:
	setups an application,
	emits some sensor data, 
	checks for some specific outcome.

In particular, sensor data, i.e., test inputs, are manually provided. 

This is OK but can be improved.

A possible way to go, which is the core part of my work, is to define these test inputs by describing how they look like,
instead of manually providing actual, hard-coded input data. ( That is why the approach is called declarative).

For example, we can ask a test driver to provide us with a value to resize the system from 1 to 4 given a specific
configuration of the autoscaling policy,  so instead of writing (quoted from an actual test case)

// workload 200 so requires doubling size to 10 to handle: (200*5)/100 = 10
tc.setAttribute(MY_ATTRIBUTE, 200);

We could write something like:

tc.setAttribute(MY_ATTRIBUTE, mock.resize(5,10));

And expect at execution time that the call "mock.resize(5,10)" produces a value (included - but not limited to - 200) that will trigger the AutoScalerPolicy
to resize the application from 5 to 10. 

Similarly, we can also think of something like :

mock.createScenario({mock.SCALE_UP, mock.SCALE_DOWN, mock.SCALE_DOWN});

And expect a sequence of values (with time stamp, i.e., a trace-like) that can force the policy to perform a first scale up followed by two different scale down.

Or even:

mock.createPeriodicScenario(monitoringPeriod, {mock.SCALE_UP, mock.SCALE_DOWN, mock.SCALE_DOWN});

To generate a stream (or a big array) that produces a set of sensor data that are periodically emitted.
================================================================

At the moment, I studied the available test cases for the AutoScalerPolicy and extrapolated some of the main requirements
(from the testing point of view). This allowed me to generate “declaratively" test inputs for testing

	- normal resize operations triggered by emits
	- abnormal resize over/under the max/min poll size (that must not happen)
	- concurrent action/emit
	- sustained action/emit (Different HOT/COLD_SENSOR messages within the same stability period)
	- “blips” in the emit values (HOT_SENSOR followed by OK_SENSOR)

BTW, I integrated the code with the testing/eclipse setup so I can run the actual java tests.

This is a nice result in my opinion, but I can improve it further more … with your help !

Other than general comments (as mentioned at the beginning of this email) I am also seeking additional requests or suggestions
on possible (new) tests that "could have been defined” or “it would be beneficial for the quality of the project to have” by employing this technology.

I am pretty sure that you can provide really relevant comments !

Thanks to everybody !

Best,
— Alessio

PS: I am usually connected to the brooklyn channel on IRC, so you can also contact me there