[Testing Convention] Stress tests: requirements review needed

["Shipilov, Alexander D" <al...@intel.com>]

Hi Alex,

Alex Orlov wrote,
>BTW do we have any way to include stress/reliability tests in the
project?

We still don't have a solution about stress/reliability testing in
Harmony. I think this is important part of testing if we want to enable
Enterprise servers running on Harmony stack for years.

Let me summarize requirements for stress/reliability test suite. Simply
speaking, stress testing puts VM functionality under stress conditions.

The main requirement is the following
o	The system should generate relevant bugs 
It means either reproducible crashes, or severe degradations compared to
the reference implementation (RI). For example, if our tests 100% pass
on RI, we can use failure rate as such metrics to indicate problematic
points.

To get good coverage of VM functionality we need to have a possibility
of using existence functional and unit tests for stress/reliability
testing. So we get the following requirement
o	System should have an ability to stress any test from functional
test suite. 

Since we need a system which is easy to maintain, I suggest the
following requirement
o	System should have a possibility to vary test behavior and
stress conditions using arguments 
Actually I'm thinking here about a test suite built from a simple blocks
like small and fast unit tests, or functional tests
 
o	Test execution time should be adjustable 
This is an analogue of "Keep tests small and fast" for this test suite.

I also get the following requirements from people around my working
place.
o	Maintenance effort should be low.
o	Test code should be user-friendly, e.g. well documented, with
good log messages. This helps to analyze failures a lot.

Thoughts?

Thanks,
Alexander Shipilov,
Intel Middleware Products Division

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org