Grouping tests (was: Latest tests)

[Mike Looijmans <nl...@natlab.research.philips.com>]

What I have been doing in a totally unrelated Python project is to create test groups simply by 
putting them into separate modules. The main test module test.py looks like this:

## (test.py)
import unittest
from test_something import *
from test_someother import *
from test_yetmore import *
if __name__ == '__main__':
     unittest.main(module=__import__(__name__))
##

This works because unittest takes all 'things' that start with 'test' in the provided module and 
runs them. So anything we bring into our namespace gets run. This also makes it possible to import 
tests from other projects, and share these tests between projects.
The other test_ modules look much alike:

## (test_something.py)
import unittest
import test_peer
class test04MultiPeerSystem(test_peer.BaseTestRealThing):
     def test03DiscoveryC08B20(self):
         "Multiple clients w/ discovery:  8 peers, 20 blocks each"
         self.runDiscoveryTest(nclients = 8, nblocks=20)
if __name__ == '__main__':
     unittest.main(module=__import__(__name__))
##

This makes it very easy to handle test subsets, and run single test suites. Just run

$python test.py

to run ALL the tests. To run just a single set, run

$python test_something.py

And to run a single test, either of these will do:

$python test_something.py test04MultiPeerSystem
$python test.py test04MultiPeerSystem

The real power shows when you want to run 4 or 5 test sets, and/or only parts of some test sets. 
Just create a new "main" test unit that imports the desired ones, and you're set:

## (test_few.py)
import unittest
from test_something import *
from test_someother import TestOnlyThis
if __name__ == '__main__':
     unittest.main(module=__import__(__name__))
##

Because some tests take very long to run (in my vocabulary, "long" means more than a second), this 
saves me a lot of time when working on a part of a big project, where I don't need to run all tests 
all the time.

--
Mike Looijmans
Philips Natlab / Topic Automation


Jim Gallacher wrote:
...
> I've been playing with some ideas for a new test framework, using a 
> subclass of unittest.TestLoader to find and configure tests. I want to 
> play around with it for another day or so before sharing but at this 
> point I'm pretty confident it'll work.
> 
> Creating a new set of tests could be as simple as:
> 
> testsuites/core/simpletest.py
> ----------------------------
> 
> from testlib import VirtualHostTest
> 
> class MyTests(VirtualHostTest):
>     def test_hello_world(self):
>         rsp = self.send_request()
>         self.failUnless(rsp == 'test ok')
> 
>     def test_goodbye_world(self):
>         rsp = self.send_request()
>         self.failUnless(rsp == 'test ok')
> 
> 
> htdocs/testhandlers/core/simpletest.py
> --------------------------------------
> 
> from mod_python import
> 
> def test_hello_world(req):
>     req.write('test ok')
>     return apache.OK
> 
> def test_goodbye_world(req):
>     req.write('test ok')
>     return apache.OK
> 
> $ python testrunner.py
> 
> Things like virtual host names and handler directives required for 
> configuration or send_request() are automatically derived from the test 
> class and test method names. It will still be possible to provide custom 
> apache configuration directives in a manner similar to that which we 
> currently use, but for most tests this will not be required.
> 
> Usage would look something like this:
> 
> Run all the tests
> $ python testrunner.py
> 
> Run one test
> $ python testrunner.py -t core.simpletest.MyTests.test_hello_world
> 
> Run a group of tests (this would load the TestCase subclasses in 
> testsuites/sessions/filesession.py):
> 
> $ python testrunner.py -t sessions.filesession
> 
> 
> Jim

Re: Grouping tests

[Jim Gallacher <jp...@jgassociates.ca>]

This isn't too different from what I have in mind, except that my main 
testrunner.py script uses introspection to discover the tests. Adding a 
new test set won't require editing testrunner at all. The testrunner 
will also filter the tests according to certain criteria, such as mpm or 
os, so that tests for a specific platform are automatically selected.

More later.

Jim


Mike Looijmans wrote:
> What I have been doing in a totally unrelated Python project is to 
> create test groups simply by putting them into separate modules. The 
> main test module test.py looks like this:
> 
> ## (test.py)
> import unittest
> from test_something import *
> from test_someother import *
> from test_yetmore import *
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> This works because unittest takes all 'things' that start with 'test' in 
> the provided module and runs them. So anything we bring into our 
> namespace gets run. This also makes it possible to import tests from 
> other projects, and share these tests between projects.
> The other test_ modules look much alike:
> 
> ## (test_something.py)
> import unittest
> import test_peer
> class test04MultiPeerSystem(test_peer.BaseTestRealThing):
>     def test03DiscoveryC08B20(self):
>         "Multiple clients w/ discovery:  8 peers, 20 blocks each"
>         self.runDiscoveryTest(nclients = 8, nblocks=20)
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> This makes it very easy to handle test subsets, and run single test 
> suites. Just run
> 
> $python test.py
> 
> to run ALL the tests. To run just a single set, run
> 
> $python test_something.py
> 
> And to run a single test, either of these will do:
> 
> $python test_something.py test04MultiPeerSystem
> $python test.py test04MultiPeerSystem
> 
> The real power shows when you want to run 4 or 5 test sets, and/or only 
> parts of some test sets. Just create a new "main" test unit that imports 
> the desired ones, and you're set:
> 
> ## (test_few.py)
> import unittest
> from test_something import *
> from test_someother import TestOnlyThis
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> Because some tests take very long to run (in my vocabulary, "long" means 
> more than a second), this saves me a lot of time when working on a part 
> of a big project, where I don't need to run all tests all the time.
> 
> -- 
> Mike Looijmans
> Philips Natlab / Topic Automation
> 
> 
> Jim Gallacher wrote:
> ...
> 
>> I've been playing with some ideas for a new test framework, using a 
>> subclass of unittest.TestLoader to find and configure tests. I want to 
>> play around with it for another day or so before sharing but at this 
>> point I'm pretty confident it'll work.
>>
>> Creating a new set of tests could be as simple as:
>>
>> testsuites/core/simpletest.py
>> ----------------------------
>>
>> from testlib import VirtualHostTest
>>
>> class MyTests(VirtualHostTest):
>>     def test_hello_world(self):
>>         rsp = self.send_request()
>>         self.failUnless(rsp == 'test ok')
>>
>>     def test_goodbye_world(self):
>>         rsp = self.send_request()
>>         self.failUnless(rsp == 'test ok')
>>
>>
>> htdocs/testhandlers/core/simpletest.py
>> --------------------------------------
>>
>> from mod_python import
>>
>> def test_hello_world(req):
>>     req.write('test ok')
>>     return apache.OK
>>
>> def test_goodbye_world(req):
>>     req.write('test ok')
>>     return apache.OK
>>
>> $ python testrunner.py
>>
>> Things like virtual host names and handler directives required for 
>> configuration or send_request() are automatically derived from the 
>> test class and test method names. It will still be possible to provide 
>> custom apache configuration directives in a manner similar to that 
>> which we currently use, but for most tests this will not be required.
>>
>> Usage would look something like this:
>>
>> Run all the tests
>> $ python testrunner.py
>>
>> Run one test
>> $ python testrunner.py -t core.simpletest.MyTests.test_hello_world
>>
>> Run a group of tests (this would load the TestCase subclasses in 
>> testsuites/sessions/filesession.py):
>>
>> $ python testrunner.py -t sessions.filesession
>>
>>
>> Jim
> 
> 
>