Re: Grouping tests

[Jim Gallacher <jp...@jgassociates.ca>]

This isn't too different from what I have in mind, except that my main 
testrunner.py script uses introspection to discover the tests. Adding a 
new test set won't require editing testrunner at all. The testrunner 
will also filter the tests according to certain criteria, such as mpm or 
os, so that tests for a specific platform are automatically selected.

More later.

Jim


Mike Looijmans wrote:
> What I have been doing in a totally unrelated Python project is to 
> create test groups simply by putting them into separate modules. The 
> main test module test.py looks like this:
> 
> ## (test.py)
> import unittest
> from test_something import *
> from test_someother import *
> from test_yetmore import *
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> This works because unittest takes all 'things' that start with 'test' in 
> the provided module and runs them. So anything we bring into our 
> namespace gets run. This also makes it possible to import tests from 
> other projects, and share these tests between projects.
> The other test_ modules look much alike:
> 
> ## (test_something.py)
> import unittest
> import test_peer
> class test04MultiPeerSystem(test_peer.BaseTestRealThing):
>     def test03DiscoveryC08B20(self):
>         "Multiple clients w/ discovery:  8 peers, 20 blocks each"
>         self.runDiscoveryTest(nclients = 8, nblocks=20)
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> This makes it very easy to handle test subsets, and run single test 
> suites. Just run
> 
> $python test.py
> 
> to run ALL the tests. To run just a single set, run
> 
> $python test_something.py
> 
> And to run a single test, either of these will do:
> 
> $python test_something.py test04MultiPeerSystem
> $python test.py test04MultiPeerSystem
> 
> The real power shows when you want to run 4 or 5 test sets, and/or only 
> parts of some test sets. Just create a new "main" test unit that imports 
> the desired ones, and you're set:
> 
> ## (test_few.py)
> import unittest
> from test_something import *
> from test_someother import TestOnlyThis
> if __name__ == '__main__':
>     unittest.main(module=__import__(__name__))
> ##
> 
> Because some tests take very long to run (in my vocabulary, "long" means 
> more than a second), this saves me a lot of time when working on a part 
> of a big project, where I don't need to run all tests all the time.
> 
> -- 
> Mike Looijmans
> Philips Natlab / Topic Automation
> 
> 
> Jim Gallacher wrote:
> ...
> 
>> I've been playing with some ideas for a new test framework, using a 
>> subclass of unittest.TestLoader to find and configure tests. I want to 
>> play around with it for another day or so before sharing but at this 
>> point I'm pretty confident it'll work.
>>
>> Creating a new set of tests could be as simple as:
>>
>> testsuites/core/simpletest.py
>> ----------------------------
>>
>> from testlib import VirtualHostTest
>>
>> class MyTests(VirtualHostTest):
>>     def test_hello_world(self):
>>         rsp = self.send_request()
>>         self.failUnless(rsp == 'test ok')
>>
>>     def test_goodbye_world(self):
>>         rsp = self.send_request()
>>         self.failUnless(rsp == 'test ok')
>>
>>
>> htdocs/testhandlers/core/simpletest.py
>> --------------------------------------
>>
>> from mod_python import
>>
>> def test_hello_world(req):
>>     req.write('test ok')
>>     return apache.OK
>>
>> def test_goodbye_world(req):
>>     req.write('test ok')
>>     return apache.OK
>>
>> $ python testrunner.py
>>
>> Things like virtual host names and handler directives required for 
>> configuration or send_request() are automatically derived from the 
>> test class and test method names. It will still be possible to provide 
>> custom apache configuration directives in a manner similar to that 
>> which we currently use, but for most tests this will not be required.
>>
>> Usage would look something like this:
>>
>> Run all the tests
>> $ python testrunner.py
>>
>> Run one test
>> $ python testrunner.py -t core.simpletest.MyTests.test_hello_world
>>
>> Run a group of tests (this would load the TestCase subclasses in 
>> testsuites/sessions/filesession.py):
>>
>> $ python testrunner.py -t sessions.filesession
>>
>>
>> Jim
> 
> 
>