python tests patch

[Emmanuel Okyere <eo...@gmail.com>]

Hello all:

My name is Emmanuel, and I am currently doing some work on a  
Javascript client for rabbitmq. Currently, as part of the rabbit  
automated tests, the qpid python client tests are run against the  
rabbit server.

I noticed that:

1. after specifying the spec file to run against the server, the  
tests automatically include those in the python/tests folder  
(although you can use the ignore file to configure what tests are run)
2. the "../specs" folder is assumed (our specs folder resides  
in ../../docs/specs, from where we run our tests, so this causes an  
error)

I have made local changes that:

a. allows the 'specs' folder to be passed by a command option -F/-- 
spec-folder (defaults to ../specs)
b. gives the option to run the 'default' tests in the 'tests' folder,  
with command option -D/--default-tests; if -F above is passed, -D is  
assumed
c. if -F and/or -D is/are not passed, then it runs only the tests  
specified by the passed spec file

Please find these attached as a patch if these changes are useful.

Thanks,
Emmanuel.


  
  

Re: python tests patch

[Gordon Sim <gs...@redhat.com>]

Matthias Radestock wrote:
> That used to be the case, but the tests under tests/* reference *all* 
> (in case of spec.py) or specific ("amqp.0-10-preview.xml" in codec.py) 
> specs. Instead they should all operate on the the spec that was 
> specified on the command line.
> 
> Emmanuel's patch doesn't fix that, but it allows you to exclude the 
> tests/* and hence work around the problem. Note that using an exclusion 
> file doesn't work because codec.py still gets loaded and the hard-coded 
> spec file is referenced as part of that.

Thanks for the clarification Matthias, I understand the problem now!

Re: python tests patch

[Rafael Schloming <ra...@redhat.com>]

Matthias Radestock wrote:
> Gordon,
> 
> Gordon Sim wrote:
> 
>>> 2. the "../specs" folder is assumed (our specs folder resides in 
>>> ../../docs/specs, from where we run our tests, so this causes an error)
>>
>> Thats true for using the short cut '0-8' etc as the spec parameter. 
>> However you can pass in any set of files you like (e.g. run-tests -s 
>> ../my_path/my_file.xml -e ../another_path/another_file.xml)
> 
> That used to be the case, but the tests under tests/* reference *all* 
> (in case of spec.py) or specific ("amqp.0-10-preview.xml" in codec.py) 
> specs. Instead they should all operate on the the spec that was 
> specified on the command line.

The intent of these tests is to verify that the python client code 
itself is functioning as expected, and as the python client code is 
supposed to be able to load any version of the spec file, this code must 
test all those versions.

The spec file specified on the command line describes the specific 
protocol that the broker speaks, and it could really be any protocol 
that is specified using the 0-8, 0-9, or 0-10 grammars. Using this in 
the spec.py test wouldn't be correct as the spec.py test checks that 
specific classes and methods from the 0-8, 0-9, and 0-10 versions of the 
spec are loaded correctly according to the various grammars, and there 
is no guarantee that those classes and methods would be present in the 
spec file passed on the command line.

> Emmanuel's patch doesn't fix that, but it allows you to exclude the 
> tests/* and hence work around the problem. Note that using an exclusion 
> file doesn't work because codec.py still gets loaded and the hard-coded 
> spec file is referenced as part of that.

Hopefully the patch will still work for you with the modification I 
made. Please let me know if it does not.

--Rafael

Re: python tests patch

[Matthias Radestock <ma...@lshift.net>]

Gordon,

Gordon Sim wrote:

>> 2. the "../specs" folder is assumed (our specs folder resides in 
>> ../../docs/specs, from where we run our tests, so this causes an error)
> 
> Thats true for using the short cut '0-8' etc as the spec parameter. 
> However you can pass in any set of files you like (e.g. run-tests -s 
> ../my_path/my_file.xml -e ../another_path/another_file.xml)

That used to be the case, but the tests under tests/* reference *all* 
(in case of spec.py) or specific ("amqp.0-10-preview.xml" in codec.py) 
specs. Instead they should all operate on the the spec that was 
specified on the command line.

Emmanuel's patch doesn't fix that, but it allows you to exclude the 
tests/* and hence work around the problem. Note that using an exclusion 
file doesn't work because codec.py still gets loaded and the hard-coded 
spec file is referenced as part of that.


Matthias.

Re: python tests patch

[Gordon Sim <gs...@redhat.com>]

Emmanuel Okyere wrote:
> Hello all:
> 
> My name is Emmanuel, and I am currently doing some work on a Javascript 
> client for rabbitmq. Currently, as part of the rabbit automated tests, 
> the qpid python client tests are run against the rabbit server.

Hi Emmanuel! A Javascript client sounds great.

> I noticed that:
> 
> 1. after specifying the spec file to run against the server, the tests 
> automatically include those in the python/tests folder (although you can 
> use the ignore file to configure what tests are run)

Yes, there was some debate on the list about that. As you say you can 
always add those tests to the ignore file, but perhaps a simpler option 
for turning of the default tests would indeed be useful.

> 2. the "../specs" folder is assumed (our specs folder resides in 
> ../../docs/specs, from where we run our tests, so this causes an error)

Thats true for using the short cut '0-8' etc as the spec parameter. 
However you can pass in any set of files you like (e.g. run-tests -s 
../my_path/my_file.xml -e ../another_path/another_file.xml)

> I have made local changes that:
> 
> a. allows the 'specs' folder to be passed by a command option 
> -F/--spec-folder (defaults to ../specs)
> b. gives the option to run the 'default' tests in the 'tests' folder, 
> with command option -D/--default-tests; if -F above is passed, -D is assumed
> c. if -F and/or -D is/are not passed, then it runs only the tests 
> specified by the passed spec file
> 
> Please find these attached as a patch if these changes are useful.

Rafi is really the python maintainer; he's not around this week but will 
be next week. Thanks for the patch though!

Re: python tests patch

[Rafael Schloming <ra...@redhat.com>]

Emmanuel Okyere wrote:
> Hello all:
> 
> My name is Emmanuel, and I am currently doing some work on a Javascript 
> client for rabbitmq. Currently, as part of the rabbit automated tests, 
> the qpid python client tests are run against the rabbit server.
> 
> I noticed that:
> 
> 1. after specifying the spec file to run against the server, the tests 
> automatically include those in the python/tests folder (although you can 
> use the ignore file to configure what tests are run)
> 2. the "../specs" folder is assumed (our specs folder resides in 
> ../../docs/specs, from where we run our tests, so this causes an error)
> 
> I have made local changes that:
> 
> a. allows the 'specs' folder to be passed by a command option 
> -F/--spec-folder (defaults to ../specs)
> b. gives the option to run the 'default' tests in the 'tests' folder, 
> with command option -D/--default-tests; if -F above is passed, -D is assumed
> c. if -F and/or -D is/are not passed, then it runs only the tests 
> specified by the passed spec file
> 
> Please find these attached as a patch if these changes are useful.

Thanks for the patch, I have applied it to the trunk. I modified the 
options slightly to make the "tests" code always run by default:

./run-tests -h

run-tests [options] [test*]
The name of a test is package.module.ClassName.testMethod
Options:
   -?/-h/--help             : this message
   -s/--spec <spec.xml> : URL of AMQP XML specification or one of these 
abbreviations:
                            0-8 - use the default 0-8 specification.
                            0-9 - use the default 0-9 specification.
   -e/--errata <errata.xml> : file containing amqp XML errata
   -b/--broker [<user>[/<password>]@]<host>[:<port>] : broker to connect to
   -v/--verbose             : verbose - lists tests as they are run.
   -d/--debug               : enable debug logging.
   -i/--ignore <test>       : ignore the named test.
   -I/--ignore-file         : file containing patterns to ignore.
   -S/--skip-self-test      : skips the client self tests in the 'tests 
folder'
   -F/--spec-folder         : folder that contains the specs to be loaded

I did this because the code under the tests dir checks that the python 
client itself is functioning as expected (and never interacts with the 
broker), it is therefore useful to run these by default regardless of 
which set of broker tests are being run (tests_0-8, tests_0-9, or 
tests_0-10) as a failure in the self-test suite would indicate that 
failures in the broker suites are not necessarily the fault of the 
broker being tested, and could be due to a problem in the python client 
code.

--Rafael