Proposal to consolidate qpid benchmark tests

[Irina Boverman <ib...@redhat.com>]

Currently there are 2 tests, cpp and java/jms, in respective source trees.

qpid/cpp/src/tests/qpid-cpp-benchmark
qpid/java/tools/bin/qpid-jms-benchmark

Both tests use similar options (with a couple of exceptions). They also share majority of code, the exceptions are calls to qpid-send, qpid-receive and few other minor differences.

There is a desire to provide one interface for all language implementations (as well as, extend implementation to other language clients). This goal can be accomplished in different ways:

Option 1:

Add a thin layer (new test) to check language preference and pass control to the corresponding test. In this case existing tests are preserved and continue to exist in the language specific source trees. The disadvantage is that we end up maintaining similar code in multiple places.

Option 2:

Consolidate tests into one source and place it in qpid/tests/src/py/qpid_tests/client/ location. The advantage is reduced maintenance, the disadvantage is that now one needs all qpid source to run a specific test.

My recommendation is to proceed with Option 2 based on reduced maintenance. 

I will appreciate community feedback regarding this proposal.
Regards, Irina.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Alan Conway <ac...@redhat.com>]

On 11/12/2013 02:29 PM, Irina Boverman wrote:
> This is my understanding of the discussion summary:
>
> Existing tests (qpid/cpp/src/tests/qpid-cpp-benchmark and qpid/java/tools/bin/qpid-jms-benchmark) be replaced by qpid/python/qpid-benchmark (using qpid-cpp-benchmark as a starting point). Options --qpid-send-path and --qpid-receive-path will default to "qpid-send" and "qpid-receive" and interpreted as path+binary when starting with "/" or just binary otherwise (and rely on env).
>

Yep, but you can simplify a bit and just pass the qpid_send_path value to Popen. 
Popen will check if it's a path or just an exe name and use the PATH environment 
if required. It will also do the right thing with windows vs. unix path 
separators and environment differences.

> Regards, Irina.
> ----- Original Message -----
> From: "Alan Conway" <ac...@redhat.com>
> To: dev@qpid.apache.org
> Sent: Monday, November 11, 2013 9:26:44 AM
> Subject: Re: Proposal to consolidate qpid benchmark tests
>
> On 11/11/2013 05:47 AM, Gordon Sim wrote:
>> On 11/08/2013 07:59 PM, Andrew Stitcher wrote:
>>> On Fri, 2013-11-08 at 14:24 -0500, Irina Boverman wrote:
>>>> Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to
>>>> me it is best to add "-l <lang>" option to the driver, and then use correct
>>>> method/options. It will not require renaming, nor passing method name as
>>>> another argument.
>>>
>>> It seems to me that it would make more sense to change the different
>>> programs to have the same options. Or at least to have a common subset
>>> of options that can be used by the benchmark/interop tester.
>>>
>>> It makes much more sense to me from an engineering perspective to
>>> define/accept a common interface to the programs than to add a bunch of
>>> special cases to the driving program.
>>
>> I very much agree.
>>
>
> +1
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Irina Boverman <ib...@redhat.com>]

This is my understanding of the discussion summary:

Existing tests (qpid/cpp/src/tests/qpid-cpp-benchmark and qpid/java/tools/bin/qpid-jms-benchmark) be replaced by qpid/python/qpid-benchmark (using qpid-cpp-benchmark as a starting point). Options --qpid-send-path and --qpid-receive-path will default to "qpid-send" and "qpid-receive" and interpreted as path+binary when starting with "/" or just binary otherwise (and rely on env).

Regards, Irina.
----- Original Message -----
From: "Alan Conway" <ac...@redhat.com>
To: dev@qpid.apache.org
Sent: Monday, November 11, 2013 9:26:44 AM
Subject: Re: Proposal to consolidate qpid benchmark tests

On 11/11/2013 05:47 AM, Gordon Sim wrote:
> On 11/08/2013 07:59 PM, Andrew Stitcher wrote:
>> On Fri, 2013-11-08 at 14:24 -0500, Irina Boverman wrote:
>>> Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to
>>> me it is best to add "-l <lang>" option to the driver, and then use correct
>>> method/options. It will not require renaming, nor passing method name as
>>> another argument.
>>
>> It seems to me that it would make more sense to change the different
>> programs to have the same options. Or at least to have a common subset
>> of options that can be used by the benchmark/interop tester.
>>
>> It makes much more sense to me from an engineering perspective to
>> define/accept a common interface to the programs than to add a bunch of
>> special cases to the driving program.
>
> I very much agree.
>

+1

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Alan Conway <ac...@redhat.com>]

On 11/11/2013 05:47 AM, Gordon Sim wrote:
> On 11/08/2013 07:59 PM, Andrew Stitcher wrote:
>> On Fri, 2013-11-08 at 14:24 -0500, Irina Boverman wrote:
>>> Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to
>>> me it is best to add "-l <lang>" option to the driver, and then use correct
>>> method/options. It will not require renaming, nor passing method name as
>>> another argument.
>>
>> It seems to me that it would make more sense to change the different
>> programs to have the same options. Or at least to have a common subset
>> of options that can be used by the benchmark/interop tester.
>>
>> It makes much more sense to me from an engineering perspective to
>> define/accept a common interface to the programs than to add a bunch of
>> special cases to the driving program.
>
> I very much agree.
>

+1

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Gordon Sim <gs...@redhat.com>]

On 11/08/2013 07:59 PM, Andrew Stitcher wrote:
> On Fri, 2013-11-08 at 14:24 -0500, Irina Boverman wrote:
>> Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to me it is best to add "-l <lang>" option to the driver, and then use correct method/options. It will not require renaming, nor passing method name as another argument.
>
> It seems to me that it would make more sense to change the different
> programs to have the same options. Or at least to have a common subset
> of options that can be used by the benchmark/interop tester.
>
> It makes much more sense to me from an engineering perspective to
> define/accept a common interface to the programs than to add a bunch of
> special cases to the driving program.

I very much agree.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Fraser Adams <fr...@blueyonder.co.uk>]

+1  from me on that train of thought.

I think that we should aspire wherever possible to maximise cohesion, 
there has been a lot of divergence for various reasons but I think with 
the increased moves to modularisation it's time to draw a line in the 
sand and try to bring things together again.

On 08/11/13 19:59, Andrew Stitcher wrote:
>
> It makes much more sense to me from an engineering perspective to
> define/accept a common interface to the programs than to add a bunch of
> special cases to the driving program.
>
> Andrew
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Andrew Stitcher <as...@redhat.com>]

On Fri, 2013-11-08 at 14:24 -0500, Irina Boverman wrote:
> Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to me it is best to add "-l <lang>" option to the driver, and then use correct method/options. It will not require renaming, nor passing method name as another argument.

It seems to me that it would make more sense to change the different
programs to have the same options. Or at least to have a common subset
of options that can be used by the benchmark/interop tester.

It makes much more sense to me from an engineering perspective to
define/accept a common interface to the programs than to add a bunch of
special cases to the driving program.

Andrew



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Irina Boverman <ib...@redhat.com>]

Qpid-send (cpp) and qpid-jms-send use slightly different options, it seems to me it is best to add "-l <lang>" option to the driver, and then use correct method/options. It will not require renaming, nor passing method name as another argument.
Regards, Irina.
----- Original Message -----
From: "Gordon Sim" <gs...@redhat.com>
To: dev@qpid.apache.org
Sent: Friday, November 8, 2013 12:54:14 PM
Subject: Re: Proposal to consolidate qpid benchmark tests

On 11/08/2013 05:44 PM, Justin Ross wrote:
> I'm all for a default.  If we had a qpid-send facade, I'd see no
> problem with defaulting the impl lang to C++.  I just think it's
> slightly better to avoid obscuring the language: going with qpid-send,
> qpid-java-send, and qpid-python-send leaves the user to guess what
> that first one is.

The 'first one' is the only one that is actually in use at present and 
has been for some time. I think the confusion is at present purely 
hypothetical.

The inconvenience caused by renaming the current qpid-send/qpid-receive 
would however be real (and did I mention these are the only ones in 
actual use :-).


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Gordon Sim <gs...@redhat.com>]

On 11/08/2013 05:44 PM, Justin Ross wrote:
> I'm all for a default.  If we had a qpid-send facade, I'd see no
> problem with defaulting the impl lang to C++.  I just think it's
> slightly better to avoid obscuring the language: going with qpid-send,
> qpid-java-send, and qpid-python-send leaves the user to guess what
> that first one is.

The 'first one' is the only one that is actually in use at present and 
has been for some time. I think the confusion is at present purely 
hypothetical.

The inconvenience caused by renaming the current qpid-send/qpid-receive 
would however be real (and did I mention these are the only ones in 
actual use :-).


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Justin Ross <ju...@gmail.com>]

On Fri, Nov 8, 2013 at 12:21 PM, Andrew Stitcher <as...@redhat.com> wrote:
>> For what it's worth (and I wouldn't say it's worth much), I'd prefer
>> renaming the C++ ones as well.  I think it does a better job of
>> telling the user what they're using, one of several variants of
>> qpid-send or -receive.
>
> I think there is a difference of mindset then. I view
> qpid-send/qpid-receive as generally useful utilities, that perhaps
> should be installed by default when we install qpid-clients.
>
> For simple enough cases they allow you to send/receive messages from the
> command line with no further tooling. And for this purpose on of the
> variants needs to be the default installed - I'd suggest the native code
> versions might be best suited. So keeping them called just
> qpid-send/receive works best for this. I'd certainly not object to
> installing links to them as part of the tests called
> qpid-cpp-send/receive.
>
> I think of the current set of "test" executables they are the only ons
> that qualify to be generally useful.

Yes, I do think they're primarily for testing.  Not narrow testing,
but rather the kinds of one-off testing that end users might do when
troubleshooting.  Since the client may have as much part as the server
in such troubleshooting, the language involved is still important,
according to this view, and should be visible.

I'm all for a default.  If we had a qpid-send facade, I'd see no
problem with defaulting the impl lang to C++.  I just think it's
slightly better to avoid obscuring the language: going with qpid-send,
qpid-java-send, and qpid-python-send leaves the user to guess what
that first one is.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Andrew Stitcher <as...@redhat.com>]

On Fri, 2013-11-08 at 11:15 -0500, Justin Ross wrote:
> On Fri, Nov 8, 2013 at 10:54 AM, Andrew Stitcher <as...@redhat.com> wrote:
> >> > The reason I want to do this is that I want to move to a testing regime
> >> > where we can run a specific build and use its install result (with
> >> > probably a special testing tools install location) to run the subsequent
> >> > test run. If we are to do this with multiple of the the qpid subtrees
> >> > together, say c++ and java then it will be confusing if the *-send and
> >> > *-receive executables have the same name.
> >>
> >> I wouldn't object to such an option.
> >>
> >> I'd be much less keen on renaming the existing c++ tests though.
> >
> > I agree - I tend to regard (somewhat chauvinistically I admit) the c++
> > qpid-send and qpid-receive as the default versions!
> 
> For what it's worth (and I wouldn't say it's worth much), I'd prefer
> renaming the C++ ones as well.  I think it does a better job of
> telling the user what they're using, one of several variants of
> qpid-send or -receive.

I think there is a difference of mindset then. I view
qpid-send/qpid-receive as generally useful utilities, that perhaps
should be installed by default when we install qpid-clients.

For simple enough cases they allow you to send/receive messages from the
command line with no further tooling. And for this purpose on of the
variants needs to be the default installed - I'd suggest the native code
versions might be best suited. So keeping them called just
qpid-send/receive works best for this. I'd certainly not object to
installing links to them as part of the tests called
qpid-cpp-send/receive.

I think of the current set of "test" executables they are the only ons
that qualify to be generally useful.

Andrew


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Justin Ross <ju...@gmail.com>]

On Fri, Nov 8, 2013 at 10:54 AM, Andrew Stitcher <as...@redhat.com> wrote:
>> > The reason I want to do this is that I want to move to a testing regime
>> > where we can run a specific build and use its install result (with
>> > probably a special testing tools install location) to run the subsequent
>> > test run. If we are to do this with multiple of the the qpid subtrees
>> > together, say c++ and java then it will be confusing if the *-send and
>> > *-receive executables have the same name.
>>
>> I wouldn't object to such an option.
>>
>> I'd be much less keen on renaming the existing c++ tests though.
>
> I agree - I tend to regard (somewhat chauvinistically I admit) the c++
> qpid-send and qpid-receive as the default versions!

For what it's worth (and I wouldn't say it's worth much), I'd prefer
renaming the C++ ones as well.  I think it does a better job of
telling the user what they're using, one of several variants of
qpid-send or -receive.

It would also leave space for introducing a simple wrapper with a
parameter similar to the one proposed for qpid-benchmark: "qpid-send
--lang python|java|cpp".  This would be a nicer way to expose the
variants for testing.

Justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Andrew Stitcher <as...@redhat.com>]

On Fri, 2013-11-08 at 15:45 +0000, Gordon Sim wrote:
> On 11/08/2013 03:37 PM, Andrew Stitcher wrote:
> > On Fri, 2013-11-08 at 11:43 +0000, Gordon Sim wrote:
> >> Since both the existing tests already import qpid.messaging this isn't
> >> actually imposing any significant extra dependency. There are also
> >> python equivalents of qpid-send and qpid-receive.
> >>
> >> Providing we keep the ability recently added to qpid-cpp-benchmark to
> >> specify the path to qpid-send and qpid-receive, that test can then be
> >> used to run either c++ or java based equivalents.
> >
> > One disagreement I may have here is that I think we should retain
> > different names for the different implementations of qpid-send and
> > qpid-receive. So I think we need to add an option to qpid-benchmark to
> > not only set the path it uses to find them, but also the names of the
> > executables it uses. An approach the works for the Java testing is to
> > use "profiles" that set a bunch of related parameters and that can be
> > switched as a whole.
> >
> > The reason I want to do this is that I want to move to a testing regime
> > where we can run a specific build and use its install result (with
> > probably a special testing tools install location) to run the subsequent
> > test run. If we are to do this with multiple of the the qpid subtrees
> > together, say c++ and java then it will be confusing if the *-send and
> > *-receive executables have the same name.
> 
> I wouldn't object to such an option.
> 
> I'd be much less keen on renaming the existing c++ tests though.

I agree - I tend to regard (somewhat chauvinistically I admit) the c++
qpid-send and qpid-receive as the default versions!

> 
> Since the java test scripts already use a different name, then renaming 
> the python equivalents (which are rarely used at present) would then be 
> all that was required.

That sounds good to me.

Andrew



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Gordon Sim <gs...@redhat.com>]

On 11/08/2013 03:37 PM, Andrew Stitcher wrote:
> On Fri, 2013-11-08 at 11:43 +0000, Gordon Sim wrote:
>> Since both the existing tests already import qpid.messaging this isn't
>> actually imposing any significant extra dependency. There are also
>> python equivalents of qpid-send and qpid-receive.
>>
>> Providing we keep the ability recently added to qpid-cpp-benchmark to
>> specify the path to qpid-send and qpid-receive, that test can then be
>> used to run either c++ or java based equivalents.
>
> One disagreement I may have here is that I think we should retain
> different names for the different implementations of qpid-send and
> qpid-receive. So I think we need to add an option to qpid-benchmark to
> not only set the path it uses to find them, but also the names of the
> executables it uses. An approach the works for the Java testing is to
> use "profiles" that set a bunch of related parameters and that can be
> switched as a whole.
>
> The reason I want to do this is that I want to move to a testing regime
> where we can run a specific build and use its install result (with
> probably a special testing tools install location) to run the subsequent
> test run. If we are to do this with multiple of the the qpid subtrees
> together, say c++ and java then it will be confusing if the *-send and
> *-receive executables have the same name.

I wouldn't object to such an option.

I'd be much less keen on renaming the existing c++ tests though.

Since the java test scripts already use a different name, then renaming 
the python equivalents (which are rarely used at present) would then be 
all that was required.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Andrew Stitcher <as...@redhat.com>]

On Fri, 2013-11-08 at 11:43 +0000, Gordon Sim wrote:
> ...
> > Option 1:
> >
> > Add a thin layer (new test) to check language preference and pass control to the corresponding test. In this case existing tests are preserved and continue to exist in the language specific source trees. The disadvantage is that we end up maintaining similar code in multiple places.
> 
> I'm strongly against this option.

I absolutely agree.


> Option 3:
> 
> Consolidate the tests into one source and place it in qpid/python. I.e. 
> consider the benchmark script to be part of (or related to) the python 
> client much as the qpid-python-test script is.

This makes a lot of sense to me too

> 
> Since both the existing tests already import qpid.messaging this isn't 
> actually imposing any significant extra dependency. There are also 
> python equivalents of qpid-send and qpid-receive.
> 
> Providing we keep the ability recently added to qpid-cpp-benchmark to 
> specify the path to qpid-send and qpid-receive, that test can then be 
> used to run either c++ or java based equivalents.

One disagreement I may have here is that I think we should retain
different names for the different implementations of qpid-send and
qpid-receive. So I think we need to add an option to qpid-benchmark to
not only set the path it uses to find them, but also the names of the
executables it uses. An approach the works for the Java testing is to
use "profiles" that set a bunch of related parameters and that can be
switched as a whole.

The reason I want to do this is that I want to move to a testing regime
where we can run a specific build and use its install result (with
probably a special testing tools install location) to run the subsequent
test run. If we are to do this with multiple of the the qpid subtrees
together, say c++ and java then it will be confusing if the *-send and
*-receive executables have the same name.

Andrew



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: Proposal to consolidate qpid benchmark tests

[Gordon Sim <gs...@redhat.com>]

On 11/07/2013 09:56 PM, Irina Boverman wrote:
> Currently there are 2 tests, cpp and java/jms, in respective source trees.
>
> qpid/cpp/src/tests/qpid-cpp-benchmark
> qpid/java/tools/bin/qpid-jms-benchmark
>
> Both tests use similar options (with a couple of exceptions). They also share majority of code, the exceptions are calls to qpid-send, qpid-receive and few other minor differences.
>
> There is a desire to provide one interface for all language implementations (as well as, extend implementation to other language clients). This goal can be accomplished in different ways:
>
> Option 1:
>
> Add a thin layer (new test) to check language preference and pass control to the corresponding test. In this case existing tests are preserved and continue to exist in the language specific source trees. The disadvantage is that we end up maintaining similar code in multiple places.

I'm strongly against this option. As you note, the qpid-jms-benchmark is 
essentially a copy-and-paste of qpid-cpp-benchmark with one or two lines 
commented out (presumably because they break something or aren't 
supported?) or tweaked (seemingly just due to different 'executable' 
names and again some options not supported) and the recent addition of 
options to control the path on which qpid-send/qpid-receive are located.

The latter is a good example of the usual problem with the 
copy-and-paste approach, since an update contributed to the original was 
not made to the copy. This addition is also designed to allow the same 
script to drive different implementations of send and receive.

> Option 2:
>
> Consolidate tests into one source and place it in qpid/tests/src/py/qpid_tests/client/ location. The advantage is reduced maintenance, the disadvantage is that now one needs all qpid source to run a specific test.

As above, the scripts are so close that consolidating[1] them is the 
obvious choice. However I don't think the qpid/tests location is the 
best option. So...

Option 3:

Consolidate the tests into one source and place it in qpid/python. I.e. 
consider the benchmark script to be part of (or related to) the python 
client much as the qpid-python-test script is.

Since both the existing tests already import qpid.messaging this isn't 
actually imposing any significant extra dependency. There are also 
python equivalents of qpid-send and qpid-receive.

Providing we keep the ability recently added to qpid-cpp-benchmark to 
specify the path to qpid-send and qpid-receive, that test can then be 
used to run either c++ or java based equivalents.

--Gordon

[1] In practice I think consolidation is simply a mater or renaming the 
qpid-cpp-benchmark, renaming qpid-jms-send and qpid-jms-receive to 
qpid-send and qpid-receive respectively, and then modifying those to 
accept the options passed in, even if they don't yet actually support them.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org