Command sequences in qpid-config, qpid-route

[Jonathan Robie <jo...@redhat.com>]

I was talking with Ted about qpid-config and qpid-route, and we
thought it might be useful to have a simple, consistent syntax using
one set of conventions for both utilities.

Some other developers suggested that using optparse for all the Python
command line utilities might lead to more consistency. That helps with
most utilities, but not with qpid-config and qpid-route, which each
have a syntax based on sequences of strings.

In this discussion, it became clear that different people are thinking
of different use cases, and it's not completely clear what the real
requirements are or which solution would be best.

Here are some requirements that have been mentioned:

- We need a consistent syntax.

- A consistent API that corresponds to the commands these tools offer
   would be helpful. All of the commands in qpid-config and qpid-route
   are also useful within programs

- Machine-readable logging is important for some applications.

- It's important to be able to run these commands in batch mode. Start
   up time is significant, so running a sequence of Python command-line
   utilities in a batch file is slow.

   One solution: improve start-up time.

   Another solution: design a Python API for these tasks, and use a
   Python script, with one connection, rather than a batch file.

   Another solution: allow a sequence of qpid-config or qpid-route
   commands to be placed in a file and executed in batch mode.

- Running commands interactively, in a command shell, and using the
   same commands in a batch file. That makes it easier to write / debug
   scripts.

What other requirements / use cases should we keep in mind?

What other questions should we be asking?

The functionality we need to support is illustrated by the commands
below my signature. We might choose to support this syntax in some
form, or perhaps support an API that corresponds to these commands, or
perhaps support both, with a direct mapping between the two. Are there
other approaches we should consider?

Jonathan

###
#
#  Parsed representation: verb, noun, parameters (in a map)
#
###
show object 139
show object tross
show queue
show queue tross
show exchange
show exchange xml
show binding
show binding exchange=xml
show binding exchange="xml"
show binding exchange=xml queue=pqueue
show binding exchange="xml" queue="pqueue"
show binding exchange=xml queue=139
show connection
# logging
set log on file="~/tmp/mylog.log"  # default is ./qpid.log
set log off
# add queue <name> [AddQueueOptions]
declare queue tross
declare queue tross durable
declare queue tross durable=true
# declare exchange <name> <type> [AddExchangeOptions]
declare exchange foo # should there be a default type? or is this an error?
declare exchange a topic
declare exchange b direct
declare exchange c fanout
declare exchange d headers
declare exchange x xml
declare exchange bar topic durable=true
# del exchange <name>
del exchange bar
delete exchange bar
# del queue <name> [DelQueueOptions]
delete queue xml
del queue xml
del queue xml force=True
del queue xml force=False
del queue xml force=if-not-empty
del queue xml force=if-not-used
# bind
declare binding exchange queue binding-key
declare binding xml xqueue binding-key xquery="./weather"
declare binding xml xqueue binding-key xquery-file=weather.xq
declare binding header queue any k1=v1 k2=v2
# unbind <exchange-name> <queue-name> [binding-key]
del binding topic tross
del binding exchange queue binding-key
del binding xml xqueue binding-key
del binding header queue

declare route from broker=src-broker queue=public to broker=dest-broker 
exchange=fanout
del route from broker=src-broker queue=public to broker=dest-broker 
exchange=fanout
show route from broker=src-broker queue=public to broker=dest-broker 
exchange=fanout
show route from broker=dest-broker
show route from broker=dest-broker queue=public
show route to broker=dest-broker exchange=fanout

declare route from broker=localhost:10001 exchange=amq.topic 
key=global.# to broker=localhost:10002
declare route from broker=localhost:10001 exchange=amq.topic 
key=global.# to broker=localhost:10002 exchange=amq.topic # exchanges 
must match

del route from broker=localhost:10001 exchange=amq.topic key=global.# to 
broker=localhost:10002
del route from broker=localhost:10001 exchange=amq.topic key=global.# to 
broker=localhost:10002 exchange=amq.topic # exchanges must match

del route from broker=localhost:10001 # .eqv. to flush

declare dynamic route from broker=localhost:10001 exchange=amq.topic to 
broker=localhost:10002

map route localhost:10001

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Command sequences in qpid-config, qpid-route

[Alan Conway <ac...@redhat.com>]

On 12/08/2010 06:37 PM, Jonathan Robie wrote:
> I was talking with Ted about qpid-config and qpid-route, and we
> thought it might be useful to have a simple, consistent syntax using
> one set of conventions for both utilities.
>
> Some other developers suggested that using optparse for all the Python
> command line utilities might lead to more consistency. That helps with
> most utilities, but not with qpid-config and qpid-route, which each
> have a syntax based on sequences of strings.
>
> In this discussion, it became clear that different people are thinking
> of different use cases, and it's not completely clear what the real
> requirements are or which solution would be best.
>
> Here are some requirements that have been mentioned:
>
> - We need a consistent syntax.
>
> - A consistent API that corresponds to the commands these tools offer
> would be helpful. All of the commands in qpid-config and qpid-route
> are also useful within programs
>
> - Machine-readable logging is important for some applications.
>
> - It's important to be able to run these commands in batch mode. Start
> up time is significant, so running a sequence of Python command-line
> utilities in a batch file is slow.
>
> One solution: improve start-up time.
>
> Another solution: design a Python API for these tasks, and use a
> Python script, with one connection, rather than a batch file.
>
> Another solution: allow a sequence of qpid-config or qpid-route
> commands to be placed in a file and executed in batch mode.
>
> - Running commands interactively, in a command shell, and using the
> same commands in a batch file. That makes it easier to write / debug
> scripts.
>
> What other requirements / use cases should we keep in mind?
>
> What other questions should we be asking?
>
> The functionality we need to support is illustrated by the commands
> below my signature. We might choose to support this syntax in some
> form, or perhaps support an API that corresponds to these commands, or
> perhaps support both, with a direct mapping between the two. Are there
> other approaches we should consider?

git and svn follow a common model: a single executable (or python script) that 
offers a set of command, with the general syntax:
qpid-foo cmd [options] <parameters>....

That matches quite nicely with qpid-tool style "qpid-tool show object 139". You 
could allow for a simple "shell"  mode like qpid-tool does now where the 
commands work without the qpid-tool prefix. Then you could roll the 
functionality of other random tools in as extra commands with a common syntax 
and set of base options for all.

>
> Jonathan
>
> ###
> #
> # Parsed representation: verb, noun, parameters (in a map)
> #
> ###
> show object 139
> show object tross
> show queue
> show queue tross
> show exchange
> show exchange xml
> show binding
> show binding exchange=xml
> show binding exchange="xml"
> show binding exchange=xml queue=pqueue
> show binding exchange="xml" queue="pqueue"
> show binding exchange=xml queue=139
> show connection
> # logging
> set log on file="~/tmp/mylog.log" # default is ./qpid.log
> set log off
> # add queue <name> [AddQueueOptions]
> declare queue tross
> declare queue tross durable
> declare queue tross durable=true
> # declare exchange <name> <type> [AddExchangeOptions]
> declare exchange foo # should there be a default type? or is this an error?
> declare exchange a topic
> declare exchange b direct
> declare exchange c fanout
> declare exchange d headers
> declare exchange x xml
> declare exchange bar topic durable=true
> # del exchange <name>
> del exchange bar
> delete exchange bar
> # del queue <name> [DelQueueOptions]
> delete queue xml
> del queue xml
> del queue xml force=True
> del queue xml force=False
> del queue xml force=if-not-empty
> del queue xml force=if-not-used
> # bind
> declare binding exchange queue binding-key
> declare binding xml xqueue binding-key xquery="./weather"
> declare binding xml xqueue binding-key xquery-file=weather.xq
> declare binding header queue any k1=v1 k2=v2
> # unbind <exchange-name> <queue-name> [binding-key]
> del binding topic tross
> del binding exchange queue binding-key
> del binding xml xqueue binding-key
> del binding header queue
>
> declare route from broker=src-broker queue=public to broker=dest-broker
> exchange=fanout
> del route from broker=src-broker queue=public to broker=dest-broker exchange=fanout
> show route from broker=src-broker queue=public to broker=dest-broker
> exchange=fanout
> show route from broker=dest-broker
> show route from broker=dest-broker queue=public
> show route to broker=dest-broker exchange=fanout
>
> declare route from broker=localhost:10001 exchange=amq.topic key=global.# to
> broker=localhost:10002
> declare route from broker=localhost:10001 exchange=amq.topic key=global.# to
> broker=localhost:10002 exchange=amq.topic # exchanges must match
>
> del route from broker=localhost:10001 exchange=amq.topic key=global.# to
> broker=localhost:10002
> del route from broker=localhost:10001 exchange=amq.topic key=global.# to
> broker=localhost:10002 exchange=amq.topic # exchanges must match
>
> del route from broker=localhost:10001 # .eqv. to flush
>
> declare dynamic route from broker=localhost:10001 exchange=amq.topic to
> broker=localhost:10002
>
> map route localhost:10001
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

Re: Command sequences in qpid-config, qpid-route

[Gordon Sim <gs...@redhat.com>]

On 12/08/2010 11:37 PM, Jonathan Robie wrote:
> I was talking with Ted about qpid-config and qpid-route, and we
> thought it might be useful to have a simple, consistent syntax using
> one set of conventions for both utilities.

The syntax you propose is really only more consistent in that the verb 
and the noun are in the same order (which certainly makes sense). Other 
than that there isn't a great deal of consistency between, for example:

  declare exchange a topic

and

  declare route from broker=localhost:10001 exchange=amq.topic
     key=global.# to broker=localhost:10002

In some sense qpid-config and qpid-route actually have a different 
purpose. The former is concerned with listing, creating or deleting 
objects on a single broker. The latter, it seems to me, is concerned 
with presenting a higher level conceptual view of routing between two 
(or more) brokers (how successful it is in that is open to debate).

I think before changing anything like this we should start by 
re-examining the rationale for each tool we have.

> Some other developers suggested that using optparse for all the Python
> command line utilities might lead to more consistency. That helps with
> most utilities, but not with qpid-config and qpid-route, which each
> have a syntax based on sequences of strings.

They do in addition have lots of options, however and a consistent 
approach to those is still relevant.

The use of optparse throughout primarily unifies and simplifies the code 
for maintainers. The benefit to users is that it avoids inconsistent 
weirdness in actual option *parsing*; it doesn't in and of itself result 
in consistent options being chosen. Both things are beneficial however.

Interpretation of arguments (as opposed to options) is a third issue 
(and is certainly also important).

> In this discussion, it became clear that different people are thinking
> of different use cases, and it's not completely clear what the real
> requirements are or which solution would be best.
>
> Here are some requirements that have been mentioned:
>
> - We need a consistent syntax.
>
> - A consistent API that corresponds to the commands these tools offer
> would be helpful. All of the commands in qpid-config and qpid-route
> are also useful within programs
>
> - Machine-readable logging is important for some applications.
>
> - It's important to be able to run these commands in batch mode. Start
> up time is significant, so running a sequence of Python command-line
> utilities in a batch file is slow.
>
> One solution: improve start-up time.

(The use of the QMF API in qpid-config for example seems to add an order 
of magnitude to the time taken to e.g. declare a queue over a simpler 
program that does the same thing...)

> Another solution: design a Python API for these tasks, and use a
> Python script, with one connection, rather than a batch file.
>
> Another solution: allow a sequence of qpid-config or qpid-route
> commands to be placed in a file and executed in batch mode.
 >
> - Running commands interactively, in a command shell, and using the
> same commands in a batch file. That makes it easier to write / debug
> scripts.
>
> What other requirements / use cases should we keep in mind?

One thing that none of the command line tools do at present is provide 
an easy way to invoke arbitrary management methods (e.g. purge, move 
etc). I recently came across this when adding a command to change the 
log-level.

> What other questions should we be asking?
>
> The functionality we need to support is illustrated by the commands
> below my signature. We might choose to support this syntax in some
> form, or perhaps support an API that corresponds to these commands, or
> perhaps support both, with a direct mapping between the two. Are there
> other approaches we should consider?

I think we should have a standard approach to invoking management 
commands (QMF methods at present)[1]. That in itself defines an API. We 
then need to carefully consider how exactly to expose various operations 
through the schema to ensure usability and consistency overall.

The querying and listing of management objects is the other part of the 
picture[2]. Consistency there is fairly easy to achieve I think.

[1] I checked in a little test program to do this (currently in 
cpp/src/tests/qpid-ctrl). I needed it specifically to test runtime 
changes to  log-level. An additional objective was to explore the use of 
a message based approach to programmatic broker management (i.e. 
initially at least, through direct use of QMFv2).

You get a minimal usage statement from qpid-ctrl --help. Essentially you 
supply the QMF method name and then key-value pairs for any parameters 
as arguments. There are options to choose the broker to connect to and 
some other operational aspects. By default the methods are assumed to be 
intended to be invoked on the Broker object itself E.g.

   qpid-ctrl setLogLevel level=info+
   qpid-ctrl getLogLevel

   qpid-ctrl queueMoveMessages srcQueue=a destQueue=b qty=3

   qpid-ctrl connect host=localhost port=5673

However you can also invoke methods on any object by providing the class 
of the object and an identifier for it. E.g.

   qpid-ctrl --class queue --id my-queue purge request=5

This tool would need quite a bit more work before it would be ready for 
consideration as something other than a test utility and the details of 
the implementation are not necessarily the best choices. However I think 
its a good demonstration of the basic concept of focusing on a usable 
and consistent management schema as a way to drive consistency and 
uniformity in tool support for that schema.

[2] Attached is a *very* rough example program that uses QMFv2 to list 
objects. This was really an experiment I did for my own purposes but it 
may be of interest to this thread in conjunction with qpid-ctrl.