Re: Review Request: provide means of configuring subscription queues for 'topics' over AMQP 1.0

[Gordon Sim <gs...@redhat.com>]

On 06/29/2013 03:47 PM, Fraser Adams wrote:
> Some of this is making me a little nervous in particular the response
> around my headers Address String, but I might me misunderstanding e.g.
>
> "That would now be handled in two steps. The first is a configuration
> step whereby a 'topic' is created referencing amq.match and specifying
> the desired 'policy' for subscription queues for that topic.
>
> E.g. qpidt create topic amq.match.default exchange=amq.match
> qpid.policy_type=ring qpid.max_size=500000000"
>
> So this is using your new qpidt utility (what does qpidt stand for BTW?)
> to create a "topic" called amq.match.default referencing amq.match with
> the specified policy - correct? I'm not entirely sure what a "topic"
> really relates to in the AMQP 1.0 sense but that's probably a separate
> question....

There isn't an explicit concept of 'topic' in the AMQP 1.0 
specification. I'm using that term in the JMS sense. (Note that previous 
versions of AMQP did not had any explicit notion of topic either).

> "Then you would create your receiver using an address that referenced
> the topic created, and specifying the filter instead of the x-bindings.
> E.g.
>
> amq.match.default; {link:{filter:{value:x-match:all,data-service:
> amqp-delivery, item-owner:
> fadams},name:data1,descriptor:"apache.org:legacy-amqp-headers-binding:map"}}}"
>
>
>
> So this is using the "amq.match.default" topic in essence as a factory
> and will create a subscription using what amounts to the same header
> matches as my previous example using a 500K ring queue - correct?

Correct. Essentially it is a way to identify the exchange and the set of 
desired subscription queue properties simply through the node name. 
(also allowing different sets of properties for different use cases.

> So far so good, but in my example:
>
> "testqueue; {create: receiver, node: {x-declare: {arguments:
> {'qpid.policy_type': ring, 'qpid.max_size': 500000000}}, x-bindings:
> [{exchange: 'amq.match', queue: 'testqueue', key: 'data1', arguments:
> {x-match: all, data-service: amqp-delivery, item-owner: fadams}}]}}"
>
> The fact that the queue had a particular name is significant! So in this
> example the first consumer to create a destination with this Address
> String causes a queue called "testqueue" to be created and binds it to
> the headers exchange using the bindings specified in the x-bindings. But
> (significantly for my use cases!) if that consumer were to close down
> and restart it would simply reconnect to "testqueue" consuming any
> messages that may have been left on it. Moreover if my consumer were to
> note that the queue depth was rising too much I could scale out
> additional instances which again would simply consume off "testqueue".

The first part is where the subscription continues to exist even when 
there are no active links. I.e. something like a durable subscription in 
the JMS sense, which you get by specifying durable:True in your link 
properties (which in AMQP 1.0 is translated into the terminus durability).

The second part is the notion of a 'shared' subscription. You can 
specify a name in your link and request the 'shared' node capability. 
This capability is specific to qpidd at present, but is simple and 
obvious enough that I could see it being more widely adopted. (In the 
case of qpidd the queue name will match the link name).

So the address might look like:

amq.match.default; {link:{durable:true, name:testqueue, 
filter:{value:x-match:all,data-service: amqp-delivery, item-owner:
fadams},name:data1,descriptor:"apache.org:legacy-amqp-headers-binding:map"}}, 
node:{capabilities:[shared]}}"

> I guess what I'm saying is that the following use cases are significant
> for me:
> a) The ability to use *named* non-exclusive queue where consumers can
> come and go and multiple consumers may be attached for scaling purposes.

That is supported through the concept of shared subscriptions 
(orthogonal to the use of 'topic' as factory for subscription queues of 
a particular configuration).

> b) My consumers operate in a "self-service" model, so a consumer creates
> a destination via an Address String and that string may specify the
> queue we wish to create.

The name of the subscription is currently identified by the name of the 
link.

Note that the specification requires the name of the link to be unique 
for the pair of containers it connects, i.e. you can't have more than 
one sender/receiver with the same link name over the same connection. 
Supporting a special link property to identify the shared subscription 
in cases where this restriction is not acceptable (e.g. JMS 2.0) is I 
think a reasonable extension.

> c) There may be a non-trivial number of logical consumers, each of which
> may use different queue names and each may or may not need to scale out
> to multiple instances consuming of a given queue.

You could have multiple, distinct shared subscriptions (each identified 
by a different name).

> I'd be happy enough with the idea of "administratively" creating what
> amounts to a single "factory Node" if all of my consumers were able to
> use that to create their (different) queues in the sort of "self service
> done via the Address String" model I allude to above, because all of my
> queues tend to have the same parameters albeit with different names.
>
> If I've just misinterpreted and you can suggest an Address String that
> will do this sort of thing then fantastic, but I've got something of a
> bad feeling that your example is much more akin to the "traditional"
> topic subscription where consumers are logically consuming data off a
> topic and the queue is incidental.

The 'subscription' is named and shareable by multiple (competing) 
consumers. In the case of qpidd that is currently backed by a queue, but 
that aspect I think is (and should be) incidental.

> In your example Address String I expect that if I were to stand up
> multiple consumers using that Address they'd each receive a copy of any
> given message matching the said headers as opposed to receiving some of
> the messages depending on how many consumer instances were consuming.

True unless they have requested the 'shared' capability and are using 
the same link name, in which case the messages are shared out between them.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org