You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2013/03/01 09:24:36 UTC
Re: Review Request: Selectors (JMS style) for C++ broker
This looks really interesting. I've certainly been wondering about JMS
style message selectors for a while, it was on my "TODO" list to have a
play with when I get a break from QMFing so that's one I can park :-)
I think that this is a pretty important new feature that we should
publicise properly and keep users properly in the loop on.
I've got a couple of questions though.
1) I was looking through the Jira list that Irina Boverman sent out the
other day and noticed https://issues.apache.org/jira/browse/QPID-1766
which looks to have originated back in 2009. There seemed to be a fairly
long comment thread on that Jira but I'm not aware that it made it in to
any release? Andrew's work seems unrelated (other than being another
implementation of message selectors). Does anyone have the background??
The Xquery approach seemed quite interesting (and I guess consistent
with the XML exchange) does anyone have an idea of the performance
differences between that approach and Andrew's??
2) The approach for describing the selectors on the client side looks to
be done in the AddressString, That looks like a pretty good approach and
is consistent with a lot of things in Qpid but:
a) ISTR that with JMS the selector was passed from client to broker
using an "x-" property (can't recall exactly the
property but I'm pretty sure that it started x- whereas this
uses "qpid.selector") "qpid.selector" in itself isn't a big deal
but I'd be keen on absolute compatibility between C++ and Java
at both the broker and client level. I'd go so far as to
say interoperability is a prerequisite.
b) Following on from a) so does the Java client/broker allow
message selectors to be specified in an AddressString in
addition to being specified in createConsumer() on Session.
c) Following on from b) is there an intention to enhance the
qpid::messaging Session interface to provide an overloaded
createReceiver() method to take an Address and a message
selector. This last one is possibly a little moot and
arguably a little redundant but (apart from making things very
consistent with JMS) one advantage is that it allows one
to separate the concepts of subscription and filtering if so
desired. As an example I've got some cases where I've got
a mixture of C++ and Java consumers that subscribe to the same
data feed. As it happens some use different subsets so we've ended up
with quite contrived headers bindings to enable this, but it might be
cleaner to use selectors. In this case I'd quite like to enforce the
same Address String subscription (as that's the general "type" of the
data that they are allowed to see) but allow them to choose to
filter subsets if the so wish.
I'm quite excited by this though - nice one!!
Frase
On 28/02/13 21:59, Chug Rolke wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9494/#review17208
> -----------------------------------------------------------
>
> Ship it!
>
>
> Ship It!
>
> - Chug Rolke
>
>
> On Feb. 18, 2013, 5:21 p.m., Andrew Stitcher wrote:
>> -----------------------------------------------------------
>> This is an automatically generated e-mail. To reply, visit:
>> https://reviews.apache.org/r/9494/
>> -----------------------------------------------------------
>>
>> (Updated Feb. 18, 2013, 5:21 p.m.)
>>
>>
>> Review request for qpid, Gordon Sim and Kenneth Giusti.
>>
>>
>> Description
>> -------
>>
>> This is a potential first cut at JMS style selectors for the c++ broker and client.
>>
>> Currently what's implemented:
>>
>> Selector parser/evaluator:
>> Parse a subset of the selector language including:
>> Conditional expressions : "=" "<>" "IS NULL" "IS NOT NULL" "NOT "AND" "OR"
>> All values are treated as strings (no numerics or boolean literals) so there are no numeric
>> operators or comparison operators (strings can only be compared for equality)
>> "LIKE" & "NOT LIKE", "IN" are not (yet) supported.
>>
>> To specify a selector you add a link property called qpid.selector to a queue type address, for example:
>>
>> queue; {link:{qpid.selector:"amqp.delivery_mode='PERSISTENT' or not TransactionType='red'"}}
>>
>> The code is currently implemented in the 0-10 code paths of the broker and the messaging library:
>> The selector is sent from client to broker as an argument property in the message subscription.
>>
>> Still to do:
>> * amqp 1.0 support.
>> * Improved "special" identifier handling - currently not all the amqp.* identifiers are handled and
>> we may need some extra message hooks to get the rest uniformly across 0-10 and 1.0 messages.
>>
>> * Work on parser/evaluator:
>> ** Handle type other than string
>> ** Booleans
>> ** Numerics
>> ** Numeric operators and comparisons
>> ** more complex operators:
>> *** string: "LIKE", "NOT LIKE", "IN"
>> *** numeric: "BETWEEN"
>>
>> Having said all that needs still to be done, I'm reasonably happy that what's currently there is useful and stable so could go into the tree is currently is.
>>
>>
>> This addresses bug QPID-4558.
>> https://issues.apache.org/jira/browse/QPID-4558
>>
>>
>> Diffs
>> -----
>>
>> /trunk/qpid/cpp/src/CMakeLists.txt 1446233
>> /trunk/qpid/cpp/src/Makefile.am 1446233
>> /trunk/qpid/cpp/src/qpid/broker/Selector.h PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/Selector.cpp PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/SelectorExpression.h PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/SelectorExpression.cpp PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/SelectorToken.h PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/SelectorToken.cpp PRE-CREATION
>> /trunk/qpid/cpp/src/qpid/broker/SemanticState.h 1446233
>> /trunk/qpid/cpp/src/qpid/broker/SemanticState.cpp 1446233
>> /trunk/qpid/cpp/src/qpid/client/amqp0_10/AddressResolution.cpp 1446233
>> /trunk/qpid/cpp/src/tests/CMakeLists.txt 1446233
>> /trunk/qpid/cpp/src/tests/Makefile.am 1446233
>> /trunk/qpid/cpp/src/tests/Selector.cpp PRE-CREATION
>>
>> Diff: https://reviews.apache.org/r/9494/diff/
>>
>>
>> Testing
>> -------
>>
>> Selector parser/evaluator tested reasonably thoroughly using new unit tests.
>>
>> Current make check still works.
>>
>> Informally tested selector functionality using qpid-send, qpid-receive.
>>
>>
>> Thanks,
>>
>> Andrew Stitcher
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: Review Request: Selectors (JMS style) for C++ broker
Posted by Andrew Stitcher <as...@redhat.com>.
On Fri, 2013-03-01 at 08:24 +0000, Fraser Adams wrote:
> This looks really interesting. I've certainly been wondering about JMS
> style message selectors for a while, it was on my "TODO" list to have a
> play with when I get a break from QMFing so that's one I can park :-)
>
> I think that this is a pretty important new feature that we should
> publicise properly and keep users properly in the loop on.
I plan to be getting my work so far into 0.22 and will (try to) properly
publicise on the user list.
>
> I've got a couple of questions though.
> 1) I was looking through the Jira list that Irina Boverman sent out the
> other day and noticed https://issues.apache.org/jira/browse/QPID-1766
> which looks to have originated back in 2009. There seemed to be a fairly
> long comment thread on that Jira but I'm not aware that it made it in to
> any release? Andrew's work seems unrelated (other than being another
> implementation of message selectors). Does anyone have the background??
I don't have specific background, but the my selector work is really
only possible because Gordon performed a thorough refactoring of the C++
broker queue code and created a single uniform place to do the
filtering. I doubt that this old code would work anymore. However there
is no reason why there couldn't be multiple "selector" implementations
in the C++ broker and indeed the amqp 1.0 spec allows for multiple
filters on consume which is how we model this for 1.0.
> The Xquery approach seemed quite interesting (and I guess consistent
> with the XML exchange) does anyone have an idea of the performance
> differences between that approach and Andrew's??
None whatever, it would strongly depend on the selector itself in any
case.
> 2) The approach for describing the selectors on the client side looks to
> be done in the AddressString, That looks like a pretty good approach and
> is consistent with a lot of things in Qpid but:
Well, I think it would be more correct to say that the selector is a
property of the link (which can be specified using the address syntax).
> a) ISTR that with JMS the selector was passed from client to broker
> using an "x-" property (can't recall exactly the
> property but I'm pretty sure that it started x- whereas this
> uses "qpid.selector") "qpid.selector" in itself isn't a big deal
> but I'd be keen on absolute compatibility between C++ and Java
> at both the broker and client level. I'd go so far as to
> say interoperability is a prerequisite.
Currently interop isn't my focus, the functionality is. However
supporting JMS selectors is obviously important so I will make sure that
the Java client can use the new C++ broker selectors, however I can't
guarantee that older Java clients will be able to use it. Having said
this I'm not going to go out of my way to make it fail either if there
are simple things I can do to make it work. As far as I can tell
(without looking very hard) some work will need to happen in the current
Java client to use these new selectors anyway.
> b) Following on from a) so does the Java client/broker allow
> message selectors to be specified in an AddressString in
> addition to being specified in createConsumer() on Session.
The default address parser allows you to put arbitrary properties in
there, so I'd guess yes, although I don't think that'd be a good way to
do it. In the C++ client I'm doing a small amount of work to decode the
link property and do the correct thing for 0-10 (and a different thing
for 1.0).
> c) Following on from b) is there an intention to enhance the
> qpid::messaging Session interface to provide an overloaded
> createReceiver() method to take an Address and a message
> selector. This last one is possibly a little moot and
> arguably a little redundant but (apart from making things very
> consistent with JMS) one advantage is that it allows one
> to separate the concepts of subscription and filtering if so
> desired. As an example I've got some cases where I've got
> a mixture of C++ and Java consumers that subscribe to the same
> data feed. As it happens some use different subsets so we've ended up
> with quite contrived headers bindings to enable this, but it might be
> cleaner to use selectors. In this case I'd quite like to enforce the
> same Address String subscription (as that's the general "type" of the
> data that they are allowed to see) but allow them to choose to
> filter subsets if the so wish.
This definitely an interesting idea, I absolutely agree that adding a
selector to an existing address is important. I'm also wondering whether
bound variables ala SQL might also be important for a client binding too
(to nip "bobby tables" in the bud - ie injection attacks).
I should point out that the current selector language is by no means the
full language although I do hope to have it much more fleshed out by the
actual release. There are 2 points here:
* I'm aiming at implementing the language specified in:
https://svn.apache.org/repos/asf/qpid/trunk/qpid/specs/apache-filters.xml#type-selector-filter
Which is NOT the same as the specified JMS language and needs some
translation of identifiers - this is why I say that some work will need
to happen on the Java client side. The good news is that this is a
(semi) standard for amqp 1.0.
* My current implementation (which is more flashed out than the posted
review - new review coming!) still does not support:
(strings)
- LIKE
- IN
(numbers)
- any arithmetic
- BETWEEN
>
> I'm quite excited by this though - nice one!!
Thank you for your comments.
Andrew
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org