You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Andrew Wright <at...@mac.com> on 2009/09/14 23:56:11 UTC
Java/JMS LVQ interaction
Hi all,
We've been using LVQs from our JMS clients and while they work well
enough, the way we're going about feels a little clunky. The process
we use breaks down as:
- Declare a LVQ with the NO_BROWSE option set for the cache
- Declare a second (non-lvq) to receive real-time updates (likely on a
fanout exchange)
- clients connect to the second queue but don't start consuming msgs yet
- clients then connect to the LVQ and browse the entire thing for the
'snapshot' state
- they then start consuming from the fanout queue and have nothing
more to do with the LVQ
- producers send msgs to both queues
This gives us a way of having multiple clients receive a snapshot when
they start up, and receive additional updates afterwards. It just
feels like we're doing too much work. I guess what I had in mind when
I first heard about LVQs was more of a broker-side key/value cache,
with an efficient way of accessing it from a client. What we'd really
like is a way for clients to say 'give me the last value for key x',
have that call return in a short amount of time (hash-based), and do
it non-destructively (ie, multiple clients could do this and not
remove the msg for other clients). Currently the jms QueueBrowser
implementation doesn't work like that, which is why we've ended up
with the above workaround. (aside thought: perhaps an LVQ in
conjunction with a fanout exchange might be of some use?)
Have I just been completely blind and missed something obvious, or is
this the way it works today? If so, are there any thoughts around
providing something more 'LVQ-oriented' vs 'JMS-oriented'? For
something like this we wouldn't feel strongly about sticking to JMS,
although using a selector on a QueueBrowser would seem an obvious fit.
Thanks very much,
Andrew
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java/JMS LVQ interaction
Posted by Andrew Wright <at...@mac.com>.
>>> <snip>
>>> Dare I wonder out loud what might be involved in hooking this up?
>>> (java
>>> client/c++ broker combination of particular interest for us)
>>>
>>
>> Its not a trivial change I'm afraid, and would require some initial
>> refactoring around the c++ brokers queue implementation. However in
>> my
>> opinion it would be a very worthwhile task and I'll turn my
>> attention to it
>> as soon as I can. I've created a Jira at
>> https://issues.apache.org/jira/browse/QPID-2104, feel free to add any
>> comments or re-assign.
>>
>>
> For the Java Broker, as I said previously, I have a patch which I
> wrote ages
> ago which I'm planning on applying before the 0.6 release... I'll
> probably
> switch my attention to that once I've finished my work in implementing
> AMQP0-10 on the Java Broker...
>
> -- Rob
Sounds great, will await developments with interest!
Andrew
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java/JMS LVQ interaction
Posted by Robert Godfrey <ro...@gmail.com>.
2009/9/16 Gordon Sim <gs...@redhat.com>
> On 09/15/2009 08:39 PM, Andrew Wright wrote:
>
>>
>> On 15 Sep 2009, at 13:33, Gordon Sim wrote:
>>
>> On 09/15/2009 01:13 PM, Robert Godfrey wrote:
>>>
>>>> Hi Gordon,
>>>>
>>>> 2009/9/15 Gordon Sim<gs...@redhat.com>
>>>>
>>>> I think clunky is a fair description of the current capabilities and I
>>>>> think we can improve on that.
>>>>>
>>>>> My view of what ideal 'LVQ' behaviour whould be like is that the
>>>>> 'queue'
>>>>> would really be more like a 'topic' where the last message for each
>>>>> key was
>>>>> always saved. Clients would subscribe to it and receive the last
>>>>> message
>>>>> published for each key and subsequently any updates (i.e. any new
>>>>> messages).
>>>>>
>>>>>
>>>>> The above is pretty much how I would like an LVQ to behave... Indeed
>>>> I have
>>>> a patch for the Java Broker to provide just such LVQ functionality,
>>>> which I
>>>> should get round to applying for 0.6. The only real question is do you
>>>> inform subscribers of *all* updates... or, if the subscriber starts
>>>> to fall
>>>> behind, do you only send updates that have not themselves already been
>>>> superceded (my Java implementation does the latter).
>>>>
>>>
>>> I think the latter sounds reasonable.
>>>
>>>
>>
>> Absolutely - your description sounds spot-on. That would certainly be
>> very useful behaviour (it would greatly simplify our current application
>> code). Consumers could be artificially throttled to reduce processing
>> requirements if things got busy.
>>
>> The only other use case I can imagine would be calling into the broker
>> on an ad-hoc basis, supplying a key and being returned the message that
>> currently matches that key, or null if there never was one. That could
>> perhaps be useful in very high update rate scenarios, where you'd want
>> the broker to bear the work of maintaining the queue without making
>> clients receive every (or even every few) messages. Admittedly this does
>> go away somewhat from a 'messaging' based scenario and more towards a
>> 'remote-cache' style behaviour, not sure if that would be of concern.
>>
>> Dare I wonder out loud what might be involved in hooking this up? (java
>> client/c++ broker combination of particular interest for us)
>>
>
> Its not a trivial change I'm afraid, and would require some initial
> refactoring around the c++ brokers queue implementation. However in my
> opinion it would be a very worthwhile task and I'll turn my attention to it
> as soon as I can. I've created a Jira at
> https://issues.apache.org/jira/browse/QPID-2104, feel free to add any
> comments or re-assign.
>
>
For the Java Broker, as I said previously, I have a patch which I wrote ages
ago which I'm planning on applying before the 0.6 release... I'll probably
switch my attention to that once I've finished my work in implementing
AMQP0-10 on the Java Broker...
-- Rob
Re: Java/JMS LVQ interaction
Posted by Gordon Sim <gs...@redhat.com>.
On 09/15/2009 08:39 PM, Andrew Wright wrote:
>
> On 15 Sep 2009, at 13:33, Gordon Sim wrote:
>
>> On 09/15/2009 01:13 PM, Robert Godfrey wrote:
>>> Hi Gordon,
>>>
>>> 2009/9/15 Gordon Sim<gs...@redhat.com>
>>>
>>>> I think clunky is a fair description of the current capabilities and I
>>>> think we can improve on that.
>>>>
>>>> My view of what ideal 'LVQ' behaviour whould be like is that the
>>>> 'queue'
>>>> would really be more like a 'topic' where the last message for each
>>>> key was
>>>> always saved. Clients would subscribe to it and receive the last
>>>> message
>>>> published for each key and subsequently any updates (i.e. any new
>>>> messages).
>>>>
>>>>
>>> The above is pretty much how I would like an LVQ to behave... Indeed
>>> I have
>>> a patch for the Java Broker to provide just such LVQ functionality,
>>> which I
>>> should get round to applying for 0.6. The only real question is do you
>>> inform subscribers of *all* updates... or, if the subscriber starts
>>> to fall
>>> behind, do you only send updates that have not themselves already been
>>> superceded (my Java implementation does the latter).
>>
>> I think the latter sounds reasonable.
>>
>
>
> Absolutely - your description sounds spot-on. That would certainly be
> very useful behaviour (it would greatly simplify our current application
> code). Consumers could be artificially throttled to reduce processing
> requirements if things got busy.
>
> The only other use case I can imagine would be calling into the broker
> on an ad-hoc basis, supplying a key and being returned the message that
> currently matches that key, or null if there never was one. That could
> perhaps be useful in very high update rate scenarios, where you'd want
> the broker to bear the work of maintaining the queue without making
> clients receive every (or even every few) messages. Admittedly this does
> go away somewhat from a 'messaging' based scenario and more towards a
> 'remote-cache' style behaviour, not sure if that would be of concern.
>
> Dare I wonder out loud what might be involved in hooking this up? (java
> client/c++ broker combination of particular interest for us)
Its not a trivial change I'm afraid, and would require some initial
refactoring around the c++ brokers queue implementation. However in my
opinion it would be a very worthwhile task and I'll turn my attention to
it as soon as I can. I've created a Jira at
https://issues.apache.org/jira/browse/QPID-2104, feel free to add any
comments or re-assign.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java/JMS LVQ interaction
Posted by Andrew Wright <at...@mac.com>.
On 15 Sep 2009, at 13:33, Gordon Sim wrote:
> On 09/15/2009 01:13 PM, Robert Godfrey wrote:
>> Hi Gordon,
>>
>> 2009/9/15 Gordon Sim<gs...@redhat.com>
>>
>>> I think clunky is a fair description of the current capabilities
>>> and I
>>> think we can improve on that.
>>>
>>> My view of what ideal 'LVQ' behaviour whould be like is that the
>>> 'queue'
>>> would really be more like a 'topic' where the last message for
>>> each key was
>>> always saved. Clients would subscribe to it and receive the last
>>> message
>>> published for each key and subsequently any updates (i.e. any new
>>> messages).
>>>
>>>
>> The above is pretty much how I would like an LVQ to behave...
>> Indeed I have
>> a patch for the Java Broker to provide just such LVQ functionality,
>> which I
>> should get round to applying for 0.6. The only real question is do
>> you
>> inform subscribers of *all* updates... or, if the subscriber starts
>> to fall
>> behind, do you only send updates that have not themselves already
>> been
>> superceded (my Java implementation does the latter).
>
> I think the latter sounds reasonable.
>
Absolutely - your description sounds spot-on. That would certainly be
very useful behaviour (it would greatly simplify our current
application code). Consumers could be artificially throttled to reduce
processing requirements if things got busy.
The only other use case I can imagine would be calling into the broker
on an ad-hoc basis, supplying a key and being returned the message
that currently matches that key, or null if there never was one. That
could perhaps be useful in very high update rate scenarios, where
you'd want the broker to bear the work of maintaining the queue
without making clients receive every (or even every few) messages.
Admittedly this does go away somewhat from a 'messaging' based
scenario and more towards a 'remote-cache' style behaviour, not sure
if that would be of concern.
Dare I wonder out loud what might be involved in hooking this up?
(java client/c++ broker combination of particular interest for us)
Cheers,
Andrew
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java/JMS LVQ interaction
Posted by Gordon Sim <gs...@redhat.com>.
On 09/15/2009 01:13 PM, Robert Godfrey wrote:
> Hi Gordon,
>
> 2009/9/15 Gordon Sim<gs...@redhat.com>
>
>> I think clunky is a fair description of the current capabilities and I
>> think we can improve on that.
>>
>> My view of what ideal 'LVQ' behaviour whould be like is that the 'queue'
>> would really be more like a 'topic' where the last message for each key was
>> always saved. Clients would subscribe to it and receive the last message
>> published for each key and subsequently any updates (i.e. any new messages).
>>
>> I.e. it would do pretty much what you describe, but through a simpler
>> interaction. The client would just need to 'subscribe' to the 'LVQ'.
>>
>> I would think that in JMS this would be made available as a Destination you
>> could create consumers and producers from for normal use. It would behave
>> more like a JMS topic than a queue though (i.e. consumers would not compete
>> for messages).
>>
>> Are there other views or use cases on what it should do?
>>
>>
> The above is pretty much how I would like an LVQ to behave... Indeed I have
> a patch for the Java Broker to provide just such LVQ functionality, which I
> should get round to applying for 0.6. The only real question is do you
> inform subscribers of *all* updates... or, if the subscriber starts to fall
> behind, do you only send updates that have not themselves already been
> superceded (my Java implementation does the latter).
I think the latter sounds reasonable.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Java/JMS LVQ interaction
Posted by Robert Godfrey <ro...@gmail.com>.
Hi Gordon,
2009/9/15 Gordon Sim <gs...@redhat.com>
> I think clunky is a fair description of the current capabilities and I
> think we can improve on that.
>
> My view of what ideal 'LVQ' behaviour whould be like is that the 'queue'
> would really be more like a 'topic' where the last message for each key was
> always saved. Clients would subscribe to it and receive the last message
> published for each key and subsequently any updates (i.e. any new messages).
>
> I.e. it would do pretty much what you describe, but through a simpler
> interaction. The client would just need to 'subscribe' to the 'LVQ'.
>
> I would think that in JMS this would be made available as a Destination you
> could create consumers and producers from for normal use. It would behave
> more like a JMS topic than a queue though (i.e. consumers would not compete
> for messages).
>
> Are there other views or use cases on what it should do?
>
>
The above is pretty much how I would like an LVQ to behave... Indeed I have
a patch for the Java Broker to provide just such LVQ functionality, which I
should get round to applying for 0.6. The only real question is do you
inform subscribers of *all* updates... or, if the subscriber starts to fall
behind, do you only send updates that have not themselves already been
superceded (my Java implementation does the latter).
Cheers,
Rob
Re: Java/JMS LVQ interaction
Posted by Gordon Sim <gs...@redhat.com>.
On 09/14/2009 10:56 PM, Andrew Wright wrote:
> Hi all,
>
> We've been using LVQs from our JMS clients and while they work well
> enough, the way we're going about feels a little clunky. The process we
> use breaks down as:
>
> - Declare a LVQ with the NO_BROWSE option set for the cache
> - Declare a second (non-lvq) to receive real-time updates (likely on a
> fanout exchange)
> - clients connect to the second queue but don't start consuming msgs yet
> - clients then connect to the LVQ and browse the entire thing for the
> 'snapshot' state
> - they then start consuming from the fanout queue and have nothing more
> to do with the LVQ
> - producers send msgs to both queues
>
> This gives us a way of having multiple clients receive a snapshot when
> they start up, and receive additional updates afterwards. It just feels
> like we're doing too much work. I guess what I had in mind when I first
> heard about LVQs was more of a broker-side key/value cache, with an
> efficient way of accessing it from a client. What we'd really like is a
> way for clients to say 'give me the last value for key x', have that
> call return in a short amount of time (hash-based), and do it
> non-destructively (ie, multiple clients could do this and not remove the
> msg for other clients). Currently the jms QueueBrowser implementation
> doesn't work like that, which is why we've ended up with the above
> workaround. (aside thought: perhaps an LVQ in conjunction with a fanout
> exchange might be of some use?)
>
> Have I just been completely blind and missed something obvious, or is
> this the way it works today?If so, are there any thoughts around
> providing something more 'LVQ-oriented' vs 'JMS-oriented'?
I think clunky is a fair description of the current capabilities and I
think we can improve on that.
My view of what ideal 'LVQ' behaviour whould be like is that the 'queue'
would really be more like a 'topic' where the last message for each key
was always saved. Clients would subscribe to it and receive the last
message published for each key and subsequently any updates (i.e. any
new messages).
I.e. it would do pretty much what you describe, but through a simpler
interaction. The client would just need to 'subscribe' to the 'LVQ'.
I would think that in JMS this would be made available as a Destination
you could create consumers and producers from for normal use. It would
behave more like a JMS topic than a queue though (i.e. consumers would
not compete for messages).
Are there other views or use cases on what it should do?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org