[java] Question on implementing queue browsing

["Das, Kapali Tejeswar" <te...@iona.com>]

I am currently working on implementing the JMS queue browsing
functionality. Qpid uses java.util.concurrent.SynchronousQueue for its
internal queue for enqueing and dequeing messages. This class doesn't
allow to peek/browse messages in the queue. It only has methods to put a
message or get a message (with a timeout).

 

Any suggestions for how we should implement the queue browsing feature?
I also noticed that there is no native implementation of queue browsing
in Qpid.

 

Thanks and regards

Tejeswar

Re: [java] Question on implementing queue browsing

[Steven Shaw <st...@gmail.com>]

On 05/12/06, John O'Hara <jo...@gmail.com> wrote:
> MULE uses queue browsing like this:
> http://mule.mulesource.org/jira/browse/MULE-1004

Seems strange. Why don't they use MessageConsumer.receive(long timeout)
or MessageConsumer.receiveNoWait() ?

Re: [java] Question on implementing queue browsing

[Steven Shaw <st...@gmail.com>]

On 08/12/06, Gordon Sim <gs...@redhat.com> wrote:
> > By "locked" you mean unacknowledged?
>
> Yes. I.e. it is unavailable for any other consumer.

I am seeing the similarity to "lockmodes" in SQL.

Re: [java] Question on implementing queue browsing

[Gordon Sim <gs...@redhat.com>]

Steven Shaw wrote:
> On 07/12/06, Gordon Sim <gs...@redhat.com> wrote:
>> The other view is that the browse should show all messages that are
>> logically still on the queue even if they are 'locked' by a consumer. In
>> this case, it seems to me that at the very least you would want some
>> indication of whether or not the message was locked. This starts to feel
>> more like a management interface.
> 
> By "locked" you mean unacknowledged?

Yes. I.e. it is unavailable for any other consumer.

Re: [java] Question on implementing queue browsing

[Steven Shaw <st...@gmail.com>]

On 07/12/06, Gordon Sim <gs...@redhat.com> wrote:
> The other view is that the browse should show all messages that are
> logically still on the queue even if they are 'locked' by a consumer. In
> this case, it seems to me that at the very least you would want some
> indication of whether or not the message was locked. This starts to feel
> more like a management interface.

By "locked" you mean unacknowledged?

> The section of the JMS spec I read on this was brief and didn't seem to
> require one approach or the other.

The entire spec is very quiet on the semantics of queue browsing. It
seems purposely quiet in order to give vendors a wide choice of
implementation. One thing I noticed is that QueueBrowser is only valid
in JMS for PTP domain. That doesn't make as much sense with AMQP as
you may want to browse messages on a queue that is bound to a "topic"
exchange.

The spec certainly mentions that you don't need to provide a
"snapshot". i.e. there does not need to be any consistency to the
Enumeration returned from the QueueBrowser.

  http://java.sun.com/javaee/5/docs/api/javax/jms/QueueBrowser.html

It just mentions that you can browse messages "on a queue". I don't
believe it is described anywhere what it means to be "on the queue".
i.e. perhaps it includes unacknowledged messages, perhaps not. I'd be
inclined to interpret that is messages that have not been sent to a
consumer. However with large prefetch windows, perhaps this point of
view isn't very useful for some use cases.

> Are there any good use cases for queue browsing that anyone can share?
> To me it seems to be either for some form of management or for use where
> the queue contents don't change much. In the second case I think it
> would be better not to show unacked messages, and in the first case I
> think you would want to know the status of each message shown (i.e. is
> it unacked or fully available).

System admins will sometimes use a queue browsing tool to view
messages on the queues (particularly if one is causing a problem -
they will then perhaps delete it). Dead letter queues may be of
particular interest. A better management interface (I'm thinking GUI
but it would be support by some API) would allow messages on (dead
letter) queues to be dropped back on to regular queues or otherwise
moved/copied around or deleted. This certainly comes under
"management" or "administration".

Perhaps we should try to understand more the Mule use case. Maybe
they're not being silly. Even though it seems odd what they are doing.
Perhaps there is a good reason. Personally, I think that the Mule
would not be using a QueueBrowser if it could do some kind of select()
over queues to get a readiness notification. Probably good to ask on
the Mule list to the author of that code that John was linking to.

Cheers,
Steve.

Re: [java] Question on implementing queue browsing

[Gordon Sim <gs...@redhat.com>]

Robert Greig wrote:
> On 06/12/06, Gordon Sim <gs...@redhat.com> wrote:
>> Colin Crist wrote:
>> > 2. It should show messages that have been delivered but not
>> > acknowledged/committed.
>>
>> This is a good point and wasn't considered in the rough sketch I sent
>> out earlier. Both the java and c++ brokers would need some modification
>> to make this easy to do. Unacked messages are currently held in the
>> 'channels' for the consumers they were delivered to, and not in the
>> queue itself.
> 
> Would we actually want to support this except through the management
> interface (I think in our management interface we only support getting
> the count of the number of unacked msgs in the channel but I agree we
> should enhance this to allow retrieval of the actual messages)?

After thinking about this some more I had the same question. One way of 
looking at the browse functionality is that it provides an indication of 
what would be available to consume at a particular point in time (though 
in the face of continuous activity that isn't terribly useful anyway). 
Messages that have been sent but not acked are not available (at least 
not until the consumer that received them does without acking).

The other view is that the browse should show all messages that are 
logically still on the queue even if they are 'locked' by a consumer. In 
this case, it seems to me that at the very least you would want some 
indication of whether or not the message was locked. This starts to feel 
more like a management interface.

The section of the JMS spec I read on this was brief and didn't seem to 
require one approach or the other.

Are there any good use cases for queue browsing that anyone can share? 
To me it seems to be either for some form of management or for use where 
the queue contents don't change much. In the second case I think it 
would be better not to show unacked messages, and in the first case I 
think you would want to know the status of each message shown (i.e. is 
it unacked or fully available).

Re: [java] Question on implementing queue browsing

[Robert Greig <ro...@gmail.com>]

On 06/12/06, Gordon Sim <gs...@redhat.com> wrote:
> Colin Crist wrote:
> > 2. It should show messages that have been delivered but not
> > acknowledged/committed.
>
> This is a good point and wasn't considered in the rough sketch I sent
> out earlier. Both the java and c++ brokers would need some modification
> to make this easy to do. Unacked messages are currently held in the
> 'channels' for the consumers they were delivered to, and not in the
> queue itself.

Would we actually want to support this except through the management
interface (I think in our management interface we only support getting
the count of the number of unacked msgs in the channel but I agree we
should enhance this to allow retrieval of the actual messages)?

Do any JMS providers provide this via the JMS QueueBrowser API?

RG

Re: [java] Question on implementing queue browsing

[Gordon Sim <gs...@redhat.com>]

Colin Crist wrote:
> 2. It should show messages that have been delivered but not
> acknowledged/committed.

This is a good point and wasn't considered in the rough sketch I sent 
out earlier. Both the java and c++ brokers would need some modification 
to make this easy to do. Unacked messages are currently held in the 
'channels' for the consumers they were delivered to, and not in the 
queue itself.

RE: [java] Question on implementing queue browsing

[Colin Crist <co...@hermesjms.com>]

Hi,  
 
I'm not sure how to help from the AMQP side of things as I don't know the
protocol well enough so here are some general comments thay may or may not
be helpful
 
0. Minimal or zero overhead.
1. A queue browse is best effort - it does not reflect what a consumer would
see if they were to read it at that time as that would imply locking to get
a transactionaly intact image of the queue.
2. It should show messages that have been delivered but not
acknowledged/committed.
3. It should stream messages out to the browser from the head down taking
into account (2).
4. It should not collect the entire queue image before sending. JBossMQ does
this and it ain't nice as it takes ages for a big queue as you can imagine
(its why Hermes lets you browse using a consumer that does not ack and this
sux due to the side affects on real consumers)
5. Using a queue browser to see whats in a queue and then trying to use
those message ids in a selector does not mean the message will still be
there. Obvious but a common misconception.
6. In the topic domain a similar support is needed to browse messages
waiting for a duable subsription as its really just a queue - and in AMQP's
case I beleive that's actually the case so support it for durable
subscriptions too.
7. Cursors are interesting. When Hermes browses a queue and then a user does
a refresh it compares message ids from the previous browse and then updates
the Swing models appropriately so avoiding too much screen flicker. A cursor
would have a potentially longer lifetime (e.g. based on next-previous
buttons in a browser) and in the absense of any transational co-ordination
you would not get a good idea as to what really happening.  The JMS browse
semantic of head to tail message reading is simple and without any
compelling use-case I'd stick with it.
8. Persistent messaging != database. Another common gotcha. 
9. Selectors should be server side. WebSphereMQ for example does them client
side leading to extra traffice in a queue browse and shameful locking
problems with a real message consumer when there are many consumers. Imagine
it - pre-fetch 500 messages so no other consumer can take them, run the
selector and only release those not matched on a commit so another consumer
can do the same whilst the real consumer waits and waits and waits.... I did
this once and still shudder at the woes it caused. Not a queue brower issue
this of course.
10. No duplicates. 
 
Its been a while since I've had to digest the JMS spec but I don't thing the
above are against its spirit. For example  a queue browser with a topic URL
encoding the subscription id could work, of course you could do this via JMX
or a semantically AMQP correct java binding.
 
I guess if i was implementing it, I'd let the queue change under me and
steam out a best effort of its content without duplicates. You can always
get cleverer later if needs demand.
 
If anyone has specific questions I'd be glad to throw in further
comments.....
 
 
Colin.
 
btw hat Mule browse thing is weird.


  _____  

From: John O'Hara [mailto:john.r.ohara@gmail.com] 
Sent: 05 December 2006 20:25
To: qpid-dev@incubator.apache.org
Cc: Colin Crist
Subject: Re: [java] Question on implementing queue browsing


I've asked Colin Crist if he'd mind sharing any ideas about how an idea
browser would work, given his work on HermesJMS...

Colin :-) ?

MULE uses queue browsing like this:
http://mule.mulesource.org/jira/browse/MULE-1004

Which implies it had better construct quickly, since they are likely to be
used as a non-blocking poll of a sort.

John



On 05/12/06, Carl Trieloff <cc...@redhat.com> wrote: 



Tim Fox wrote:
>
>
> Das, Kapali Tejeswar wrote:
>> I am currently working on implementing the JMS queue browsing
>> functionality. Qpid uses java.util.concurrent.SynchronousQueue for its
>> internal queue for enqueing and dequeing messages. This class doesn't
>> allow to peek/browse messages in the queue. It only has methods to put a
>> message or get a message (with a timeout). 
>
> Right. Also, for JMS, you'll need to implement some kind of priority
> queue, which SynchronousQueue won't handle.

Tej,

In most use cases queue browsing can be counter productive for clients 
that want throughput or guarantees so we should not penalize the
non-browsing and priority use cases. I also think that we might need to
add to specification to complete this as I don't think it can be done on
0-8 without being very hacky ...  Have you been able to map this through
the current specification or is the starting point to define a peek cmd
needed in the spec that we can provide to the working group. The
interaction pattern and semantics between peeking the queue and messages 
being removed needs also be defined.


Carl.

Re: [java] Question on implementing queue browsing

[John O'Hara <jo...@gmail.com>]

I've asked Colin Crist if he'd mind sharing any ideas about how an idea
browser would work, given his work on HermesJMS...

Colin :-) ?

MULE uses queue browsing like this:
http://mule.mulesource.org/jira/browse/MULE-1004

Which implies it had better construct quickly, since they are likely to be
used as a non-blocking poll of a sort.

John


On 05/12/06, Carl Trieloff <cc...@redhat.com> wrote:
>
>
>
> Tim Fox wrote:
> >
> >
> > Das, Kapali Tejeswar wrote:
> >> I am currently working on implementing the JMS queue browsing
> >> functionality. Qpid uses java.util.concurrent.SynchronousQueue for its
> >> internal queue for enqueing and dequeing messages. This class doesn't
> >> allow to peek/browse messages in the queue. It only has methods to put
> a
> >> message or get a message (with a timeout).
> >
> > Right. Also, for JMS, you'll need to implement some kind of priority
> > queue, which SynchronousQueue won't handle.
>
> Tej,
>
> In most use cases queue browsing can be counter productive for clients
> that want throughput or guarantees so we should not penalize the
> non-browsing and priority use cases. I also think that we might need to
> add to specification to complete this as I don't think it can be done on
> 0-8 without being very hacky ...  Have you been able to map this through
> the current specification or is the starting point to define a peek cmd
> needed in the spec that we can provide to the working group. The
> interaction pattern and semantics between peeking the queue and messages
> being removed needs also be defined.
>
>
> Carl.
>
>
>

Re: [java] Question on implementing queue browsing

[Carl Trieloff <cc...@redhat.com>]

Tim Fox wrote:
>
>
> Das, Kapali Tejeswar wrote:
>> I am currently working on implementing the JMS queue browsing
>> functionality. Qpid uses java.util.concurrent.SynchronousQueue for its
>> internal queue for enqueing and dequeing messages. This class doesn't
>> allow to peek/browse messages in the queue. It only has methods to put a
>> message or get a message (with a timeout).
>
> Right. Also, for JMS, you'll need to implement some kind of priority 
> queue, which SynchronousQueue won't handle.

Tej,

In most use cases queue browsing can be counter productive for clients 
that want throughput or guarantees so we should not penalize the 
non-browsing and priority use cases. I also think that we might need to 
add to specification to complete this as I don't think it can be done on 
0-8 without being very hacky ...  Have you been able to map this through 
the current specification or is the starting point to define a peek cmd 
needed in the spec that we can provide to the working group. The 
interaction pattern and semantics between peeking the queue and messages 
being removed needs also be defined.


Carl.

Re: [java] Question on implementing queue browsing

[Tim Fox <ti...@jboss.com>]

Das, Kapali Tejeswar wrote:
> I am currently working on implementing the JMS queue browsing
> functionality. Qpid uses java.util.concurrent.SynchronousQueue for its
> internal queue for enqueing and dequeing messages. This class doesn't
> allow to peek/browse messages in the queue. It only has methods to put a
> message or get a message (with a timeout).

Right. Also, for JMS, you'll need to implement some kind of priority 
queue, which SynchronousQueue won't handle.

> 
>  
> 
> Any suggestions for how we should implement the queue browsing feature?
> I also noticed that there is no native implementation of queue browsing
> in Qpid.
> 
>  
> 
> Thanks and regards
> 
> Tejeswar
> 
> 

-- 
Tim Fox
JBoss Messaging Technical Lead
JBoss - a division of Red Hat
T: +44 2088006768
M: +44 7957983205
E: tim.fox@jboss.com tim.fox@redhat.com