Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Michael Doberenz <Mi...@Valtech.com>]

Hello all,

We have a set of publishers each of which publish messages in order, and a single durable subscriber. We're not using transactions and we're set up to use client acknowledgment. If the subscriber is taken down in between the call of MessageListener.onMessage and Message.acknowledge, we are getting the retransmission of any unacknowledged messages, as expected. However, these messages are received after any newly published messages.  As an example:

A publisher sends messages 1, 2 and 3.
The subscriber receives and acks messages 1 and 2, but goes down in between receiving and acking message 3.
A publisher sends messages 4 and 5.
The subscriber restarts and receives message 4, then 5, then 3.

Is this expected behavior? If so, is there any way to expect these out of order messages?

We built our broker/client from revision 512501 of the java directory.

Thanks in advance,
Michael Doberenz

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Robert Godfrey <ro...@gmail.com>]

I think that topic subscription is, anyway, a special case.  More
particularly if there is an *exclusive* (in AMQP-speak) consumer on a queue,
if it recovers, rollsback or closes then we have a guarantee (since it's
exclusive) that there will be no other concurrent access to the queue...
further we know that there will be no other unacknowledged messages on the
queue other than those from the consumer which is
recovering/rolling-back/closing... which makes everything much easier to
reason about.

On 08/03/07, Gordon Sim <gs...@redhat.com> wrote:
>
> Martin Ritchie wrote:
> > Which would mean that the ordering 4,5,3 may be within the bounds of
> > JMS. However, if you had sent 5000 messages message 3 would arrive
> > after those 5000 as this is the default prefetch count in qpid. This
> > isn't ideal so the current head of qpid now flushes all the messages
> > from the client and requests new messages. Those messages that have
> > been flushed return to the front of the queue. However when priority
> > queues and message expiration are implemented the ordering will
> > reflect those values.
> >
>
> The behaviour was reported against rev. 512501, it looked to me like
> 510897 was the revision at which you introduced the ability to requeue
> at the head. Is that right? Were there subsequent fixes needed as well
> and if so what revs? Or perhaps more simply, is it worth trying again
> against the current head?
>

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Martin Ritchie <ri...@apache.org>]

Revision 510897 was an initial check in for QPID-346 that included the
broker changes but not the client changes.

Revision 515127 includes all the changes needed to flush the client
prefetch queues.

However, there does seem to be a reference counting/race condition
problem that I am working on just now. I believe I have fixed it but a
couple of the tests still fail so it is not quite there yet.


On 08/03/07, Gordon Sim <gs...@redhat.com> wrote:
> Martin Ritchie wrote:
> > Which would mean that the ordering 4,5,3 may be within the bounds of
> > JMS. However, if you had sent 5000 messages message 3 would arrive
> > after those 5000 as this is the default prefetch count in qpid. This
> > isn't ideal so the current head of qpid now flushes all the messages
> > from the client and requests new messages. Those messages that have
> > been flushed return to the front of the queue. However when priority
> > queues and message expiration are implemented the ordering will
> > reflect those values.
> >
>
> The behaviour was reported against rev. 512501, it looked to me like
> 510897 was the revision at which you introduced the ability to requeue
> at the head. Is that right? Were there subsequent fixes needed as well
> and if so what revs? Or perhaps more simply, is it worth trying again
> against the current head?
>


-- 
Martin Ritchie

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Gordon Sim <gs...@redhat.com>]

Martin Ritchie wrote:
> Which would mean that the ordering 4,5,3 may be within the bounds of
> JMS. However, if you had sent 5000 messages message 3 would arrive
> after those 5000 as this is the default prefetch count in qpid. This
> isn't ideal so the current head of qpid now flushes all the messages
> from the client and requests new messages. Those messages that have
> been flushed return to the front of the queue. However when priority
> queues and message expiration are implemented the ordering will
> reflect those values.
> 

The behaviour was reported against rev. 512501, it looked to me like 
510897 was the revision at which you introduced the ability to requeue 
at the head. Is that right? Were there subsequent fixes needed as well 
and if so what revs? Or perhaps more simply, is it worth trying again 
against the current head?

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Martin Ritchie <ri...@apache.org>]

On 07/03/07, Gordon Sim <gs...@redhat.com> wrote:
> Gordon Sim wrote:
> > Currently, messages are taken off the queue when sent to consumers then
> > requeued again (at the end of the queue) if the client fails to
> > acknowledge them, so yes it is expected in one sense.
>
> Just a quick update on this: I answered based on old knowledge of the
> codebase and gave an incorrect answer as a result - apologies!
>
> On a quick inspection of the latest code it appears that this issue has
> been at least considered as of rev 510897. As that seems to be later
> than the version you are running it appears it may not be working as
> expected. Martin, I believe you addressed this issue? If so you are
> probably in a better position to comment on the issue as raised?

The JMS Spec 4.4.11 says:
"A session's recover method is used to stop a session and restart it
with its first
unacknowledged message. In effect, the session's series of delivered messages
is reset to the point after its last acknowledged message. The messages it now
delivers may be different from those that were originally delivered due to
message expiration and the arrival of higher-priority messages.
A session must set the redelivered flag of messages it redelivers due to a
recovery."

Which would mean that the ordering 4,5,3 may be within the bounds of
JMS. However, if you had sent 5000 messages message 3 would arrive
after those 5000 as this is the default prefetch count in qpid. This
isn't ideal so the current head of qpid now flushes all the messages
from the client and requests new messages. Those messages that have
been flushed return to the front of the queue. However when priority
queues and message expiration are implemented the ordering will
reflect those values.

-- 
Martin Ritchie

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Gordon Sim <gs...@redhat.com>]

Gordon Sim wrote:
> Currently, messages are taken off the queue when sent to consumers then 
> requeued again (at the end of the queue) if the client fails to 
> acknowledge them, so yes it is expected in one sense. 

Just a quick update on this: I answered based on old knowledge of the 
codebase and gave an incorrect answer as a result - apologies!

On a quick inspection of the latest code it appears that this issue has 
been at least considered as of rev 510897. As that seems to be later 
than the version you are running it appears it may not be working as 
expected. Martin, I believe you addressed this issue? If so you are 
probably in a better position to comment on the issue as raised?

Re: Unacknowledged messages arrive after new messages when recreating a durable subscriber.

[Gordon Sim <gs...@redhat.com>]

Michael Doberenz wrote:
> We have a set of publishers each of which publish messages in order,
> and a single durable subscriber. We're not using transactions and
> we're set up to use client acknowledgment. If the subscriber is taken
> down in between the call of MessageListener.onMessage and
> Message.acknowledge, we are getting the retransmission of any
> unacknowledged messages, as expected. However, these messages are
> received after any newly published messages.  As an example:
> 
> A publisher sends messages 1, 2 and 3. The subscriber receives and
> acks messages 1 and 2, but goes down in between receiving and acking
> message 3. A publisher sends messages 4 and 5. The subscriber
> restarts and receives message 4, then 5, then 3.
> 
> Is this expected behavior? If so, is there any way to expect these
> out of order messages?

Currently, messages are taken off the queue when sent to consumers then 
requeued again (at the end of the queue) if the client fails to 
acknowledge them, so yes it is expected in one sense. In the scenario 
you describe though that behaviour is clearly not desirable. I'm afraid 
thats a bug in the current design.

Redelivered messages should have the redelivered flag set which can be 
used to alert applications that the message was delivered, not 
acknowledged and thus re-delivered. I'm not sure that helps much in your 
scenario though as you will already have processed 4 & 5 before 
realising that 3 is redelivered.

Order should really be preserved for redelivered messages as far as is 
possible, and in the case of a single consumer that would seem always to 
be possible.