Unacknowledged messages locking up the journal?

[Joakim Sernbrant <Jo...@trioptima.com>]

Qpid 0.12

Scenario:

Fill a queue with durable messages
Receive one message but defer acknowledge of it
Drain the queue of the remaining messages

Sending a message on the queue will now generate an "Enqueue capacity threshold exceeded on queue" error. The queue now contains one message but the journal is full it seems.

Is this "working as expected"?
Is there a way to set a timeout on how long a message can go unacknowledged?
Is there a way to monitor journal free space?

See example blow for how to reproduce.

Regards,

Joakim

import qpid.messaging

addr = 'ack-test; { create: always, node: {type: queue, durable: True, x-declare: {arguments: {"qpid.file_count": 4, "qpid.file_size": 1}}}}'

def fill():
    conn = qpid.messaging.Connection('localhost')
    conn.open()
    ssn = conn.session()
    snd = ssn.sender(addr)
    n = 0
    while True:
        try:
            msg = qpid.messaging.Message(durable=True, content='foo')
            snd.send(msg)
            #print "fill:", msg
            n += 1
        except qpid.messaging.exceptions.ConnectionError, e:
            print e
            break
    print "sent %d messages"% n

def drain(count=None, ack=True):
    conn = qpid.messaging.Connection('localhost')
    conn.open()
    ssn = conn.session()
    rcv = ssn.receiver(addr)
    while count == None or count > 0:
        try:
            msg = rcv.fetch(timeout=0)
            if ack:
                ssn.acknowledge()
            if count != None:
                count -= 1
        except qpid.messaging.Empty:
            break

if __name__=="__main__":
    drain() # start with an empty queue
    fill()
    drain(count=1, ack=False)
    drain()
    fill() # will not be able to send any message

Re: Unacknowledged messages locking up the journal?

[Kim van der Riet <ki...@redhat.com>]

On Tue, 2012-04-24 at 17:53 +0100, Gordon Sim wrote:
> Kim, what does recordDepth track in the journals management schema?
> What 
> about available file count, would that be a reasonable approximation
> for 
> free space?
> 
recordDepth tracks the number of non-transactional or committed
transactional records enqueued on a queue (journal). It does not take
into account open transactions, nor the placement of the records on the
queue (which is important in a circular buffer). I would not recommend
using it to estimate how full the store is unless:

1. The enqueued records have a well-known (and preferably uniform) size;
2. Transactions are NOT in use;
3. Records are deqeued in the same order in which they were enqueued.

If these conditions are true, then the recordDepth can be used to
provide an estimate of the amount of used space in the store, and hence
the amount of free space.

Kim


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Unacknowledged messages locking up the journal?

[Gordon Sim <gs...@redhat.com>]

On 04/24/2012 01:10 PM, Joakim Sernbrant wrote:
> Qpid 0.12
>
> Scenario:
>
> Fill a queue with durable messages
> Receive one message but defer acknowledge of it
> Drain the queue of the remaining messages
>
> Sending a message on the queue will now generate an "Enqueue capacity threshold exceeded on queue" error. The queue now contains one message but the journal is full it seems.
>
> Is this "working as expected"?

Yes, I'm afraid it is.

> Is there a way to set a timeout on how long a message can go unacknowledged?

Not at present.

> Is there a way to monitor journal free space?

Kim, what does recordDepth track in the journals management schema? What 
about available file count, would that be a reasonable approximation for 
free space?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org