You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Pavel Moravec <pm...@redhat.com> on 2011/12/13 16:33:43 UTC
Weird behavior of flow_to_disk policy
Hi all,
I noticed a weird behavior of flow-to-disk policy. When the relevant queue is utilized over its max-queue-count and when I am sending and receiving messages there, my sender gets "Enqueue capacity threshold".
One particular scenario:
qpid-config add queue ToFillQueue --durable --max-queue-count=1000 --limit-policy flow-to-disk --file-count 2 --file-size 2
./spout -c 1000 ToFillQueue
./spout -c 1000 ToFillQueue
(due to flow control, we have to break the program)
./drain -c 1000 ToFillQueue
./spout -c 1000 ToFillQueue
2011-12-13 16:15:50 warning Broker closed connection: 501, Enqueue capacity threshold exceeded on queue "ToFillQueue". (JournalImpl.cpp:587)
2011-12-13 16:15:50 error session-busy: Session detached by peer
session-busy: Session detached by peer
qpid-tool shows:
arguments {u'qpid.file_size': 2, u'qpid.file_count': 2, u'qpid.max_count': 1000, u'qpid.policy_type': u'flow_to_disk'}
altExchange absent
msgTotalEnqueues 2536
msgTotalDequeues 1000
msgTxnEnqueues 0
msgTxnDequeues 0
msgPersistEnqueues 0
msgPersistDequeues 0
msgDepth 1536
Interestingly, when I pump messages to the queue (without any draining) to fill the journals, the exception is raised when exactly the same number of msgTotalEnqueues appears.
Playing with more scenarios, sometimes the msgTotalEnqueues is higher, sometimes even lower than the total capacity of the queue(+its journals).
Almost the same result I achieve when sending durable messages (with the main difference that all messages go to journal direclt, not storing any in memory).
Anyway, it seems for me that consuming a message from flow-to-disk queue (when the policy is already applied) is wrongly implemented.
I noticed this on qpid 0.10 and 0.12 while 0.8 seems ok.
Kind regards,
Pavel
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Weird behavior of flow_to_disk policy
Posted by Gordon Sim <gs...@redhat.com>.
On 12/15/2011 03:14 PM, Pavel Moravec wrote:
> Hi Gordon,
>
> ----- Original Message -----
>> From: "Gordon Sim"<gs...@redhat.com>
>> To: users@qpid.apache.org
>> Sent: Thursday, December 15, 2011 3:33:22 PM
>> Subject: Re: Weird behavior of flow_to_disk policy
>>
>> On 12/13/2011 03:33 PM, Pavel Moravec wrote:
>>> Hi all,
>>> I noticed a weird behavior of flow-to-disk policy. When the
>>> relevant queue is utilized over its max-queue-count and when I am
>>> sending and receiving messages there, my sender gets "Enqueue
>>> capacity threshold".
>>>
>>> One particular scenario:
>>> qpid-config add queue ToFillQueue --durable --max-queue-count=1000
>>> --limit-policy flow-to-disk --file-count 2 --file-size 2
>>> ./spout -c 1000 ToFillQueue
>>> ./spout -c 1000 ToFillQueue
>>> (due to flow control, we have to break the program)
>>> ./drain -c 1000 ToFillQueue
>>
>> At this point the journal still has 1000 messages in it. The queue
>> also
>> has 1000 messages, so any further messages will be written to journal
>> as
>> part of the flow-to-disk behaviour...
> I was under the impression that flow_to_disk is applied when max-queue-count messages are in memory, not in the whole queue.. Then the behavior makes sense.
>
> So, this means that once FTD is applied, messages (msg bodies) will be stored to disk and not to memory until the queue depth drops below max-queue-count. And there will be messages on disk while memory not "fully utilized"(*) until there will be no message in the journal.
>
> (*)memory fully utilized = memory keeps max-queue-count messages.
>
> That is unexpected but in my opinion acceptable behavior - the second option in having max-queue-count in memory every time would gain some performance but more complex to implement.
Agreed. I don't like the current flow to disk behaviour at all (and I am
to blame for it!). It was a quick hack that I think only barely meets
*any* use case and certainly doesn't cover lots of important ones.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Weird behavior of flow_to_disk policy
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Gordon,
----- Original Message -----
> From: "Gordon Sim" <gs...@redhat.com>
> To: users@qpid.apache.org
> Sent: Thursday, December 15, 2011 3:33:22 PM
> Subject: Re: Weird behavior of flow_to_disk policy
>
> On 12/13/2011 03:33 PM, Pavel Moravec wrote:
> > Hi all,
> > I noticed a weird behavior of flow-to-disk policy. When the
> > relevant queue is utilized over its max-queue-count and when I am
> > sending and receiving messages there, my sender gets "Enqueue
> > capacity threshold".
> >
> > One particular scenario:
> > qpid-config add queue ToFillQueue --durable --max-queue-count=1000
> > --limit-policy flow-to-disk --file-count 2 --file-size 2
> > ./spout -c 1000 ToFillQueue
> > ./spout -c 1000 ToFillQueue
> > (due to flow control, we have to break the program)
> > ./drain -c 1000 ToFillQueue
>
> At this point the journal still has 1000 messages in it. The queue
> also
> has 1000 messages, so any further messages will be written to journal
> as
> part of the flow-to-disk behaviour...
I was under the impression that flow_to_disk is applied when max-queue-count messages are in memory, not in the whole queue.. Then the behavior makes sense.
So, this means that once FTD is applied, messages (msg bodies) will be stored to disk and not to memory until the queue depth drops below max-queue-count. And there will be messages on disk while memory not "fully utilized"(*) until there will be no message in the journal.
(*)memory fully utilized = memory keeps max-queue-count messages.
That is unexpected but in my opinion acceptable behavior - the second option in having max-queue-count in memory every time would gain some performance but more complex to implement.
>
> > ./spout -c 1000 ToFillQueue
> > 2011-12-13 16:15:50 warning Broker closed connection: 501, Enqueue
> > capacity threshold exceeded on queue "ToFillQueue".
> > (JournalImpl.cpp:587)
> > 2011-12-13 16:15:50 error session-busy: Session detached by peer
> > session-busy: Session detached by peer
>
> I think this is simply a journal capacity problem. It cannot
> accommodate
> 2000 messages. Journal size is 2*64k*2=256k, which assuming the
> smallest
> enqueue record of 128 bytes would allow for only 2000 in theory and
> in
> practice you get less (I believe due to other control records plus
> file
> boundaries etc).
Yes, I chosen the journal size just for this example. The journal itself stores 1536 of the smallest messages.
>
> > qpid-tool shows:
> >
> > arguments {u'qpid.file_size': 2,
> > u'qpid.file_count': 2, u'qpid.max_count': 1000,
> > u'qpid.policy_type': u'flow_to_disk'}
> > altExchange absent
> > msgTotalEnqueues 2536
> > msgTotalDequeues 1000
> > msgTxnEnqueues 0
> > msgTxnDequeues 0
> > msgPersistEnqueues 0
> > msgPersistDequeues 0
> > msgDepth 1536
> >
> > Interestingly, when I pump messages to the queue (without any
> > draining) to fill the journals, the exception is raised when
> > exactly the same number of msgTotalEnqueues appears.
> >
> > Playing with more scenarios, sometimes the msgTotalEnqueues is
> > higher, sometimes even lower than the total capacity of the
> > queue(+its journals).
> >
> > Almost the same result I achieve when sending durable messages
> > (with the main difference that all messages go to journal direclt,
> > not storing any in memory).
> >
> > Anyway, it seems for me that consuming a message from flow-to-disk
> > queue (when the policy is already applied) is wrongly implemented.
> >
> > I noticed this on qpid 0.10 and 0.12 while 0.8 seems ok.
> >
> > Kind regards,
> > Pavel
> >
> >
> >
> > ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Weird behavior of flow_to_disk policy
Posted by Gordon Sim <gs...@redhat.com>.
On 12/13/2011 03:33 PM, Pavel Moravec wrote:
> Hi all,
> I noticed a weird behavior of flow-to-disk policy. When the relevant queue is utilized over its max-queue-count and when I am sending and receiving messages there, my sender gets "Enqueue capacity threshold".
>
> One particular scenario:
> qpid-config add queue ToFillQueue --durable --max-queue-count=1000 --limit-policy flow-to-disk --file-count 2 --file-size 2
> ./spout -c 1000 ToFillQueue
> ./spout -c 1000 ToFillQueue
> (due to flow control, we have to break the program)
> ./drain -c 1000 ToFillQueue
At this point the journal still has 1000 messages in it. The queue also
has 1000 messages, so any further messages will be written to journal as
part of the flow-to-disk behaviour...
> ./spout -c 1000 ToFillQueue
> 2011-12-13 16:15:50 warning Broker closed connection: 501, Enqueue capacity threshold exceeded on queue "ToFillQueue". (JournalImpl.cpp:587)
> 2011-12-13 16:15:50 error session-busy: Session detached by peer
> session-busy: Session detached by peer
I think this is simply a journal capacity problem. It cannot accommodate
2000 messages. Journal size is 2*64k*2=256k, which assuming the smallest
enqueue record of 128 bytes would allow for only 2000 in theory and in
practice you get less (I believe due to other control records plus file
boundaries etc).
> qpid-tool shows:
>
> arguments {u'qpid.file_size': 2, u'qpid.file_count': 2, u'qpid.max_count': 1000, u'qpid.policy_type': u'flow_to_disk'}
> altExchange absent
> msgTotalEnqueues 2536
> msgTotalDequeues 1000
> msgTxnEnqueues 0
> msgTxnDequeues 0
> msgPersistEnqueues 0
> msgPersistDequeues 0
> msgDepth 1536
>
> Interestingly, when I pump messages to the queue (without any draining) to fill the journals, the exception is raised when exactly the same number of msgTotalEnqueues appears.
>
> Playing with more scenarios, sometimes the msgTotalEnqueues is higher, sometimes even lower than the total capacity of the queue(+its journals).
>
> Almost the same result I achieve when sending durable messages (with the main difference that all messages go to journal direclt, not storing any in memory).
>
> Anyway, it seems for me that consuming a message from flow-to-disk queue (when the policy is already applied) is wrongly implemented.
>
> I noticed this on qpid 0.10 and 0.12 while 0.8 seems ok.
>
> Kind regards,
> Pavel
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org