Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

I am getting the following error in my broker. Although nothing is crashing
(at least so far)

*2013-01-31 01:53:42 [Broker] warning Exception on notification of dequeue
for queue LVQTest-5Min-3: Flow limit count underflow on dequeue.
Queue=LVQTest-5Min-3 (qpid\broker\QueueFlowLimit.cpp:184)*

Any idea why I am getting this what affect does this have ? How I could
resolve it ?

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 01/31/2013 09:21 AM, Gordon Sim wrote:
> On 01/31/2013 09:10 AM, Rajesh Khan wrote:
>> I am still looking for an answer to this question however I just
>> wanted to
>> add that my sender is sending messages at a very high frequency and it
>> has
>> a capcaity of 10 (requires ack from broker after 10 messages) and my
>> receiver has a capacity of 20 and acknowledges after 10 messages. Any
>> chance these factors might be affecting the Broker ?
>
> No. I haven't ha d time to go through your log in detail yet. However if
> you turn of the max-size on the queue (by specifying 0 as the value) the
> problem won't occur, so that may be acceptable workaround in the meantime.


One other useful piece of information would be the management stats for 
the queue atthe point the error occurs (you can get that through e.g. 
qpid-stat -q)


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 02/01/2013 01:13 PM, Rajesh Khan wrote:
> I currently have a sender(C++) and a receiver(C#). The sender sends about
> 30-40 messages in a millisecond. I have set the capacity of the sender to
> 40,000 and the receiver capacity to 0. The receiver also sends back
> acknowledgment after 30,000 messages.

I would recommend setting your receiver to browse mode (e.g. via an 
address of the form 'my-queue; {mode: browse}'). That should bring your 
memory usage down a lot. If you can't browse for some reason, then I 
would acknowledge much for frequently.

I don't understand why you want the sender capacity to be so high 
either. I would probably leave the default value there unless you have a 
reason to change it.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 02/04/2013 04:26 PM, Rajesh Khan wrote:
> Hi Gordon turning out setting to Browse fixed the memory issue. Also one of
> the queues did not having a limit on it. Setting a limit resolved the
> issue. Thanks

Great, thanks for updating us! I still think you have most likely 
uncovered some bug, but haven't yet had time to try and reproduce it. I 
haven't forgotten about it though.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

Hi Gordon turning out setting to Browse fixed the memory issue. Also one of
the queues did not having a limit on it. Setting a limit resolved the
issue. Thanks

On Fri, Feb 1, 2013 at 6:42 AM, Rajesh Khan <ra...@gmail.com> wrote:

> Just looked at the code and to my surprise I am not browsing at that queue
> - while I am at other LVQ. I thought that would not make much of a
> difference. I'll add the browsing option as soon as I get to the code.
> Could you kindly explain what effect browsing would have ?
>
>
>
> On Fri, Feb 1, 2013 at 6:42 AM, Gordon Sim <gs...@redhat.com> wrote:
>
>> On 02/01/2013 01:30 PM, Rajesh Khan wrote:
>>
>>> I'll be running the same project in an hour. However with different
>>> capacity sizes as you suggested earlier (the sender C++ will have a
>>> capacity of 300 and the receiver C# will have a 0 capacity and return
>>> acknowledgment of messages after every 10 messages). Maybe this might
>>> help.I am almost 80% sure this was the warning exception that I saw.
>>> However Ill be certain to note the exception that comes up and post it
>>> back. Also did you have any thoughts on the memory issue that I just
>>> mentioned. Why the broker released all the memory only when the C#
>>> receiver
>>> reconnected.(I had to disconnect the receiver and reconnect).
>>>
>>
>> My guess is that there were still a lot of unacknowledged messages being
>> tracked, and closing the receiver allowed the broker to cleanup its records
>> of those. Browsing or acking more frequently would reduce the number of
>> deliveries the broker was tracking in this regard.
>>
>> Can I ask if there is a reason you are not browsing? (In that case the
>> queue would always hold the last value for each key; the receiver would
>> never delete anything from the queue and messages would be removed only
>> when replaced by an update with the same key).
>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

Just looked at the code and to my surprise I am not browsing at that queue
- while I am at other LVQ. I thought that would not make much of a
difference. I'll add the browsing option as soon as I get to the code.
Could you kindly explain what effect browsing would have ?


On Fri, Feb 1, 2013 at 6:42 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 02/01/2013 01:30 PM, Rajesh Khan wrote:
>
>> I'll be running the same project in an hour. However with different
>> capacity sizes as you suggested earlier (the sender C++ will have a
>> capacity of 300 and the receiver C# will have a 0 capacity and return
>> acknowledgment of messages after every 10 messages). Maybe this might
>> help.I am almost 80% sure this was the warning exception that I saw.
>> However Ill be certain to note the exception that comes up and post it
>> back. Also did you have any thoughts on the memory issue that I just
>> mentioned. Why the broker released all the memory only when the C#
>> receiver
>> reconnected.(I had to disconnect the receiver and reconnect).
>>
>
> My guess is that there were still a lot of unacknowledged messages being
> tracked, and closing the receiver allowed the broker to cleanup its records
> of those. Browsing or acking more frequently would reduce the number of
> deliveries the broker was tracking in this regard.
>
> Can I ask if there is a reason you are not browsing? (In that case the
> queue would always hold the last value for each key; the receiver would
> never delete anything from the queue and messages would be removed only
> when replaced by an update with the same key).
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 02/01/2013 01:30 PM, Rajesh Khan wrote:
> I'll be running the same project in an hour. However with different
> capacity sizes as you suggested earlier (the sender C++ will have a
> capacity of 300 and the receiver C# will have a 0 capacity and return
> acknowledgment of messages after every 10 messages). Maybe this might
> help.I am almost 80% sure this was the warning exception that I saw.
> However Ill be certain to note the exception that comes up and post it
> back. Also did you have any thoughts on the memory issue that I just
> mentioned. Why the broker released all the memory only when the C# receiver
> reconnected.(I had to disconnect the receiver and reconnect).

My guess is that there were still a lot of unacknowledged messages being 
tracked, and closing the receiver allowed the broker to cleanup its 
records of those. Browsing or acking more frequently would reduce the 
number of deliveries the broker was tracking in this regard.

Can I ask if there is a reason you are not browsing? (In that case the 
queue would always hold the last value for each key; the receiver would 
never delete anything from the queue and messages would be removed only 
when replaced by an update with the same key).

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

I'll be running the same project in an hour. However with different
capacity sizes as you suggested earlier (the sender C++ will have a
capacity of 300 and the receiver C# will have a 0 capacity and return
acknowledgment of messages after every 10 messages). Maybe this might
help.I am almost 80% sure this was the warning exception that I saw.
However Ill be certain to note the exception that comes up and post it
back. Also did you have any thoughts on the memory issue that I just
mentioned. Why the broker released all the memory only when the C# receiver
reconnected.(I had to disconnect the receiver and reconnect).


On Fri, Feb 1, 2013 at 6:23 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 02/01/2013 01:13 PM, Rajesh Khan wrote:
>
>> Hi Gordon , thank you for your reply. Here is the result of qpid-config
>> for
>> the specific queue
>> *
>> LVQChart-5Min-2   --max-queue-size=0
>> --max-queue-count=0 --limit-policy=ring --lvq-key=Symbol*
>>
>
> [...]
>
>  For now my aim is not to get this exception message. It seems setting the
>> queue size to 0 isn't helping.
>>
>
> I don't believe it is possible to get the exception you originally posted
> with a queue as configured above (with both size and count set to 0 the
> codepath on which it occurs will not even be used on the queue in
> question). Can you post the latest error you see?
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 02/01/2013 01:13 PM, Rajesh Khan wrote:
> Hi Gordon , thank you for your reply. Here is the result of qpid-config for
> the specific queue
> *
> LVQChart-5Min-2   --max-queue-size=0
> --max-queue-count=0 --limit-policy=ring --lvq-key=Symbol*

[...]
> For now my aim is not to get this exception message. It seems setting the
> queue size to 0 isn't helping.

I don't believe it is possible to get the exception you originally 
posted with a queue as configured above (with both size and count set to 
0 the codepath on which it occurs will not even be used on the queue in 
question). Can you post the latest error you see?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

Hi Gordon , thank you for your reply. Here is the result of qpid-config for
the specific queue
*
LVQChart-5Min-2   --max-queue-size=0
--max-queue-count=0 --limit-policy=ring --lvq-key=Symbol*

It seems that the queue size did get passed.Also let me explain my scenario
a little more in detail maybe that might help.
I currently have a sender(C++) and a receiver(C#). The sender sends about
30-40 messages in a millisecond. I have set the capacity of the sender to
40,000 and the receiver capacity to 0. The receiver also sends back
acknowledgment after 30,000 messages.The reason for having a 0 at the
receiver side is because the queue is a LVQ type and I always want the
latest value from it setting a capacity gets previous values so I have set
a 0 in order to prevent collecting messages at the receiving side for now i
do not mind the round trips.Anyways with this configuration both the sender
and receiver worked very well for a while (however I did realize from the
task manager that the memory consumption of Qpid broker got upto 9 GB after
that I started receiving this warning for every message that the sender
sent.Also the transmitter got out of synch with the receiver.This means
that the transmitter stopped transmitting but the receiver was still
receiving messages from the broker.Also I noticed after all the messages
had been read (as I assume since the broker console just stopped displaying
the messages). The memory 9GB did not go down.It only went down altogether
when i stopped my C# receiver and started it again (the receiver started
reading messages Instantly when it reconnected to the broker). I am still
not sure what happened here. Why did I have to restart my C# receiver.No
exceptions were logged from the receiver or the sender side.I dont really
understand this behavior any suggestions on what might be going on. Here
are my questions
*
1-The memory 9GB did not go down.It only went down altogether when I
stopped my C# receiver and started it again (the receiver started reading
messages Instantly when it reconnected to the broker)*
One theory I have regarding this behavior is that after I started getting
that exception (when things got out of syn (i.e) the receiver wasn't
displaying the latest value) is that the broker wasn't sending messages to
the receiver.And when the receiver disconnected and reconnected it
retrieved all the read messages and memory of the broker went down
altogether. If this theory is correct then why was the receiver still
updating (although out of sync)  messages it had receiver from the broker
after that exception showed up.I would appreciate it if someone could
explain this behavior too.

For now my aim is not to get this exception message. It seems setting the
queue size to 0 isn't helping. Any suggestions/ideas would definitely be
appreciated and welcomed



On Fri, Feb 1, 2013 at 1:59 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 01/31/2013 06:39 PM, Rajesh Khan wrote:
>
>> Gordon , any suggestion what might be causing this warning exception ?
>>
>
> It comes from some internal error around the tracking of the aggregate
> queue size. When a message is enqueued the size is incremented, when the
> message is dequeued the size is decremented. This error indicates and
> attempt to dequeue more data than it was actually aware of, which means
> something internally is broken.
>
>  This exception shows up in the broker after an hour of running my
>> application.
>> I did try setting up a queu with max size = 0 using the following command
>>
>> *python qpid-config add queue LVQTest-5Min-3 --argument
>> qpid.last_value_queue_key=**MChartSymbol --max-queue-size=0*
>>
>
> If there is no maximum size then you couldn't get the same message as the
> size counter would not be tracked. Perhaps the setting didn't get passed
> through correctly... Does qpid-config queues echo the correct settings back
> to you?
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 01/31/2013 06:39 PM, Rajesh Khan wrote:
> Gordon , any suggestion what might be causing this warning exception ?

It comes from some internal error around the tracking of the aggregate 
queue size. When a message is enqueued the size is incremented, when the 
message is dequeued the size is decremented. This error indicates and 
attempt to dequeue more data than it was actually aware of, which means 
something internally is broken.

> This exception shows up in the broker after an hour of running my
> application.
> I did try setting up a queu with max size = 0 using the following command
>
> *python qpid-config add queue LVQTest-5Min-3 --argument
> qpid.last_value_queue_key=MChartSymbol --max-queue-size=0*

If there is no maximum size then you couldn't get the same message as 
the size counter would not be tracked. Perhaps the setting didn't get 
passed through correctly... Does qpid-config queues echo the correct 
settings back to you?


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

Gordon , any suggestion what might be causing this warning exception ?
This exception shows up in the broker after an hour of running my
application.
I did try setting up a queu with max size = 0 using the following command

*python qpid-config add queue LVQTest-5Min-3 --argument
qpid.last_value_queue_key=MChartSymbol --max-queue-size=0*


On Thu, Jan 31, 2013 at 2:46 AM, Rajesh Khan <ra...@gmail.com>wrote:

> I am using qpid-0.18
> BTW now i am setting a max queue size of 0 and I am still getting the same
> message.
> Here is how i am specifying it:
>
> *python qpid-config add queue LVQTest-5Min-3 --argument
> qpid.last_value_queue_key=MChartSymbol --max-queue-size=0
> *
> Any suggestions?
>
>
>
> On Thu, Jan 31, 2013 at 2:40 AM, Gordon Sim <gs...@redhat.com> wrote:
>
>> On 01/31/2013 09:22 AM, Rajesh Khan wrote:
>>
>>> Thank you for your reply. Out of curiosity what is the default que limit
>>> when it is created ? Does it vary or each system ?
>>>
>>
>> It is configurable via the --default-queue-limit option to qpidd. What
>> versoin are you using btw?
>>
>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

I am using qpid-0.18
BTW now i am setting a max queue size of 0 and I am still getting the same
message.
Here is how i am specifying it:

*python qpid-config add queue LVQTest-5Min-3 --argument
qpid.last_value_queue_key=MChartSymbol --max-queue-size=0
*
Any suggestions?


On Thu, Jan 31, 2013 at 2:40 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 01/31/2013 09:22 AM, Rajesh Khan wrote:
>
>> Thank you for your reply. Out of curiosity what is the default que limit
>> when it is created ? Does it vary or each system ?
>>
>
> It is configurable via the --default-queue-limit option to qpidd. What
> versoin are you using btw?
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 01/31/2013 09:22 AM, Rajesh Khan wrote:
> Thank you for your reply. Out of curiosity what is the default que limit
> when it is created ? Does it vary or each system ?

It is configurable via the --default-queue-limit option to qpidd. What 
versoin are you using btw?


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

Thank you for your reply. Out of curiosity what is the default que limit
when it is created ? Does it vary or each system ?

On Thu, Jan 31, 2013 at 2:21 AM, Gordon Sim <gs...@redhat.com> wrote:

> On 01/31/2013 09:10 AM, Rajesh Khan wrote:
>
>> I am still looking for an answer to this question however I just wanted to
>> add that my sender is sending messages at a very high frequency and it has
>> a capcaity of 10 (requires ack from broker after 10 messages) and my
>> receiver has a capacity of 20 and acknowledges after 10 messages. Any
>> chance these factors might be affecting the Broker ?
>>
>
> No. I haven't ha d time to go through your log in detail yet. However if
> you turn of the max-size on the queue (by specifying 0 as the value) the
> problem won't occur, so that may be acceptable workaround in the meantime.
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.**org<us...@qpid.apache.org>
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Gordon Sim <gs...@redhat.com>]

On 01/31/2013 09:10 AM, Rajesh Khan wrote:
> I am still looking for an answer to this question however I just wanted to
> add that my sender is sending messages at a very high frequency and it has
> a capcaity of 10 (requires ack from broker after 10 messages) and my
> receiver has a capacity of 20 and acknowledges after 10 messages. Any
> chance these factors might be affecting the Broker ?

No. I haven't ha d time to go through your log in detail yet. However if 
you turn of the max-size on the queue (by specifying 0 as the value) the 
problem won't occur, so that may be acceptable workaround in the meantime.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Getting "[Broker] warning Exception on notification of dequeue" What does this mean ?

[Rajesh Khan <ra...@gmail.com>]

I am still looking for an answer to this question however I just wanted to
add that my sender is sending messages at a very high frequency and it has
a capcaity of 10 (requires ack from broker after 10 messages) and my
receiver has a capacity of 20 and acknowledges after 10 messages. Any
chance these factors might be affecting the Broker ?

On Thu, Jan 31, 2013 at 1:57 AM, Rajesh Khan <ra...@gmail.com>wrote:

> I am getting the following error in my broker. Although nothing is
> crashing (at least so far)
>
> *2013-01-31 01:53:42 [Broker] warning Exception on notification of
> dequeue for queue LVQTest-5Min-3: Flow limit count underflow on dequeue.
> Queue=LVQTest-5Min-3 (qpid\broker\QueueFlowLimit.cpp:184)*
>
> Any idea why I am getting this what affect does this have ? How I could
> resolve it ?
>