You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by rammohan ganapavarapu <ra...@gmail.com> on 2016/10/24 16:04:10 UTC
qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Hi
I am running cpp broker v1.35 and i hve producers pumping messages and my
broker got crashed with bellow error, i know messages in queue exceeded the
configure queue limit but will it kill the broker? how to make my broker up
and running even if it exceed the configured queue capacity?
2016-10-24 15:36:21 [Broker] warning Exchange ax-ex-eaxgroup002 cannot
deliver to queue ax-q-eaxgroup002-consumer-group-001:
resource-limit-exceeded: Maximum depth exceeded on
ax-q-eaxgroup002-consumer-group-001: current=[count: 5843, size:
1073740027], max=[count: 1000000, size: 1073741824]
(/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662)
Current queue stats:
qpid-stat -q
Queues
queue dur autoDel excl msg
msgIn msgOut bytes bytesIn bytesOut cons bind
==========================================================================================================================
ax-q-eaxgroup002-consumer-group-001 Y 5.94k
5.94k 0 1.07g 1.07g 0 0 2
ax-q-eaxgroup002-consumer-group-001-dl Y 68
68 0 12.9m 12.9m 0 0 2
Queue configuration:
qpid-config queues
Queue Name Attributes
=================================================================
4dee1a3f-b4be-4580-9549-670510c0f075:0.0 auto-del excl
ax-q-eaxgroup002-consumer-group-001 --durable --file-size=5120
--file-count=64 --max-queue-size=1073741824 --max-queue-count=1000000
--limit-policy=flow-to-disk --argument no-local=False
ax-q-eaxgroup002-consumer-group-001-dl --durable --file-size=6000
--file-count=4 --max-queue-size=52428800 --max-queue-count=100000
--limit-policy=flow-to-disk --argument no-local=False
Thanks,
Ram
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by Robbie Gemmell <ro...@gmail.com>.
This isnt an area I have any experience with, but...
I don't particularly see evidence of a crash below, it looks like a
log message just warning that you hit the limits for the queue? If
that is the case, then you'd want to consume messages to make room for
more, or if the limits aren't to your needs then maybe youd instead
need to adjust them to more appropriate values for your scenario.
Robbie
On 24 October 2016 at 17:04, rammohan ganapavarapu
<ra...@gmail.com> wrote:
> Hi
>
> I am running cpp broker v1.35 and i hve producers pumping messages and my
> broker got crashed with bellow error, i know messages in queue exceeded the
> configure queue limit but will it kill the broker? how to make my broker up
> and running even if it exceed the configured queue capacity?
>
>
> 2016-10-24 15:36:21 [Broker] warning Exchange ax-ex-eaxgroup002 cannot
> deliver to queue ax-q-eaxgroup002-consumer-group-001:
> resource-limit-exceeded: Maximum depth exceeded on
> ax-q-eaxgroup002-consumer-group-001: current=[count: 5843, size:
> 1073740027], max=[count: 1000000, size: 1073741824]
> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp:1662)
>
> Current queue stats:
>
> qpid-stat -q
> Queues
> queue dur autoDel excl msg
> msgIn msgOut bytes bytesIn bytesOut cons bind
>
> ==========================================================================================================================
> ax-q-eaxgroup002-consumer-group-001 Y 5.94k
> 5.94k 0 1.07g 1.07g 0 0 2
> ax-q-eaxgroup002-consumer-group-001-dl Y 68
> 68 0 12.9m 12.9m 0 0 2
>
> Queue configuration:
>
> qpid-config queues
> Queue Name Attributes
> =================================================================
> 4dee1a3f-b4be-4580-9549-670510c0f075:0.0 auto-del excl
> ax-q-eaxgroup002-consumer-group-001 --durable --file-size=5120
> --file-count=64 --max-queue-size=1073741824 --max-queue-count=1000000
> --limit-policy=flow-to-disk --argument no-local=False
> ax-q-eaxgroup002-consumer-group-001-dl --durable --file-size=6000
> --file-count=4 --max-queue-size=52428800 --max-queue-count=100000
> --limit-policy=flow-to-disk --argument no-local=False
>
>
> Thanks,
> Ram
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by Gordon Sim <gs...@redhat.com>.
On 24/10/16 22:10, rammohan ganapavarapu wrote:
> Yes, looks like my system went out of memory and killed qpidd, i have
> restarted broker with trace log level, lets see if i can reproduce the
> issue.
>
> [45527559.740820] Out of memory: Kill process 8311 (qpidd) score 176 or
> sacrifice child
> [45527559.740827] Killed process 8311 (qpidd) total-vm:1753484kB,
> anon-rss:1356896kB, file-rss:48kB
You may want to tune the queue limits to prevent this. There is a
broker-wide default-queue-limit setting, but you can also set limits per
queue.
Also worth monitoring your queues - e.g. with periodic qpid-stat -q - to
see if/where you have unexpected build up of messages (e.g. possibly
'stale' subscription queues).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by rammohan ganapavarapu <ra...@gmail.com>.
Yes, looks like my system went out of memory and killed qpidd, i have
restarted broker with trace log level, lets see if i can reproduce the
issue.
[45527559.740820] Out of memory: Kill process 8311 (qpidd) score 176 or
sacrifice child
[45527559.740827] Killed process 8311 (qpidd) total-vm:1753484kB,
anon-rss:1356896kB, file-rss:48kB
Thanks,
Ram
On Mon, Oct 24, 2016 at 12:33 PM, Jakub Scholz <ja...@scholz.cz> wrote:
> Can you reproduce it?
>
> Maybe you can try to increase the log levels in the broker to get some more
> information about why it crashed (
> http://qpid.apache.org/releases/qpid-cpp-1.35.0/cpp-
> broker/book/ch01.html#RASC-logging-options).
> If the broker crashed, you can also try to get a core dump for further
> analysis. Following our discussion from the previous thread, it might be
> also interesting to check the system logs. When the broker is killed by the
> kernel because of missing memory, it also doesn't print anything into the
> log.
>
> J.
>
> On Mon, Oct 24, 2016 at 8:46 PM, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
> > on't know why it crashed but I don't see any other message in the log
> > other than that and broker processes is not running, i can run it in
> debug
> > mode and see if i can get any clue. So ideally if the queue siz
> >
>
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by Jakub Scholz <ja...@scholz.cz>.
Can you reproduce it?
Maybe you can try to increase the log levels in the broker to get some more
information about why it crashed (
http://qpid.apache.org/releases/qpid-cpp-1.35.0/cpp-broker/book/ch01.html#RASC-logging-options).
If the broker crashed, you can also try to get a core dump for further
analysis. Following our discussion from the previous thread, it might be
also interesting to check the system logs. When the broker is killed by the
kernel because of missing memory, it also doesn't print anything into the
log.
J.
On Mon, Oct 24, 2016 at 8:46 PM, rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> on't know why it crashed but I don't see any other message in the log
> other than that and broker processes is not running, i can run it in debug
> mode and see if i can get any clue. So ideally if the queue siz
>
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by rammohan ganapavarapu <ra...@gmail.com>.
I don't know why it crashed but I don't see any other message in the log
other than that and broker processes is not running, i can run it in debug
mode and see if i can get any clue. So ideally if the queue size limit
exceeds broker just drops the incoming messages till it flush out the
messages in the queue?
Thanks,
Ram
On Mon, Oct 24, 2016 at 11:38 AM, Jakub Scholz <ja...@scholz.cz> wrote:
> Hi Ram,
>
> This should normally not crash the broker. This error should be only sent
> to the client which exceeded the max queue size limit and that particular
> client might be kicked out. But the broker should continue running and
> serving other clients. Why do you think the broker crashed because of this?
>
> J.
>
> On Mon, Oct 24, 2016 at 6:04 PM, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
> > -10-24 15:36:21 [Broker] warning Exchange ax-ex-eaxgroup002 cannot
> > deliver to queue ax-q-eaxgroup002-consumer-group-001:
> > resource-limit-exceeded: Maximum depth exceeded on
> > ax-q-eaxgroup002-consumer-group-001: current=[count: 5843, size:
> > 1073740027], max=[count: 1000000, size: 1073741824]
> > (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp
> >
>
Re: qpid-cpp-1.35 broker crashed with resource-limit-exceeded
Posted by Jakub Scholz <ja...@scholz.cz>.
Hi Ram,
This should normally not crash the broker. This error should be only sent
to the client which exceeded the max queue size limit and that particular
client might be kicked out. But the broker should continue running and
serving other clients. Why do you think the broker crashed because of this?
J.
On Mon, Oct 24, 2016 at 6:04 PM, rammohan ganapavarapu <
rammohanganap@gmail.com> wrote:
> -10-24 15:36:21 [Broker] warning Exchange ax-ex-eaxgroup002 cannot
> deliver to queue ax-q-eaxgroup002-consumer-group-001:
> resource-limit-exceeded: Maximum depth exceeded on
> ax-q-eaxgroup002-consumer-group-001: current=[count: 5843, size:
> 1073740027], max=[count: 1000000, size: 1073741824]
> (/builddir/build/BUILD/qpid-cpp-1.35.0/src/qpid/broker/Queue.cpp
>