Re: Consumers hanging on a queue although there are messages in it

[Stefanic <sn...@movingintelligence.com>]

It has been a while so here's an update.
The same problem has been occurring on and off for the past two months now
and there is one suspect always coming back: message grouping.

We have found an tried several things and here are some of the findings:

*Message groups cache*
ActiveMQ defaults to an LRU cache with size 1024 for storing hashes of the
JMX message group header.
We where grouping on a higher number and could not find where to change this
setting so we went to 1024 message groups in code.
That did not help the 'hanging' problem at all.

*Broker page size*
Because the ActiveMQ broker sends all messages of a group to a single
consumer it needs to load messages in memory. When all messages in memory
are for a single consumer all other messages in the queue are not processed.
Max page size is the parameter to enable the broker to load more messages in
memory and hopefully will find messages for another consumer so flow is not
impacted heavily.

That problem with message groups and some kind of bug within client and/or
broker seems to trigger the hanging state.
When we simulate a lot of messages for a single broker, even within the max
page size, we encounter the hanging state issue (although lately another
variant, more below). Strangely after restart of the client and broker
failover the hanging state disappears so it must be something when running
for a while instead of a full queue when just started.

After changing the maxPageSize (increasing it from 1000 to 10000) we did see
a major decline in incidents, so that definitely has effect (and supporting
the theory above how that causes the hanging state).

The hanging state we encounter recently is a failover transport handler in
the client that seems to think the broker is down/unresponsive and blocks
all consumers for a specific timeout (3 seconds default I think). After that
timeout everything continues for a few seconds and the timeout is triggered
again in an endless loop.
Only way we know how to stop this is restarting the client and performing a
broker failover.

*Next steps*
We are now researching how number of consumers, maxPageSize and client
preFetch settings interact with each other to hopefully find a good setting
for all those parameters. Mostly because the number of consumers directly
affects the number of messages groups per consumer.

Also we upgraded to the latest activemq and camel client libraries and the
latest ActiveMQ broker.
The broker is running quite some time now and the issues continued, the
client libraries update will be released to production soon.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Consumers hanging on a queue although there are messages in it

[alan protasio <al...@gmail.com>]

I dont know if this is exactly what is happening with you but the code
below shows that if the broker memory is full, the broker will be unable to
page in messages (and dispatch them).

https://www.paste.org/98451

If this is your case this issue somewhat mitigated by
https://issues.apache.org/jira/browse/AMQ-7126

Alan Protasio


On Thu, May 2, 2019 at 11:31 AM alan protasio <al...@gmail.com> wrote:

> Hi...
>
> Did you guys see some log lines like the following?
>
>  WARN  Queue                          - queue://STORE, subscriptions=1,
> memory=0%, size=200, pending=0 cursor blocked, no space available to page
> in messages; usage: Usage(Main:memory:queue://STORE:memory)
> percentUsage=0%, usage=0, limit=1048576,
> percentUsageMinDelta=1%;Parent:Usage(Main:memory) percentUsage=1962%,
> usage=20581376, limit=1048576, percentUsageMinDelta=1%
>
> Att..
> Alan Diego
>
>
> On Mon, Apr 29, 2019 at 11:05 AM miksonx <mi...@gmail.com> wrote:
>
>> Hi
>> We are facing same issue as listed above where messages are Enqueued but
>> broker is stuck Dequeued messages.
>> Is there any update on this ?
>>
>> Thank you.
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>

Re: Consumers hanging on a queue although there are messages in it

[alan protasio <al...@gmail.com>]

Hi...

Did you guys see some log lines like the following?

 WARN  Queue                          - queue://STORE, subscriptions=1,
memory=0%, size=200, pending=0 cursor blocked, no space available to page
in messages; usage: Usage(Main:memory:queue://STORE:memory)
percentUsage=0%, usage=0, limit=1048576,
percentUsageMinDelta=1%;Parent:Usage(Main:memory) percentUsage=1962%,
usage=20581376, limit=1048576, percentUsageMinDelta=1%

Att..
Alan Diego


On Mon, Apr 29, 2019 at 11:05 AM miksonx <mi...@gmail.com> wrote:

> Hi
> We are facing same issue as listed above where messages are Enqueued but
> broker is stuck Dequeued messages.
> Is there any update on this ?
>
> Thank you.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Consumers hanging on a queue although there are messages in it

[miksonx <mi...@gmail.com>]

Hi
We are facing same issue as listed above where messages are Enqueued but
broker is stuck Dequeued messages. 
Is there any update on this ?

Thank you.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Consumers hanging on a queue although there are messages in it

[PG <gv...@gmail.com>]

Hello There,
We are facing identical issue with ActiveMQ 5.15.5 in our pre-prod
environment, where the client program stops reading messages from queue and
we observed this happens when the broker restarts. 

Untill we failover the broker to another node, client does not start
consuming messages again from the queue.

Are there any other findings or suggestions that can be implemented or
looked into for this hanging state? Any help is appreciated.

Also , we are using the statistics plugin to keep track of the connections.
Is there a possibility that this is causing the hung state. 

Thank you 
GV






--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html