You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Stefanic <sn...@movingintelligence.com> on 2018/01/22 11:56:35 UTC
Re: Consumers hanging on a queue although there are messages in it
Haris wrote
> Here is something more, really weird: Almost always, the following dirty
> hack "unblocked" the consumers: refreshing the queue browser page, i.e.
> http://localhost:8161/admin/queues.jsp
Hi Haris,
Did you ever find out what caused this behavior?
We recently upgraded both the ActiveMQ server and the ActiveMQ (and Camel)
client library and this problem has suddenly popped up for the first time..
We went from ActiveMQ server 5.11.1 to 5.15.2 and client library 5.12.2 to
5.15.2.
It does look like a problem in the server because restarting the client
results in the exact same behavior, and when opening the queue browser it is
resolved immediately.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by alan protasio <al...@gmail.com>.
I dont know if this is exactly what is happening with you but the code
below shows that if the broker memory is full, the broker will be unable to
page in messages (and dispatch them).
https://www.paste.org/98451
If this is your case this issue somewhat mitigated by
https://issues.apache.org/jira/browse/AMQ-7126
Alan Protasio
On Thu, May 2, 2019 at 11:31 AM alan protasio <al...@gmail.com> wrote:
> Hi...
>
> Did you guys see some log lines like the following?
>
> WARN Queue - queue://STORE, subscriptions=1,
> memory=0%, size=200, pending=0 cursor blocked, no space available to page
> in messages; usage: Usage(Main:memory:queue://STORE:memory)
> percentUsage=0%, usage=0, limit=1048576,
> percentUsageMinDelta=1%;Parent:Usage(Main:memory) percentUsage=1962%,
> usage=20581376, limit=1048576, percentUsageMinDelta=1%
>
> Att..
> Alan Diego
>
>
> On Mon, Apr 29, 2019 at 11:05 AM miksonx <mi...@gmail.com> wrote:
>
>> Hi
>> We are facing same issue as listed above where messages are Enqueued but
>> broker is stuck Dequeued messages.
>> Is there any update on this ?
>>
>> Thank you.
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>
Re: Consumers hanging on a queue although there are messages in it
Posted by alan protasio <al...@gmail.com>.
Hi...
Did you guys see some log lines like the following?
WARN Queue - queue://STORE, subscriptions=1,
memory=0%, size=200, pending=0 cursor blocked, no space available to page
in messages; usage: Usage(Main:memory:queue://STORE:memory)
percentUsage=0%, usage=0, limit=1048576,
percentUsageMinDelta=1%;Parent:Usage(Main:memory) percentUsage=1962%,
usage=20581376, limit=1048576, percentUsageMinDelta=1%
Att..
Alan Diego
On Mon, Apr 29, 2019 at 11:05 AM miksonx <mi...@gmail.com> wrote:
> Hi
> We are facing same issue as listed above where messages are Enqueued but
> broker is stuck Dequeued messages.
> Is there any update on this ?
>
> Thank you.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Consumers hanging on a queue although there are messages in it
Posted by miksonx <mi...@gmail.com>.
Hi
We are facing same issue as listed above where messages are Enqueued but
broker is stuck Dequeued messages.
Is there any update on this ?
Thank you.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by PG <gv...@gmail.com>.
Hello There,
We are facing identical issue with ActiveMQ 5.15.5 in our pre-prod
environment, where the client program stops reading messages from queue and
we observed this happens when the broker restarts.
Untill we failover the broker to another node, client does not start
consuming messages again from the queue.
Are there any other findings or suggestions that can be implemented or
looked into for this hanging state? Any help is appreciated.
Also , we are using the statistics plugin to keep track of the connections.
Is there a possibility that this is causing the hung state.
Thank you
GV
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Stefanic <sn...@movingintelligence.com>.
It has been a while so here's an update.
The same problem has been occurring on and off for the past two months now
and there is one suspect always coming back: message grouping.
We have found an tried several things and here are some of the findings:
*Message groups cache*
ActiveMQ defaults to an LRU cache with size 1024 for storing hashes of the
JMX message group header.
We where grouping on a higher number and could not find where to change this
setting so we went to 1024 message groups in code.
That did not help the 'hanging' problem at all.
*Broker page size*
Because the ActiveMQ broker sends all messages of a group to a single
consumer it needs to load messages in memory. When all messages in memory
are for a single consumer all other messages in the queue are not processed.
Max page size is the parameter to enable the broker to load more messages in
memory and hopefully will find messages for another consumer so flow is not
impacted heavily.
That problem with message groups and some kind of bug within client and/or
broker seems to trigger the hanging state.
When we simulate a lot of messages for a single broker, even within the max
page size, we encounter the hanging state issue (although lately another
variant, more below). Strangely after restart of the client and broker
failover the hanging state disappears so it must be something when running
for a while instead of a full queue when just started.
After changing the maxPageSize (increasing it from 1000 to 10000) we did see
a major decline in incidents, so that definitely has effect (and supporting
the theory above how that causes the hanging state).
The hanging state we encounter recently is a failover transport handler in
the client that seems to think the broker is down/unresponsive and blocks
all consumers for a specific timeout (3 seconds default I think). After that
timeout everything continues for a few seconds and the timeout is triggered
again in an endless loop.
Only way we know how to stop this is restarting the client and performing a
broker failover.
*Next steps*
We are now researching how number of consumers, maxPageSize and client
preFetch settings interact with each other to hopefully find a good setting
for all those parameters. Mostly because the number of consumers directly
affects the number of messages groups per consumer.
Also we upgraded to the latest activemq and camel client libraries and the
latest ActiveMQ broker.
The broker is running quite some time now and the issues continued, the
client libraries update will be released to production soon.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Stefanic <sn...@movingintelligence.com>.
Hi Tim,
It happened again yesterday so I made 2 CPU sampling snapshots.
They don't tell me much so if you'd like I can supply them both, contact me
at snicodem AT movingintelligence D com
Because it is our production environment we have reverted back to the
library versions of activemq and camel we did not see this issue with. The
versions are 5.12.2 and 2.15.6 respectively.
I did see some behavior in our code that needs further analysis/testing.
I will try and post again about any findings or when the downgrade does in
fact stop this issue from occurring.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you're able to analyze the snapshot, that's great. If you want me to, I
can. If you're using any custom broker plugins, it's possible that your
package, class, and method names will appear in the stack trace information
captured by the snapshot, so if you're using custom plugins, make sure you
consider whether any of that is sensitive before sharing it publicly on
this list.
Ideally you'd only capture the snapshot while the problem is occurring; the
more time you capture from when everything is normal, the harder it will be
to zero in on the behavior from when things are bad. So I recommend
capturing for a while, then if the problem hasn't started occurring after
an hour or two, stop the sampling session and start a new one.
Tim
On Feb 2, 2018 7:57 AM, "Stefanic" <sn...@movingintelligence.com> wrote:
> I will try to remember doing so, are you interested in the CPU sampling
> snapshot? And if so over what amount of time?
>
> (I'm not very experienced with profiling/sampling, only really done
> heapdump
> and threaddump analysis)
>
>
> Tim Bain wrote
> > Would you be able to use JVisualVM to perform CPU sampling while this is
> > occurring, to find out what those threads are doing? **WARNING** Do NOT
> do
> > CPU *profiling*, which would slow your broker significantly; sampling is
> > lightweight and generally held to be safe to do against an operational
> > process, but profiling is very heavyweight. Make sure you're on the right
> > tab in JVisualVM.
> >
> > Tim
> >
> > On Feb 2, 2018 12:05 AM, "Stefanic" <
>
> > snicodem@
>
> > > wrote:
> >
> >> Hi Tim,
> >>
> >> There are no selectors, so there are no messages left behind.
> >> We have seen this three times now in production and every time when the
> >> queue is empty (after about 30 minutes of refreshing) the problem goes
> >> away.
> >>
> >> Threads behavior is rather strange, when the blocking issue is on-going
> >> we
> >> see 1 thread constantly running (100%) in visualvm, and all other
> threads
> >> (31 in our case) almost seem synchronized on the same object because all
> >> of
> >> them start running at the same time and go into timed_waiting state at
> >> the
> >> same time.
> >> That results in 33% running of all other threads and that is not enough
> >> to
> >> keep our queue empty.
> >>
> >> Reproducing this behavior is nearly impossible for us, it occurs
> randomly
> >> within about a week.
> >>
> >> Here are the things we tried so far:
> >> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1
> >> but
> >> the problem occurred again on that version so we assume it is not the
> >> broker/server
> >> - Yesterday we downgraded both ActiveMQ and Camel client libraries:
> >> ActiveMQ
> >> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather
> >> only downgraded 1 at a time but it's our production environment so
> cannot
> >> play around too much)
> >>
> >> I will reply here if we experience the blocked state again.
> >>
> >>
> >>
> >> --
> >> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> >> f2341805.html
> >>
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Consumers hanging on a queue although there are messages in it
Posted by Stefanic <sn...@movingintelligence.com>.
I will try to remember doing so, are you interested in the CPU sampling
snapshot? And if so over what amount of time?
(I'm not very experienced with profiling/sampling, only really done heapdump
and threaddump analysis)
Tim Bain wrote
> Would you be able to use JVisualVM to perform CPU sampling while this is
> occurring, to find out what those threads are doing? **WARNING** Do NOT do
> CPU *profiling*, which would slow your broker significantly; sampling is
> lightweight and generally held to be safe to do against an operational
> process, but profiling is very heavyweight. Make sure you're on the right
> tab in JVisualVM.
>
> Tim
>
> On Feb 2, 2018 12:05 AM, "Stefanic" <
> snicodem@
> > wrote:
>
>> Hi Tim,
>>
>> There are no selectors, so there are no messages left behind.
>> We have seen this three times now in production and every time when the
>> queue is empty (after about 30 minutes of refreshing) the problem goes
>> away.
>>
>> Threads behavior is rather strange, when the blocking issue is on-going
>> we
>> see 1 thread constantly running (100%) in visualvm, and all other threads
>> (31 in our case) almost seem synchronized on the same object because all
>> of
>> them start running at the same time and go into timed_waiting state at
>> the
>> same time.
>> That results in 33% running of all other threads and that is not enough
>> to
>> keep our queue empty.
>>
>> Reproducing this behavior is nearly impossible for us, it occurs randomly
>> within about a week.
>>
>> Here are the things we tried so far:
>> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1
>> but
>> the problem occurred again on that version so we assume it is not the
>> broker/server
>> - Yesterday we downgraded both ActiveMQ and Camel client libraries:
>> ActiveMQ
>> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather
>> only downgraded 1 at a time but it's our production environment so cannot
>> play around too much)
>>
>> I will reply here if we experience the blocked state again.
>>
>>
>>
>> --
>> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
>> f2341805.html
>>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Tim Bain <tb...@alumni.duke.edu>.
Would you be able to use JVisualVM to perform CPU sampling while this is
occurring, to find out what those threads are doing? **WARNING** Do NOT do
CPU *profiling*, which would slow your broker significantly; sampling is
lightweight and generally held to be safe to do against an operational
process, but profiling is very heavyweight. Make sure you're on the right
tab in JVisualVM.
Tim
On Feb 2, 2018 12:05 AM, "Stefanic" <sn...@movingintelligence.com> wrote:
> Hi Tim,
>
> There are no selectors, so there are no messages left behind.
> We have seen this three times now in production and every time when the
> queue is empty (after about 30 minutes of refreshing) the problem goes
> away.
>
> Threads behavior is rather strange, when the blocking issue is on-going we
> see 1 thread constantly running (100%) in visualvm, and all other threads
> (31 in our case) almost seem synchronized on the same object because all of
> them start running at the same time and go into timed_waiting state at the
> same time.
> That results in 33% running of all other threads and that is not enough to
> keep our queue empty.
>
> Reproducing this behavior is nearly impossible for us, it occurs randomly
> within about a week.
>
> Here are the things we tried so far:
> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1 but
> the problem occurred again on that version so we assume it is not the
> broker/server
> - Yesterday we downgraded both ActiveMQ and Camel client libraries:
> ActiveMQ
> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather
> only downgraded 1 at a time but it's our production environment so cannot
> play around too much)
>
> I will reply here if we experience the blocked state again.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Consumers hanging on a queue although there are messages in it
Posted by Stefanic <sn...@movingintelligence.com>.
Hi Tim,
There are no selectors, so there are no messages left behind.
We have seen this three times now in production and every time when the
queue is empty (after about 30 minutes of refreshing) the problem goes away.
Threads behavior is rather strange, when the blocking issue is on-going we
see 1 thread constantly running (100%) in visualvm, and all other threads
(31 in our case) almost seem synchronized on the same object because all of
them start running at the same time and go into timed_waiting state at the
same time.
That results in 33% running of all other threads and that is not enough to
keep our queue empty.
Reproducing this behavior is nearly impossible for us, it occurs randomly
within about a week.
Here are the things we tried so far:
- Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1 but
the problem occurred again on that version so we assume it is not the
broker/server
- Yesterday we downgraded both ActiveMQ and Camel client libraries: ActiveMQ
down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather
only downgraded 1 at a time but it's our production environment so cannot
play around too much)
I will reply here if we experience the blocked state again.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Tim Bain <tb...@alumni.duke.edu>.
Are any selectors in use on the consumers of the queue that's getting
"blocked"? If so, are there any messages published that don't match any of
the selectors?
If you take a thread dump on the broker, where do you see threads waiting?
Can you reproduce this on demand in a test environment with a minimum setup
(one queue, one or two consumers, a known, repeatable set of messages), or
is this the type of problem that only occurs in ops after weeks or months
of heavy usage? Hopefully it's the former, because the latter problem is
usually very hard to diagnose.
Tim
On Jan 22, 2018 4:56 AM, "Stefanic" <sn...@movingintelligence.com> wrote:
> Haris wrote
> > Here is something more, really weird: Almost always, the following dirty
> > hack "unblocked" the consumers: refreshing the queue browser page, i.e.
> > http://localhost:8161/admin/queues.jsp
>
> Hi Haris,
>
> Did you ever find out what caused this behavior?
>
> We recently upgraded both the ActiveMQ server and the ActiveMQ (and Camel)
> client library and this problem has suddenly popped up for the first time..
>
> We went from ActiveMQ server 5.11.1 to 5.15.2 and client library 5.12.2 to
> 5.15.2.
>
> It does look like a problem in the server because restarting the client
> results in the exact same behavior, and when opening the queue browser it
> is
> resolved immediately.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Consumers hanging on a queue although there are messages in it
Posted by Stefanic <sn...@movingintelligence.com>.
Hi Haris,
We also have this issue on a queue with high activity, in our case we do
more than 3 million messages a day (spread out over the day so never a
moment with 0 activity).
But this was never an issue before we updated..
See my previous post today, if we still encounter this issue again with the
minor downgrade we will return to the original versions where everything
worked fine (yes, don't fix/update anything if it ain't broken).
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Consumers hanging on a queue although there are messages in it
Posted by Haris <ha...@gmail.com>.
Hi Stefanic,
Unfortunately not. It appeared again, and then went away. It /looks like/
it's related to a "lot of activity" on a queue: lots of messages and/or lots
of consumer (but more probably the first: lots of messages). However, the
root cause we haven't been able to find. Especially since we have faced
another problem, and that are restarts every now and then without a clear
reason.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html