You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/09/19 23:55:38 UTC
All messages getting stuck in 5.10 until restart…
We’ve been having this problem for a while now but it’s becoming more
apparent as we deploy in production.
Essentially all messages in ActiveMQ get stuck until I restart it. Today
it took three restarts… then it just started working.
Here’s my config:
http://pastebin.com/F5L99biA
We’re using LevelDB… the box seems to be idle while this is happening.
There isn’t much data either. < 1GB… should all fit into memory.
It’s entirely possible it’s the client … but I’m using the official
activemq client.
I think I might try browsing the queue from the command line next time this
happens.
Any other pointers to help track this down?
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: All messages getting stuck in 5.10 until restart…
Posted by James Green <ja...@gmail.com>.
Sounds like the issue I was observing last month. ActiveMQ was randomly
deciding to just "stop" delivering messages to consumers - both the shipped
web console and hawtio showed things absolutely as I would expect them to
be.
I ended up suspending new operations, allowed existing data to be processed
and shut everything down. There should have been no data but kahadb kept a
few hundred megs for reasons unknown. I moved the storage folder and
started things back up. No problems since.
This whole thing started after (first observed two weeks after) we upgraded
from 5.7.0 to 5.10.0 and only one of our fleet experienced the problem.
On 20 September 2014 01:13, Kevin Burton <bu...@spinn3r.com> wrote:
> though it’s entirely possible that flow control is still enabled.. maybe
> the config isn’t parsed correctly. Is there a way to verify ? It isn’t in
> the UI.
>
> On Fri, Sep 19, 2014 at 5:09 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > - I ruled out memory issues… I tuned up memory to 3G (should be plenty)
> > and enabled GC logging and there are no full GCs.
> >
> > - I disabled producerFlowControl…
> >
> > <policyEntry queue=">"
> > producerFlowControl="false"
> > advisoryForConsumed="true"
> > gcInactiveDestinations="true"
> > inactiveTimoutBeforeGC="60000”>
> >
> > … still experiencing the same symptoms… A restart of ActiveMQ resolve the
> > issue for about 5 minutes… then my workers come to a crawl :-(…
> >
> >
> > On Fri, Sep 19, 2014 at 3:28 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> >> Use JConsole to look at which consumers are connected (make sure at
> least
> >> one is!), how many messages they're currently working on
> >> (MessageCountAwaitingAcknowlegement), and whether the numbers appear to
> be
> >> increasing. For each queue, confirm whether the EnqueueCount is
> >> increasing; if it's not, the broker isn't accepting messages due to
> >> Producer Flow Control, or the producer isn't attempting to produce any
> for
> >> other reasons.
> >>
> >> On Fri, Sep 19, 2014 at 3:55 PM, Kevin Burton <bu...@spinn3r.com>
> wrote:
> >>
> >> > We’ve been having this problem for a while now but it’s becoming more
> >> > apparent as we deploy in production.
> >> >
> >> > Essentially all messages in ActiveMQ get stuck until I restart it.
> >> Today
> >> > it took three restarts… then it just started working.
> >> >
> >> > Here’s my config:
> >> >
> >> > http://pastebin.com/F5L99biA
> >> >
> >> > We’re using LevelDB… the box seems to be idle while this is happening.
> >> >
> >> > There isn’t much data either. < 1GB… should all fit into memory.
> >> >
> >> > It’s entirely possible it’s the client … but I’m using the official
> >> > activemq client.
> >> >
> >> > I think I might try browsing the queue from the command line next time
> >> this
> >> > happens.
> >> >
> >> > Any other pointers to help track this down?
> >> >
> >> > --
> >> >
> >> > Founder/CEO Spinn3r.com
> >> > Location: *San Francisco, CA*
> >> > blog: http://burtonator.wordpress.com
> >> > … or check out my Google+ profile
> >> > <https://plus.google.com/102718274791889610666/posts>
> >> > <http://spinn3r.com>
> >> >
> >>
> >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
> >
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Re: All messages getting stuck in 5.10 until restart…
Posted by Kevin Burton <bu...@spinn3r.com>.
though it’s entirely possible that flow control is still enabled.. maybe
the config isn’t parsed correctly. Is there a way to verify ? It isn’t in
the UI.
On Fri, Sep 19, 2014 at 5:09 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> - I ruled out memory issues… I tuned up memory to 3G (should be plenty)
> and enabled GC logging and there are no full GCs.
>
> - I disabled producerFlowControl…
>
> <policyEntry queue=">"
> producerFlowControl="false"
> advisoryForConsumed="true"
> gcInactiveDestinations="true"
> inactiveTimoutBeforeGC="60000”>
>
> … still experiencing the same symptoms… A restart of ActiveMQ resolve the
> issue for about 5 minutes… then my workers come to a crawl :-(…
>
>
> On Fri, Sep 19, 2014 at 3:28 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
>
>> Use JConsole to look at which consumers are connected (make sure at least
>> one is!), how many messages they're currently working on
>> (MessageCountAwaitingAcknowlegement), and whether the numbers appear to be
>> increasing. For each queue, confirm whether the EnqueueCount is
>> increasing; if it's not, the broker isn't accepting messages due to
>> Producer Flow Control, or the producer isn't attempting to produce any for
>> other reasons.
>>
>> On Fri, Sep 19, 2014 at 3:55 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>> > We’ve been having this problem for a while now but it’s becoming more
>> > apparent as we deploy in production.
>> >
>> > Essentially all messages in ActiveMQ get stuck until I restart it.
>> Today
>> > it took three restarts… then it just started working.
>> >
>> > Here’s my config:
>> >
>> > http://pastebin.com/F5L99biA
>> >
>> > We’re using LevelDB… the box seems to be idle while this is happening.
>> >
>> > There isn’t much data either. < 1GB… should all fit into memory.
>> >
>> > It’s entirely possible it’s the client … but I’m using the official
>> > activemq client.
>> >
>> > I think I might try browsing the queue from the command line next time
>> this
>> > happens.
>> >
>> > Any other pointers to help track this down?
>> >
>> > --
>> >
>> > Founder/CEO Spinn3r.com
>> > Location: *San Francisco, CA*
>> > blog: http://burtonator.wordpress.com
>> > … or check out my Google+ profile
>> > <https://plus.google.com/102718274791889610666/posts>
>> > <http://spinn3r.com>
>> >
>>
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: All messages getting stuck in 5.10 until restart…
Posted by Kevin Burton <bu...@spinn3r.com>.
- I ruled out memory issues… I tuned up memory to 3G (should be plenty) and
enabled GC logging and there are no full GCs.
- I disabled producerFlowControl…
<policyEntry queue=">"
producerFlowControl="false"
advisoryForConsumed="true"
gcInactiveDestinations="true"
inactiveTimoutBeforeGC="60000”>
… still experiencing the same symptoms… A restart of ActiveMQ resolve the
issue for about 5 minutes… then my workers come to a crawl :-(…
On Fri, Sep 19, 2014 at 3:28 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Use JConsole to look at which consumers are connected (make sure at least
> one is!), how many messages they're currently working on
> (MessageCountAwaitingAcknowlegement), and whether the numbers appear to be
> increasing. For each queue, confirm whether the EnqueueCount is
> increasing; if it's not, the broker isn't accepting messages due to
> Producer Flow Control, or the producer isn't attempting to produce any for
> other reasons.
>
> On Fri, Sep 19, 2014 at 3:55 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > We’ve been having this problem for a while now but it’s becoming more
> > apparent as we deploy in production.
> >
> > Essentially all messages in ActiveMQ get stuck until I restart it. Today
> > it took three restarts… then it just started working.
> >
> > Here’s my config:
> >
> > http://pastebin.com/F5L99biA
> >
> > We’re using LevelDB… the box seems to be idle while this is happening.
> >
> > There isn’t much data either. < 1GB… should all fit into memory.
> >
> > It’s entirely possible it’s the client … but I’m using the official
> > activemq client.
> >
> > I think I might try browsing the queue from the command line next time
> this
> > happens.
> >
> > Any other pointers to help track this down?
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: All messages getting stuck in 5.10 until restart…
Posted by Tim Bain <tb...@alumni.duke.edu>.
Use JConsole to look at which consumers are connected (make sure at least
one is!), how many messages they're currently working on
(MessageCountAwaitingAcknowlegement), and whether the numbers appear to be
increasing. For each queue, confirm whether the EnqueueCount is
increasing; if it's not, the broker isn't accepting messages due to
Producer Flow Control, or the producer isn't attempting to produce any for
other reasons.
On Fri, Sep 19, 2014 at 3:55 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> We’ve been having this problem for a while now but it’s becoming more
> apparent as we deploy in production.
>
> Essentially all messages in ActiveMQ get stuck until I restart it. Today
> it took three restarts… then it just started working.
>
> Here’s my config:
>
> http://pastebin.com/F5L99biA
>
> We’re using LevelDB… the box seems to be idle while this is happening.
>
> There isn’t much data either. < 1GB… should all fit into memory.
>
> It’s entirely possible it’s the client … but I’m using the official
> activemq client.
>
> I think I might try browsing the queue from the command line next time this
> happens.
>
> Any other pointers to help track this down?
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>