Performance falls dramatically when GCing large numbers of inactive queues?

[Kevin Burton <bu...@spinn3r.com>]

We use a ton of queues… like something like 400-500k.

The problem is that we also GC them when we’re inactive for long amounts of
time. I’m a bit worried that the’ll end up wasting tons of resources.

the problem is that AMQ seems to go through “GC" storms (not Java GC but
inactive queue GC) where it literally does NO work during this period.

2015-01-29 16:27:06,239 | INFO  | foo-queue Inactive for longer than 900000
ms - removing ... | org.apache.activemq.broker.region.Queue | ActiveMQ
Broker[util0044.example.com] Scheduler

During this period, AMQ is completely locked up and performs no work.

I suspect there is some sort of mutex lock here which preventing AMQ from
doing work while performing queue GC and that most people don’t see this
(ever) because typical GC is about 1-2 queues.

Would this be a correct assumption?

Kevin

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Re: Performance falls dramatically when GCing large numbers of inactive queues?

[Kevin Burton <bu...@spinn3r.com>]

Thing definitely seem to crawl dramatically when this kicks in.  It
basically spends 5-10 minutes doing nothing but printing messages saying
it’s removing in inactive queue.

So either a mutex or a controlling thread is waiting for the inactive
queues to be removed.

It turns out it’s < 50k.. still high but not insane.  The plan is to scale
this out.  So each box would handle about 50k queues.

I have a work around.  The reason it has so many that need to be GCd is
that one of our daemons keeps a consumer on them and never releases it even
if the queue is empty.

So if I just release it the queues will be inactive and then get GCd more
often.

On Thu, Jan 29, 2015 at 4:10 PM, artnaseef <ar...@artnaseef.com> wrote:

> I'm not sure about a mutex in this case, although there are many inside the
> code, but I would say that 400k-500k queues is an insane number.
>
> Is it possible to re-architect the solution to bring that number down?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Performance-falls-dramatically-when-GCing-large-numbers-of-inactive-queues-tp4690685p4690691.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Re: Performance falls dramatically when GCing large numbers of inactive queues?

[artnaseef <ar...@artnaseef.com>]

I'm not sure about a mutex in this case, although there are many inside the
code, but I would say that 400k-500k queues is an insane number.

Is it possible to re-architect the solution to bring that number down?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Performance-falls-dramatically-when-GCing-large-numbers-of-inactive-queues-tp4690685p4690691.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.