You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by aarontc <na...@aarontc.com> on 2016/04/14 22:39:17 UTC
Consumers inexplicably blocked while producers run fine. Help?
I'm looking for some pointers in diagnosing an issue we're seeing. I'll try
to describe:
ActiveMQ version: 5.12.1
OS: Ubuntu Linux 12.04
STOMP clients using Ruby 2.0.0 stomp gem v 1.1.10
We have two queues on a host with 8GiB of RAM and 20GiB of disk space. Under
steady-state conditions...
Queue A receives 10 to 500,000 byte messages at about 10 per second. There
are 10 consumers on this queue, and messages are essentially consumed as
fast as they can be produced.
Queue B receives 850,000 to 4,000,000 byte messages about 5 per minute.
There is 1 consumer on this queue, and messages are essentially consumed
within a few seconds of production.
Now, when a particular issue arises elsewhere in our system, we need to
backfill Queue B with a few thousand messages. We load those in much, much
faster than the Queue B consumer can process them. Everything is fine until
1500 or so messages are enqueued. At this point, the consumers on Queue A
STOP receiving messages. Queue A producers are still delivering 10 per
second, and the Queue B consumer is consuming a message every few seconds.
How do we troubleshoot this? I've tried loading Queue B more slowly - the
problem still occurs when ActiveMQ hits some memory threshold. The only way
to resolve the problem, apparently, is to delete Queue B. (An attempt to
"Purge" Queue B from the web interface will result in an OutOfMemory error
being returned.)
Thanks!
-Aaron
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumers inexplicably blocked while producers run fine. Help?
Posted by aarontc <na...@aarontc.com>.
Thanks for the link, Tim - that article appears to exactly cover the problem
we were seeing. I applied policies like
<policyEntry queue="QueueB" useCache="false" cursorMemoryHighWaterMark="10"
/>
to the queues which receive lots of large messages. The combination of
settings for the policy seems to prevent overrunning the heap cache, and
prevents starvation of consumers on other queues.
-Aaron
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793p4710862.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumers inexplicably blocked while producers run fine. Help?
Posted by Timothy Bish <ta...@gmail.com>.
On 04/14/2016 05:33 PM, aarontc wrote:
> Hi Tim,
>
> Yes - the consumers on Queue A will eventually start receiving messages once
> Queue B has had a bunch of messages removed.
>
> FWIW, I just verified this behavior is still present in 5.13.2.
>
> Thanks,
> -Aaron
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793p4710796.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Some interesting reading on situations such as this:
https://t.co/WOMyMIA67f
May or may not be relevant depending on your specific configuration.
Generally a good idea to add your broker configuration to help folks see
what your setup might entail.
--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: Consumers inexplicably blocked while producers run fine. Help?
Posted by aarontc <na...@aarontc.com>.
Hi Tim,
Yes - the consumers on Queue A will eventually start receiving messages once
Queue B has had a bunch of messages removed.
FWIW, I just verified this behavior is still present in 5.13.2.
Thanks,
-Aaron
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793p4710796.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumers inexplicably blocked while producers run fine. Help?
Posted by Tim Bain <tb...@alumni.duke.edu>.
A few thousand 1MB to 4MB messages will run you into the heap limit in a
hurry, even with your 8GB of heap. (I assume that's an 8GB JVM on a host
with more than 8GB RAM, not a smaller JVM on an 8GB host.)
Do you really need to dump gigs of messages on the broker when this
happens? (A few thousand is hours and hours of content; that's really the
best way to handle your issue?) If you do, Tim's suggestion of slowing
down your producer could help, or you can enable Producer Flow Control (you
should probably do this anyway, given this scenario).
Alternatively, make these persistent messages and configure a KahaDB
instance to hold them, which won't mind when you throw lots of data at it
all at once. Or stay non-persistent and configure to overflow to temp.
Tim
On Apr 14, 2016 3:33 PM, "Timothy Bish" <ta...@gmail.com> wrote:
On 04/14/2016 04:39 PM, aarontc wrote:
> I'm looking for some pointers in diagnosing an issue we're seeing. I'll try
> to describe:
>
> ActiveMQ version: 5.12.1
> OS: Ubuntu Linux 12.04
> STOMP clients using Ruby 2.0.0 stomp gem v 1.1.10
>
> We have two queues on a host with 8GiB of RAM and 20GiB of disk space.
> Under
> steady-state conditions...
>
> Queue A receives 10 to 500,000 byte messages at about 10 per second. There
> are 10 consumers on this queue, and messages are essentially consumed as
> fast as they can be produced.
>
> Queue B receives 850,000 to 4,000,000 byte messages about 5 per minute.
> There is 1 consumer on this queue, and messages are essentially consumed
> within a few seconds of production.
>
> Now, when a particular issue arises elsewhere in our system, we need to
> backfill Queue B with a few thousand messages. We load those in much, much
> faster than the Queue B consumer can process them. Everything is fine until
> 1500 or so messages are enqueued. At this point, the consumers on Queue A
> STOP receiving messages. Queue A producers are still delivering 10 per
> second, and the Queue B consumer is consuming a message every few seconds.
>
> How do we troubleshoot this? I've tried loading Queue B more slowly - the
> problem still occurs when ActiveMQ hits some memory threshold. The only way
> to resolve the problem, apparently, is to delete Queue B. (An attempt to
> "Purge" Queue B from the web interface will result in an OutOfMemory error
> being returned.)
>
> Thanks!
> -Aaron
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> If you stop producing to Queue B and let the consumer drain it completely
do the consumers on Queue A start to get messages again?
--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/
Re: Consumers inexplicably blocked while producers run fine. Help?
Posted by Timothy Bish <ta...@gmail.com>.
On 04/14/2016 04:39 PM, aarontc wrote:
> I'm looking for some pointers in diagnosing an issue we're seeing. I'll try
> to describe:
>
> ActiveMQ version: 5.12.1
> OS: Ubuntu Linux 12.04
> STOMP clients using Ruby 2.0.0 stomp gem v 1.1.10
>
> We have two queues on a host with 8GiB of RAM and 20GiB of disk space. Under
> steady-state conditions...
>
> Queue A receives 10 to 500,000 byte messages at about 10 per second. There
> are 10 consumers on this queue, and messages are essentially consumed as
> fast as they can be produced.
>
> Queue B receives 850,000 to 4,000,000 byte messages about 5 per minute.
> There is 1 consumer on this queue, and messages are essentially consumed
> within a few seconds of production.
>
> Now, when a particular issue arises elsewhere in our system, we need to
> backfill Queue B with a few thousand messages. We load those in much, much
> faster than the Queue B consumer can process them. Everything is fine until
> 1500 or so messages are enqueued. At this point, the consumers on Queue A
> STOP receiving messages. Queue A producers are still delivering 10 per
> second, and the Queue B consumer is consuming a message every few seconds.
>
> How do we troubleshoot this? I've tried loading Queue B more slowly - the
> problem still occurs when ActiveMQ hits some memory threshold. The only way
> to resolve the problem, apparently, is to delete Queue B. (An attempt to
> "Purge" Queue B from the web interface will result in an OutOfMemory error
> being returned.)
>
> Thanks!
> -Aaron
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumers-inexplicably-blocked-while-producers-run-fine-Help-tp4710793.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
If you stop producing to Queue B and let the consumer drain it
completely do the consumers on Queue A start to get messages again?
--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/