You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Takawale, Pankaj" <pa...@dowjones.com> on 2016/06/24 19:34:37 UTC
slow consumers
ActiveMQ version 5.11.3
No producer flow control.
<policyEntry queue=">" timeBeforeDispatchStarts="5000"
producerFlowControl="false" maxPageSize="1000" useCache="true"
expireMessagesPeriod="0" optimizedDispatch="true">
<dispatchPolicy>
<roundRobinDispatchPolicy />
</dispatchPolicy>
<messageGroupMapFactory>
<simpleMessageGroupMapFactory/>
</messageGroupMapFactory>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="-1"/>
</pendingMessageLimitStrategy>
</policyEntry>
I've noticed the following issue:
- Consumers are consuming at a consistent rate.
- Producer starts sending burst of messages
- Queue starts growing
- Once queue depth is around 400K+, ActiveMQ stops delivering messages to
the consumers.
- Eventually, Memory usage reaches to 100%, and it starts impacting other
queues, and producers/consumers.
To come out of this situation, I enable consumers to drop certain kind of
messages (to trigger faster consumption).
After this change, ActiveMQ starts delivering messages, and queue gets
drained.
Looks like to me, ActiveMQ detects the slow consumers for a growing queue,
and stops delivering messages.
Is it possible to configure ActiveMQ to deliver to all the consumers at
an equal rate? i.e. do not smartly balance based on the consumption rate?
Trying to avoid the system freeze.
Re: slow consumers
Posted by "Takawale, Pankaj" <pa...@dowjones.com>.
Thanks Tim for youe feedback!
I'm using JDBC persistence adapter, storing the messages in PostgreSQL.
I'm using GroupID for ordering the messages withing the group. So, adding
more consumers does not help if message burst is from couple of groups.
I'm using memoryUsage setting to limit the memory used by the broker.
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
I agree with you that it's good to deliver messages to consumers as per
their processing rate.
One thing I'm going to try is to have small prefetch limits. What pending
message strategy would you recommend to prevent the freeze situation?
Attached resource usage graph. Note that during 100% memory usage,
transmitted network bandwidth went up.
[image: Inline image 1]
On Sat, Jun 25, 2016 at 9:38 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> I've never seen any code that responds to slow consumers like that. There
> are a few plugins that allow you to abort (i.e. disconnect) slow consumers,
> but you have to enable them explicitly and you'd see logging about how the
> broker was aborting them.
>
> Are you using a persistence store, or just storing all messages in the
> memory store?
>
> What does the broker process's resource utilization look like when the
> problem occurs? CPU, network I/O, disk I/O, garbage collections, etc.?
>
> And why aren't you using either PFC or one of the pending message
> strategies? Preventing the broker from running out of memory or other
> resources is their primary purpose.
>
> Also, I was confused by the question about delivering messages to consumers
> at the same rate. For one thing, no, it's not possible and you wouldn't
> want it if it was (the slowest consumer would get overwhelmed and/or the
> fastest consumer would sit idle, depending on the rate). But more
> fundamentally, nothing you wrote seemed to indicate that there was a
> problem with the rate at which ActiveMQ was delivering messages to the
> different consumers. If anything, it sounds like you need to speed up your
> consumers' logic or add more consumers so that the backlog never gets
> unmanageable even during the worst burst periods.
>
> Tim
> ActiveMQ version 5.11.3
> No producer flow control.
>
> <policyEntry queue=">" timeBeforeDispatchStarts="5000"
> producerFlowControl="false" maxPageSize="1000" useCache="true"
> expireMessagesPeriod="0" optimizedDispatch="true">
> <dispatchPolicy>
> <roundRobinDispatchPolicy />
> </dispatchPolicy>
> <messageGroupMapFactory>
> <simpleMessageGroupMapFactory/>
> </messageGroupMapFactory>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="-1"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
>
>
> I've noticed the following issue:
>
> - Consumers are consuming at a consistent rate.
> - Producer starts sending burst of messages
> - Queue starts growing
> - Once queue depth is around 400K+, ActiveMQ stops delivering messages to
> the consumers.
> - Eventually, Memory usage reaches to 100%, and it starts impacting other
> queues, and producers/consumers.
>
> To come out of this situation, I enable consumers to drop certain kind of
> messages (to trigger faster consumption).
> After this change, ActiveMQ starts delivering messages, and queue gets
> drained.
>
> Looks like to me, ActiveMQ detects the slow consumers for a growing queue,
> and stops delivering messages.
> Is it possible to configure ActiveMQ to deliver to all the consumers at
> an equal rate? i.e. do not smartly balance based on the consumption rate?
> Trying to avoid the system freeze.
>
Re: slow consumers
Posted by Tim Bain <tb...@alumni.duke.edu>.
I've never seen any code that responds to slow consumers like that. There
are a few plugins that allow you to abort (i.e. disconnect) slow consumers,
but you have to enable them explicitly and you'd see logging about how the
broker was aborting them.
Are you using a persistence store, or just storing all messages in the
memory store?
What does the broker process's resource utilization look like when the
problem occurs? CPU, network I/O, disk I/O, garbage collections, etc.?
And why aren't you using either PFC or one of the pending message
strategies? Preventing the broker from running out of memory or other
resources is their primary purpose.
Also, I was confused by the question about delivering messages to consumers
at the same rate. For one thing, no, it's not possible and you wouldn't
want it if it was (the slowest consumer would get overwhelmed and/or the
fastest consumer would sit idle, depending on the rate). But more
fundamentally, nothing you wrote seemed to indicate that there was a
problem with the rate at which ActiveMQ was delivering messages to the
different consumers. If anything, it sounds like you need to speed up your
consumers' logic or add more consumers so that the backlog never gets
unmanageable even during the worst burst periods.
Tim
ActiveMQ version 5.11.3
No producer flow control.
<policyEntry queue=">" timeBeforeDispatchStarts="5000"
producerFlowControl="false" maxPageSize="1000" useCache="true"
expireMessagesPeriod="0" optimizedDispatch="true">
<dispatchPolicy>
<roundRobinDispatchPolicy />
</dispatchPolicy>
<messageGroupMapFactory>
<simpleMessageGroupMapFactory/>
</messageGroupMapFactory>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="-1"/>
</pendingMessageLimitStrategy>
</policyEntry>
I've noticed the following issue:
- Consumers are consuming at a consistent rate.
- Producer starts sending burst of messages
- Queue starts growing
- Once queue depth is around 400K+, ActiveMQ stops delivering messages to
the consumers.
- Eventually, Memory usage reaches to 100%, and it starts impacting other
queues, and producers/consumers.
To come out of this situation, I enable consumers to drop certain kind of
messages (to trigger faster consumption).
After this change, ActiveMQ starts delivering messages, and queue gets
drained.
Looks like to me, ActiveMQ detects the slow consumers for a growing queue,
and stops delivering messages.
Is it possible to configure ActiveMQ to deliver to all the consumers at
an equal rate? i.e. do not smartly balance based on the consumption rate?
Trying to avoid the system freeze.