You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by E White <ew...@synapticfire.net> on 2010/04/27 00:22:52 UTC
Topic Memory Increases until broker dies
We have a java swing client that gets its data updates by subscribing to an
ActiveMQ topic. All clients (about a hundred at any given time) subscribe
to the same topic. Subscribers vary and are not durable, but the messages
are persistent.
I have seen the "inflight count" slowly, but steadily increase and then the
"memory percent used" value spike, leaving the broker deadlocked and no
longer processing messages, but not failing. The whole scenario takes about
a week. And this makes our client application look like it is hung after it
occurs.
How do I keep messages from getting orphaned when a user closes their
client? I think that is where most of my "inflight" number is coming from.
Users close the client right as it has received a message but before it can
send the ack.
How do I make "inflight" messages expire? I've played around with setting
"timetolive" in my test environment, but it does not seem to affect messages
that are stuck "inflight".
We are using ActiveMQ 5.3.0.
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28370484.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic Memory Increases until broker dies
Posted by Bruce Snyder <br...@gmail.com>.
On Mon, May 24, 2010 at 1:50 PM, E White <ew...@synapticfire.net> wrote:
>
> I could never reproduce this effectively in the office, but it happened again
> in production.
> Here is the thread dump from when this occurred.
> http://old.nabble.com/file/p28661085/sfire-amq-td.log sfire-amq-td.log
I don't see anything out of the ordinary in the thread dump. But with
thread dumps you need 3-5 of them that are spaced apart a bit so that
you can see some changes over a short period of time.
The inflight count is the number of messages sent to a given
consumer's session. The number of messages sent to a consumer is
directly affected by the prefetch size described here:
http://activemq.apache.org/what-is-the-prefetch-limit-for.html
How many consumers do you have on the topic? Are any of these
consumers possibly processing messages slowly? Do these consumers use
a prefetch size to limit the number of messages sent to them in a
given dispatch? This will cause the broker not to send so many
messages in a given dispatch to the consumer and can help slow
consumers.
To answer your question related to message expiration, setting the
timeToLive on a message will cause that message to expire after that
amount of time.
Another strategy is to use a message limit strategy or an eviction strategy:
http://activemq.apache.org/slow-consumer-handling.html
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: Topic Memory Increases until broker dies
Posted by E White <ew...@synapticfire.net>.
I could never reproduce this effectively in the office, but it happened again
in production.
Here is the thread dump from when this occurred.
http://old.nabble.com/file/p28661085/sfire-amq-td.log sfire-amq-td.log
bsnyder wrote:
>
> On Tue, May 4, 2010 at 2:55 PM, E White <ew...@synapticfire.net> wrote:
>>
>> I'll try.
>> I have an heap dump from the last time it happened in production, but
>> cannot
>> attach it here.
>> I have replicated the inflight count inconsistency in test, but haven't
>> the
>> load to reproduce the memory problem yet. I'll lower the memory values
>> and
>> give it another try.
>
> Decrease the memory for the JVM in the bin/activemq startup script to
> cause the error to happen faster.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
>
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28661085.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic Memory Increases until broker dies
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, May 4, 2010 at 2:55 PM, E White <ew...@synapticfire.net> wrote:
>
> I'll try.
> I have an heap dump from the last time it happened in production, but cannot
> attach it here.
> I have replicated the inflight count inconsistency in test, but haven't the
> load to reproduce the memory problem yet. I'll lower the memory values and
> give it another try.
Decrease the memory for the JVM in the bin/activemq startup script to
cause the error to happen faster.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: Topic Memory Increases until broker dies
Posted by E White <ew...@synapticfire.net>.
I'll try.
I have an heap dump from the last time it happened in production, but cannot
attach it here.
I have replicated the inflight count inconsistency in test, but haven't the
load to reproduce the memory problem yet. I'll lower the memory values and
give it another try.
bsnyder wrote:
>
> On Tue, May 4, 2010 at 12:54 PM, E White <ew...@synapticfire.net> wrote:
>>
>> Attached is my activemq.xml file.
>> There are other brokers in our configuration, but this is the one that
>> the
>> clients subscribe to.
>>
>> http://old.nabble.com/file/p28451681/activemq.xml activemq.xml
>
> Grab a thread dump when the broker is deadlocked so that we can see
> what is happening.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>
>
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28453000.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic Memory Increases until broker dies
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, May 4, 2010 at 12:54 PM, E White <ew...@synapticfire.net> wrote:
>
> Attached is my activemq.xml file.
> There are other brokers in our configuration, but this is the one that the
> clients subscribe to.
>
> http://old.nabble.com/file/p28451681/activemq.xml activemq.xml
Grab a thread dump when the broker is deadlocked so that we can see
what is happening.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: Topic Memory Increases until broker dies
Posted by E White <ew...@synapticfire.net>.
Attached is my activemq.xml file.
There are other brokers in our configuration, but this is the one that the
clients subscribe to.
http://old.nabble.com/file/p28451681/activemq.xml activemq.xml
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28451681.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic Memory Increases until broker dies
Posted by Bruce Snyder <br...@gmail.com>.
On Thu, May 27, 2010 at 1:19 PM, odysseyfx <in...@bradwillard.com> wrote:
>
> I've seen this problem fairly frequently. This is caused by a message
> producer on a topic creating messages faster than a consumer can consume
> them. Those messages then get backed up on the broker internal dispatch
> queue for that session until all resources are consumed when the broker
> deadlocks.
>
> There are a few solutions to fix this problem that I have found: configuring
> and enabling flow control (which will throttle the message producer), enable
> a message eviction policy (where the broker will start throwing out the
> oldest messages of the lowest priority that aren't consumed fast enough), or
> finally you can actually use a durable subscriber if the message producer
> rate will slow down enough at some point in time for the consumers to catch
> up, otherwise you'll wind up with the same problem backing up on a queue
> instead.
>
> Another solution is Virtual Topics, and activemq has a whole page on exactly
> this scenario
>
> http://activemq.apache.org/slow-consumer-handling.html
> and
> http://activemq.apache.org/virtual-destinations.html
>
> I want to add that there is no tuning guide out there to get flow control to
> work properly, and it's a pain in the ass to setup. You'll need someone who
> knows the code to help, there is no document on this but there should be.
Are you referring to TCP flow control?
Here's the producer flow control document:
http://activemq.apache.org/producer-flow-control.html
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder
Re: Topic Memory Increases until broker dies
Posted by odysseyfx <in...@bradwillard.com>.
I've seen this problem fairly frequently. This is caused by a message
producer on a topic creating messages faster than a consumer can consume
them. Those messages then get backed up on the broker internal dispatch
queue for that session until all resources are consumed when the broker
deadlocks.
There are a few solutions to fix this problem that I have found: configuring
and enabling flow control (which will throttle the message producer), enable
a message eviction policy (where the broker will start throwing out the
oldest messages of the lowest priority that aren't consumed fast enough), or
finally you can actually use a durable subscriber if the message producer
rate will slow down enough at some point in time for the consumers to catch
up, otherwise you'll wind up with the same problem backing up on a queue
instead.
Another solution is Virtual Topics, and activemq has a whole page on exactly
this scenario
http://activemq.apache.org/slow-consumer-handling.html
and
http://activemq.apache.org/virtual-destinations.html
I want to add that there is no tuning guide out there to get flow control to
work properly, and it's a pain in the ass to setup. You'll need someone who
knows the code to help, there is no document on this but there should be.
Good luck,
Brad
E White wrote:
>
> We have a java swing client that gets its data updates by subscribing to
> an ActiveMQ topic. All clients (about a hundred at any given time)
> subscribe to the same topic. Subscribers vary and are not durable, but
> the messages are persistent.
>
> I have seen the "inflight count" slowly, but steadily increase and then
> the "memory percent used" value spike, leaving the broker deadlocked and
> no longer processing messages, but not failing. The whole scenario takes
> about a week. And this makes our client application look like it is hung
> after it occurs.
>
> How do I keep messages from getting orphaned when a user closes their
> client? I think that is where most of my "inflight" number is coming
> from. Users close the client right as it has received a message but
> before it can send the ack.
> How do I make "inflight" messages expire? I've played around with setting
> "timetolive" in my test environment, but it does not seem to affect
> messages that are stuck "inflight".
>
> We are using ActiveMQ 5.3.0.
>
>
>
>
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28698719.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic Memory Increases until broker dies
Posted by Gary Tully <ga...@gmail.com>.
you should post your avtivemq.xml and also try 5.3.1, it resolves one issue
with the file pending message currsor and message references that could be
the cause of your leak.
On 4 May 2010 15:43, E White <ew...@synapticfire.net> wrote:
>
> This must be a very difficult problem.
> Nobody here has any ideas they would like to share.
> The people at SpringSource, whom we pay for support, can't figure it out.
>
> Are we implementing this incorrectly?
> Is it folly to use a topic to send updates to a swing client?
>
> One of our developers locks up his topic easily by pausing his client while
> running in his IDE, turning his client into a slow consumer. It blocks the
> whole broker. No other client can receive updates.
>
>
> ActiveMQ is such a pleasure to get working, and such pain in the ass to
> keep
> working.
>
>
>
> E White wrote:
> >
> > We have a java swing client that gets its data updates by subscribing to
> > an ActiveMQ topic. All clients (about a hundred at any given time)
> > subscribe to the same topic. Subscribers vary and are not durable, but
> > the messages are persistent.
> >
> > I have seen the "inflight count" slowly, but steadily increase and then
> > the "memory percent used" value spike, leaving the broker deadlocked and
> > no longer processing messages, but not failing. The whole scenario takes
> > about a week. And this makes our client application look like it is hung
> > after it occurs.
> >
> > How do I keep messages from getting orphaned when a user closes their
> > client? I think that is where most of my "inflight" number is coming
> > from. Users close the client right as it has received a message but
> > before it can send the ack.
> > How do I make "inflight" messages expire? I've played around with
> setting
> > "timetolive" in my test environment, but it does not seem to affect
> > messages that are stuck "inflight".
> >
> > We are using ActiveMQ 5.3.0.
> >
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28447750.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Topic Memory Increases until broker dies
Posted by E White <ew...@synapticfire.net>.
This must be a very difficult problem.
Nobody here has any ideas they would like to share.
The people at SpringSource, whom we pay for support, can't figure it out.
Are we implementing this incorrectly?
Is it folly to use a topic to send updates to a swing client?
One of our developers locks up his topic easily by pausing his client while
running in his IDE, turning his client into a slow consumer. It blocks the
whole broker. No other client can receive updates.
ActiveMQ is such a pleasure to get working, and such pain in the ass to keep
working.
E White wrote:
>
> We have a java swing client that gets its data updates by subscribing to
> an ActiveMQ topic. All clients (about a hundred at any given time)
> subscribe to the same topic. Subscribers vary and are not durable, but
> the messages are persistent.
>
> I have seen the "inflight count" slowly, but steadily increase and then
> the "memory percent used" value spike, leaving the broker deadlocked and
> no longer processing messages, but not failing. The whole scenario takes
> about a week. And this makes our client application look like it is hung
> after it occurs.
>
> How do I keep messages from getting orphaned when a user closes their
> client? I think that is where most of my "inflight" number is coming
> from. Users close the client right as it has received a message but
> before it can send the ack.
> How do I make "inflight" messages expire? I've played around with setting
> "timetolive" in my test environment, but it does not seem to affect
> messages that are stuck "inflight".
>
> We are using ActiveMQ 5.3.0.
>
>
>
>
--
View this message in context: http://old.nabble.com/Topic-Memory-Increases-until-broker-dies-tp28370484p28447750.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.