You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by Daniel Israel <di...@liveops.com> on 2015/08/17 21:45:07 UTC

User Memory Issues

Hello All,

I am tracking down a memory issue in AMQ.  I am time to time seeing exceptions like this:


Async error occurred: javax.jms.ResourceAllocationException: Usage Manager memory limit reached


I can't tell if this is because I am exceeding the configured amount of memory in SystemUsage, or if I am exceeding the amount of memory configured per topic.

I am using only Topics right now, and I had the memory limit set to 10mb.  The error doesn't direct me in any direction for this.  I am using an old version of AMQ (First step was to request upgrade to latest version.  It's in the works, but it might be a week or two before it's completed in production) and I see changes in the source that give more details when throwing this exception.  Is there some historical record or log of Topics?  What I'd really like is to be able to see how often each Topic gets and distributes a message and how big that message was.  The dashboard and Jconsole give me some information, but because Topics are delivered then released, I don't have any information beyond how many were enqueued and delivered.

Is there any such animal available that would help me with this?  Or suggestions how to approach?  Any help is appreciated.  Thanks.

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.

Hi Tim,

Yes, you're correct, the version I have in production is older than that.  I have confirmed that it is pre-5.3.0 (It's actually a FUSE release) and the changelog in the ActiveMQ Subversion log shows the log details coming a few months later.




On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
>like the one in this page (
>http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which give
>lots of information about what limit is being hit.  I guess that detailed
>info must have been added in a version after the one you're using.  What
>version is that, anyway?
>
>Can you post the full log message?
>
>Have you explored the JMX tree to see if there is any info to tell you how
>full your topics are?  Obviously what information is available will vary
>based on what version you're running, so that might not help, but you
>should definitely check.
>On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
>
>>
>> Hi Tim, thanks for response.
>>
>> Flow control is enabled, and it's configured to fail if out of memory.  As
>> noted below, the log lines in this version don't tell us which limit we're
>> exceeding, so we're running half blind :(.  Knowing average topic size
>> would be helpful, but having individual topic size would be good.  Right
>> now, looking at the producer side to see if there is some way to track.
>>
>> Raised the topic limit to 20mb and still had an issue.  It's very hit and
>> miss.  I can run for a week without issue, then when I get hit hard, it
>> falls over.
>>
>>
>>
>>
>> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >Later versions give a few addition stats (such as average message size)
>> via
>> >JMX, but that won't help you till that upgrade in production is complete.
>> >
>> >Do you have producer flow control enabled?  The error you're getting
>> >doesn't match what I remember it being the last time I hit it, so I'm
>> >assuming you don't.  PFC gives log lines that at least tell you exactly
>> >which limit you ran into, plus it'll avoid losing any messages (but it'll
>> >"hang" producers till messages are consumed), so you could enable it to
>> >better understand what's going on.
>> >
>> >Tim
>> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
>> >
>> >>
>> >> Hello All,
>> >>
>> >> I am tracking down a memory issue in AMQ.  I am time to time seeing
>> >> exceptions like this:
>> >>
>> >>
>> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
>> Manager
>> >> memory limit reached
>> >>
>> >>
>> >> I can't tell if this is because I am exceeding the configured amount of
>> >> memory in SystemUsage, or if I am exceeding the amount of memory
>> configured
>> >> per topic.
>> >>
>> >> I am using only Topics right now, and I had the memory limit set to
>> 10mb.
>> >> The error doesn't direct me in any direction for this.  I am using an
>> old
>> >> version of AMQ (First step was to request upgrade to latest version.
>> It's
>> >> in the works, but it might be a week or two before it's completed in
>> >> production) and I see changes in the source that give more details when
>> >> throwing this exception.  Is there some historical record or log of
>> >> Topics?  What I'd really like is to be able to see how often each Topic
>> >> gets and distributes a message and how big that message was.  The
>> dashboard
>> >> and Jconsole give me some information, but because Topics are delivered
>> >> then released, I don't have any information beyond how many were
>> enqueued
>> >> and delivered.
>> >>
>> >> Is there any such animal available that would help me with this?  Or
>> >> suggestions how to approach?  Any help is appreciated.  Thanks.
>> >>
>> >>

Re: User Memory Issues

Posted by jamesTheCruncher <ja...@mcruncher.com>.

Hi Daniel,

I guess you are getting the exception in your producer and you are sending
messages asynchronously. Is this right?
It would be helpful if you share your activemq.xml configuration here.



--
View this message in context: http://activemq.2283324.n4.nabble.com/User-Memory-Issues-tp4701161p4701336.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

Fair enough, that was my first thought, but I also know that sometimes you
have to figure out a band-aid till the upgrade comes through.  But if
you're good just doing the upgrade, that sounds great to me!
On Aug 24, 2015 10:50 PM, "Daniel Israel" <di...@liveops.com> wrote:

> Hi Tim,
>
> I appreciate your efforts, but it really doesn't make sense for me (and
> you) to try to debug the issue on an environment that's 7+ years old.  It's
> really on me to get the production system updated, I'm working on
> development environment now, which is where my questions come from.  Don't
> spend a lot of time trying to figure out what happened in version 5.2, it's
> totally appropriate that I should be on the hook to upgrade to a modern
> version.
>
> .... And thanks for the confirmation.  :)
>
>
>
>
>
> On 8/24/15, 9:18 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >Yes, as long as that's on the subscription, not the topic itself.  Caveat:
> >I haven't actually looked at JMX in 5.3.2, so I'm extrapolating from what
> >I've seen in later versions, but I think that should be right.
> >On Aug 24, 2015 1:29 PM, "Daniel Israel" <di...@liveops.com> wrote:
> >
> >>
> >>
> >>
> >>
> >>
> >> On 8/21/15, 5:28 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >You can tell if a consumer is slower than the rest via JMX.  Look at
> each
> >> >subscription for the topic and see if any of them has lots of pending
> >> >messages while the rest do not.
> >>
> >> Is that the PendingQueueSize on the JMX console?
> >>
> >> Thanks.
> >>
> >>
> >> >If you discover that to be the case,
> >> >configure a slow consumer abortion strategy to kick them off and throw
> >> away
> >> >their pending messages.  (Note that this only works for topics, and
> only
> >> >for non-durable subscriptions, but I think both of those apply to your
> >> >scenario.)
> >> >On Aug 21, 2015 12:30 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >> >
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >>
> >> >> >The broker can't discard the message till all subscribers have
> consumed
> >> >> it,
> >> >>
> >> >> This is good information, thanks!  I am wondering if there is a way
> to
> >> >> detect this?
> >> >>
> >> >> >so a single slow consumer will result in increased memory/store
> usage
> >> and
> >> >> >eventually PFC.  You could set up a slow consumer abortion strategy
> if
> >> >> >you're worried about that possibility.
> >> >> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com>
> wrote:
> >> >> >
> >> >> >> Oooh, that's a good idea to add another subber.  I may be able to
> try
> >> >> that.
> >> >> >>
> >> >> >> Also, could something on the client side cause a backup?  Or will
> the
> >> >> >> broker just fall through if a client hangs?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >>
> >> >> >> >Since this is happening in production, I assume it's not
> acceptable
> >> to
> >> >> >> >attach a debugger and set a breakpoint on the line where the
> >> exception
> >> >> is
> >> >> >> >thrown.  If you could, that would let you step into the
> >> >> >> >MemoryUsage.isFull() call and see what's going on, but it'll hang
> >> the
> >> >> >> >broker when it happens and I assume that won't fly.
> >> >> >> >
> >> >> >> >You could probably figure out the size of the messages just by
> >> >> creating a
> >> >> >> >test app that just logs the size of each message it receives and
> >> >> having it
> >> >> >> >become another subscriber on the topic.  It won't tell you the
> rate
> >> at
> >> >> >> >which your consumers consume, but you'll know the input rates at
> >> least.
> >> >> >> >
> >> >> >> >If you're worried that your topics are filling up because your
> >> >> consumers
> >> >> >> >are falling behind, you could force them to disconnect (and throw
> >> away
> >> >> >> >their pending messages) if they get too far behind.
> >> >> >> >http://activemq.apache.org/slow-consumer-handling.html  Just
> make
> >> sure
> >> >> >> the
> >> >> >> >limit you set is smaller than the PFC per-destination limit, or
> >> you'll
> >> >> >> >never hit it.
> >> >> >> >
> >> >> >> >Tim
> >> >> >> >
> >> >> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <
> >> disrael@liveops.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >>
> >> >> >> >> Forgot to attach.  This is log message:
> >> >> >> >>
> >> >> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
> >> >> >> >>     - Async error occurred:
> javax.jms.ResourceAllocationException:
> >> >> Usage
> >> >> >> >> Manager memory limit reached
> >> >> >> >> javax.jms.ResourceAllocationException: Usage Manager memory
> limit
> >> >> >> reached
> >> >> >> >>     at
> >> org.apache.activemq.broker.region.Topic.send(Topic.java:293)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> >> >> >> >>     at
> >> >> >>
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> >> >> >> >>     at
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> >> >> >> >>     at java.lang.Thread.run(Thread.java:662)
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim
> Bain" <
> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >> >>
> >> >> >> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and
> >> 5.10.0)
> >> >> look
> >> >> >> >> >like the one in this page (
> >> >> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/
> ),
> >> >> which
> >> >> >> >> give
> >> >> >> >> >lots of information about what limit is being hit.  I guess
> that
> >> >> >> detailed
> >> >> >> >> >info must have been added in a version after the one you're
> >> using.
> >> >> >> What
> >> >> >> >> >version is that, anyway?
> >> >> >> >> >
> >> >> >> >> >Can you post the full log message?
> >> >> >> >> >
> >> >> >> >> >Have you explored the JMX tree to see if there is any info to
> >> tell
> >> >> you
> >> >> >> how
> >> >> >> >> >full your topics are?  Obviously what information is available
> >> will
> >> >> >> vary
> >> >> >> >> >based on what version you're running, so that might not help,
> but
> >> >> you
> >> >> >> >> >should definitely check.
> >> >> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <
> disrael@liveops.com>
> >> >> wrote:
> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >> Hi Tim, thanks for response.
> >> >> >> >> >>
> >> >> >> >> >> Flow control is enabled, and it's configured to fail if out
> of
> >> >> >> memory.
> >> >> >> >> As
> >> >> >> >> >> noted below, the log lines in this version don't tell us
> which
> >> >> limit
> >> >> >> >> we're
> >> >> >> >> >> exceeding, so we're running half blind :(.  Knowing average
> >> topic
> >> >> >> size
> >> >> >> >> >> would be helpful, but having individual topic size would be
> >> good.
> >> >> >> Right
> >> >> >> >> >> now, looking at the producer side to see if there is some
> way
> >> to
> >> >> >> track.
> >> >> >> >> >>
> >> >> >> >> >> Raised the topic limit to 20mb and still had an issue.  It's
> >> very
> >> >> hit
> >> >> >> >> and
> >> >> >> >> >> miss.  I can run for a week without issue, then when I get
> hit
> >> >> hard,
> >> >> >> it
> >> >> >> >> >> falls over.
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim
> >> Bain" <
> >> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu>
> wrote:
> >> >> >> >> >>
> >> >> >> >> >> >Later versions give a few addition stats (such as average
> >> message
> >> >> >> size)
> >> >> >> >> >> via
> >> >> >> >> >> >JMX, but that won't help you till that upgrade in
> production
> >> is
> >> >> >> >> complete.
> >> >> >> >> >> >
> >> >> >> >> >> >Do you have producer flow control enabled?  The error
> you're
> >> >> getting
> >> >> >> >> >> >doesn't match what I remember it being the last time I hit
> >> it, so
> >> >> >> I'm
> >> >> >> >> >> >assuming you don't.  PFC gives log lines that at least tell
> >> you
> >> >> >> exactly
> >> >> >> >> >> >which limit you ran into, plus it'll avoid losing any
> messages
> >> >> (but
> >> >> >> >> it'll
> >> >> >> >> >> >"hang" producers till messages are consumed), so you could
> >> enable
> >> >> >> it to
> >> >> >> >> >> >better understand what's going on.
> >> >> >> >> >> >
> >> >> >> >> >> >Tim
> >> >> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <
> disrael@liveops.com
> >> >
> >> >> >> wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >> Hello All,
> >> >> >> >> >> >>
> >> >> >> >> >> >> I am tracking down a memory issue in AMQ.  I am time to
> time
> >> >> >> seeing
> >> >> >> >> >> >> exceptions like this:
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> Async error occurred:
> javax.jms.ResourceAllocationException:
> >> >> Usage
> >> >> >> >> >> Manager
> >> >> >> >> >> >> memory limit reached
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> I can't tell if this is because I am exceeding the
> >> configured
> >> >> >> amount
> >> >> >> >> of
> >> >> >> >> >> >> memory in SystemUsage, or if I am exceeding the amount of
> >> >> memory
> >> >> >> >> >> configured
> >> >> >> >> >> >> per topic.
> >> >> >> >> >> >>
> >> >> >> >> >> >> I am using only Topics right now, and I had the memory
> limit
> >> >> set
> >> >> >> to
> >> >> >> >> >> 10mb.
> >> >> >> >> >> >> The error doesn't direct me in any direction for this.
> I am
> >> >> >> using an
> >> >> >> >> >> old
> >> >> >> >> >> >> version of AMQ (First step was to request upgrade to
> latest
> >> >> >> version.
> >> >> >> >> >> It's
> >> >> >> >> >> >> in the works, but it might be a week or two before it's
> >> >> completed
> >> >> >> in
> >> >> >> >> >> >> production) and I see changes in the source that give
> more
> >> >> details
> >> >> >> >> when
> >> >> >> >> >> >> throwing this exception.  Is there some historical
> record or
> >> >> log
> >> >> >> of
> >> >> >> >> >> >> Topics?  What I'd really like is to be able to see how
> often
> >> >> each
> >> >> >> >> Topic
> >> >> >> >> >> >> gets and distributes a message and how big that message
> was.
> >> >> The
> >> >> >> >> >> dashboard
> >> >> >> >> >> >> and Jconsole give me some information, but because Topics
> >> are
> >> >> >> >> delivered
> >> >> >> >> >> >> then released, I don't have any information beyond how
> many
> >> >> were
> >> >> >> >> >> enqueued
> >> >> >> >> >> >> and delivered.
> >> >> >> >> >> >>
> >> >> >> >> >> >> Is there any such animal available that would help me
> with
> >> >> this?
> >> >> >> Or
> >> >> >> >> >> >> suggestions how to approach?  Any help is appreciated.
> >> Thanks.
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.

Hi Tim,

I appreciate your efforts, but it really doesn't make sense for me (and you) to try to debug the issue on an environment that's 7+ years old.  It's really on me to get the production system updated, I'm working on development environment now, which is where my questions come from.  Don't spend a lot of time trying to figure out what happened in version 5.2, it's totally appropriate that I should be on the hook to upgrade to a modern version.

.... And thanks for the confirmation.  :)





On 8/24/15, 9:18 PM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>Yes, as long as that's on the subscription, not the topic itself.  Caveat:
>I haven't actually looked at JMX in 5.3.2, so I'm extrapolating from what
>I've seen in later versions, but I think that should be right.
>On Aug 24, 2015 1:29 PM, "Daniel Israel" <di...@liveops.com> wrote:
>
>>
>>
>>
>>
>>
>> On 8/21/15, 5:28 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >You can tell if a consumer is slower than the rest via JMX.  Look at each
>> >subscription for the topic and see if any of them has lots of pending
>> >messages while the rest do not.
>>
>> Is that the PendingQueueSize on the JMX console?
>>
>> Thanks.
>>
>>
>> >If you discover that to be the case,
>> >configure a slow consumer abortion strategy to kick them off and throw
>> away
>> >their pending messages.  (Note that this only works for topics, and only
>> >for non-durable subscriptions, but I think both of those apply to your
>> >scenario.)
>> >On Aug 21, 2015 12:30 AM, "Daniel Israel" <di...@liveops.com> wrote:
>> >
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >>
>> >> >The broker can't discard the message till all subscribers have consumed
>> >> it,
>> >>
>> >> This is good information, thanks!  I am wondering if there is a way to
>> >> detect this?
>> >>
>> >> >so a single slow consumer will result in increased memory/store usage
>> and
>> >> >eventually PFC.  You could set up a slow consumer abortion strategy if
>> >> >you're worried about that possibility.
>> >> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:
>> >> >
>> >> >> Oooh, that's a good idea to add another subber.  I may be able to try
>> >> that.
>> >> >>
>> >> >> Also, could something on the client side cause a backup?  Or will the
>> >> >> broker just fall through if a client hangs?
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >>
>> >> >> >Since this is happening in production, I assume it's not acceptable
>> to
>> >> >> >attach a debugger and set a breakpoint on the line where the
>> exception
>> >> is
>> >> >> >thrown.  If you could, that would let you step into the
>> >> >> >MemoryUsage.isFull() call and see what's going on, but it'll hang
>> the
>> >> >> >broker when it happens and I assume that won't fly.
>> >> >> >
>> >> >> >You could probably figure out the size of the messages just by
>> >> creating a
>> >> >> >test app that just logs the size of each message it receives and
>> >> having it
>> >> >> >become another subscriber on the topic.  It won't tell you the rate
>> at
>> >> >> >which your consumers consume, but you'll know the input rates at
>> least.
>> >> >> >
>> >> >> >If you're worried that your topics are filling up because your
>> >> consumers
>> >> >> >are falling behind, you could force them to disconnect (and throw
>> away
>> >> >> >their pending messages) if they get too far behind.
>> >> >> >http://activemq.apache.org/slow-consumer-handling.html  Just make
>> sure
>> >> >> the
>> >> >> >limit you set is smaller than the PFC per-destination limit, or
>> you'll
>> >> >> >never hit it.
>> >> >> >
>> >> >> >Tim
>> >> >> >
>> >> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <
>> disrael@liveops.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Forgot to attach.  This is log message:
>> >> >> >>
>> >> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
>> >> >> >>     - Async error occurred: javax.jms.ResourceAllocationException:
>> >> Usage
>> >> >> >> Manager memory limit reached
>> >> >> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
>> >> >> reached
>> >> >> >>     at
>> org.apache.activemq.broker.region.Topic.send(Topic.java:293)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>> >> >> >>     at
>> >> >> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>> >> >> >>     at
>> >> >> >>
>> >> >>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>> >> >> >>     at java.lang.Thread.run(Thread.java:662)
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >> >>
>> >> >> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and
>> 5.10.0)
>> >> look
>> >> >> >> >like the one in this page (
>> >> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/),
>> >> which
>> >> >> >> give
>> >> >> >> >lots of information about what limit is being hit.  I guess that
>> >> >> detailed
>> >> >> >> >info must have been added in a version after the one you're
>> using.
>> >> >> What
>> >> >> >> >version is that, anyway?
>> >> >> >> >
>> >> >> >> >Can you post the full log message?
>> >> >> >> >
>> >> >> >> >Have you explored the JMX tree to see if there is any info to
>> tell
>> >> you
>> >> >> how
>> >> >> >> >full your topics are?  Obviously what information is available
>> will
>> >> >> vary
>> >> >> >> >based on what version you're running, so that might not help, but
>> >> you
>> >> >> >> >should definitely check.
>> >> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com>
>> >> wrote:
>> >> >> >> >
>> >> >> >> >>
>> >> >> >> >> Hi Tim, thanks for response.
>> >> >> >> >>
>> >> >> >> >> Flow control is enabled, and it's configured to fail if out of
>> >> >> memory.
>> >> >> >> As
>> >> >> >> >> noted below, the log lines in this version don't tell us which
>> >> limit
>> >> >> >> we're
>> >> >> >> >> exceeding, so we're running half blind :(.  Knowing average
>> topic
>> >> >> size
>> >> >> >> >> would be helpful, but having individual topic size would be
>> good.
>> >> >> Right
>> >> >> >> >> now, looking at the producer side to see if there is some way
>> to
>> >> >> track.
>> >> >> >> >>
>> >> >> >> >> Raised the topic limit to 20mb and still had an issue.  It's
>> very
>> >> hit
>> >> >> >> and
>> >> >> >> >> miss.  I can run for a week without issue, then when I get hit
>> >> hard,
>> >> >> it
>> >> >> >> >> falls over.
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim
>> Bain" <
>> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >> >> >>
>> >> >> >> >> >Later versions give a few addition stats (such as average
>> message
>> >> >> size)
>> >> >> >> >> via
>> >> >> >> >> >JMX, but that won't help you till that upgrade in production
>> is
>> >> >> >> complete.
>> >> >> >> >> >
>> >> >> >> >> >Do you have producer flow control enabled?  The error you're
>> >> getting
>> >> >> >> >> >doesn't match what I remember it being the last time I hit
>> it, so
>> >> >> I'm
>> >> >> >> >> >assuming you don't.  PFC gives log lines that at least tell
>> you
>> >> >> exactly
>> >> >> >> >> >which limit you ran into, plus it'll avoid losing any messages
>> >> (but
>> >> >> >> it'll
>> >> >> >> >> >"hang" producers till messages are consumed), so you could
>> enable
>> >> >> it to
>> >> >> >> >> >better understand what's going on.
>> >> >> >> >> >
>> >> >> >> >> >Tim
>> >> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <disrael@liveops.com
>> >
>> >> >> wrote:
>> >> >> >> >> >
>> >> >> >> >> >>
>> >> >> >> >> >> Hello All,
>> >> >> >> >> >>
>> >> >> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
>> >> >> seeing
>> >> >> >> >> >> exceptions like this:
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> Async error occurred: javax.jms.ResourceAllocationException:
>> >> Usage
>> >> >> >> >> Manager
>> >> >> >> >> >> memory limit reached
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> I can't tell if this is because I am exceeding the
>> configured
>> >> >> amount
>> >> >> >> of
>> >> >> >> >> >> memory in SystemUsage, or if I am exceeding the amount of
>> >> memory
>> >> >> >> >> configured
>> >> >> >> >> >> per topic.
>> >> >> >> >> >>
>> >> >> >> >> >> I am using only Topics right now, and I had the memory limit
>> >> set
>> >> >> to
>> >> >> >> >> 10mb.
>> >> >> >> >> >> The error doesn't direct me in any direction for this.  I am
>> >> >> using an
>> >> >> >> >> old
>> >> >> >> >> >> version of AMQ (First step was to request upgrade to latest
>> >> >> version.
>> >> >> >> >> It's
>> >> >> >> >> >> in the works, but it might be a week or two before it's
>> >> completed
>> >> >> in
>> >> >> >> >> >> production) and I see changes in the source that give more
>> >> details
>> >> >> >> when
>> >> >> >> >> >> throwing this exception.  Is there some historical record or
>> >> log
>> >> >> of
>> >> >> >> >> >> Topics?  What I'd really like is to be able to see how often
>> >> each
>> >> >> >> Topic
>> >> >> >> >> >> gets and distributes a message and how big that message was.
>> >> The
>> >> >> >> >> dashboard
>> >> >> >> >> >> and Jconsole give me some information, but because Topics
>> are
>> >> >> >> delivered
>> >> >> >> >> >> then released, I don't have any information beyond how many
>> >> were
>> >> >> >> >> enqueued
>> >> >> >> >> >> and delivered.
>> >> >> >> >> >>
>> >> >> >> >> >> Is there any such animal available that would help me with
>> >> this?
>> >> >> Or
>> >> >> >> >> >> suggestions how to approach?  Any help is appreciated.
>> Thanks.
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

Yes, as long as that's on the subscription, not the topic itself.  Caveat:
I haven't actually looked at JMX in 5.3.2, so I'm extrapolating from what
I've seen in later versions, but I think that should be right.
On Aug 24, 2015 1:29 PM, "Daniel Israel" <di...@liveops.com> wrote:

>
>
>
>
>
> On 8/21/15, 5:28 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >You can tell if a consumer is slower than the rest via JMX.  Look at each
> >subscription for the topic and see if any of them has lots of pending
> >messages while the rest do not.
>
> Is that the PendingQueueSize on the JMX console?
>
> Thanks.
>
>
> >If you discover that to be the case,
> >configure a slow consumer abortion strategy to kick them off and throw
> away
> >their pending messages.  (Note that this only works for topics, and only
> >for non-durable subscriptions, but I think both of those apply to your
> >scenario.)
> >On Aug 21, 2015 12:30 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >
> >>
> >>
> >>
> >>
> >>
> >> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >The broker can't discard the message till all subscribers have consumed
> >> it,
> >>
> >> This is good information, thanks!  I am wondering if there is a way to
> >> detect this?
> >>
> >> >so a single slow consumer will result in increased memory/store usage
> and
> >> >eventually PFC.  You could set up a slow consumer abortion strategy if
> >> >you're worried about that possibility.
> >> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >> >
> >> >> Oooh, that's a good idea to add another subber.  I may be able to try
> >> that.
> >> >>
> >> >> Also, could something on the client side cause a backup?  Or will the
> >> >> broker just fall through if a client hangs?
> >> >>
> >> >>
> >> >>
> >> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >>
> >> >> >Since this is happening in production, I assume it's not acceptable
> to
> >> >> >attach a debugger and set a breakpoint on the line where the
> exception
> >> is
> >> >> >thrown.  If you could, that would let you step into the
> >> >> >MemoryUsage.isFull() call and see what's going on, but it'll hang
> the
> >> >> >broker when it happens and I assume that won't fly.
> >> >> >
> >> >> >You could probably figure out the size of the messages just by
> >> creating a
> >> >> >test app that just logs the size of each message it receives and
> >> having it
> >> >> >become another subscriber on the topic.  It won't tell you the rate
> at
> >> >> >which your consumers consume, but you'll know the input rates at
> least.
> >> >> >
> >> >> >If you're worried that your topics are filling up because your
> >> consumers
> >> >> >are falling behind, you could force them to disconnect (and throw
> away
> >> >> >their pending messages) if they get too far behind.
> >> >> >http://activemq.apache.org/slow-consumer-handling.html  Just make
> sure
> >> >> the
> >> >> >limit you set is smaller than the PFC per-destination limit, or
> you'll
> >> >> >never hit it.
> >> >> >
> >> >> >Tim
> >> >> >
> >> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <
> disrael@liveops.com>
> >> >> wrote:
> >> >> >
> >> >> >>
> >> >> >> Forgot to attach.  This is log message:
> >> >> >>
> >> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
> >> >> >>     - Async error occurred: javax.jms.ResourceAllocationException:
> >> Usage
> >> >> >> Manager memory limit reached
> >> >> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
> >> >> reached
> >> >> >>     at
> org.apache.activemq.broker.region.Topic.send(Topic.java:293)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> >> >> >>     at
> >> >> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> >> >> >>     at
> >> >> >>
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> >> >> >>     at java.lang.Thread.run(Thread.java:662)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >>
> >> >> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and
> 5.10.0)
> >> look
> >> >> >> >like the one in this page (
> >> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/),
> >> which
> >> >> >> give
> >> >> >> >lots of information about what limit is being hit.  I guess that
> >> >> detailed
> >> >> >> >info must have been added in a version after the one you're
> using.
> >> >> What
> >> >> >> >version is that, anyway?
> >> >> >> >
> >> >> >> >Can you post the full log message?
> >> >> >> >
> >> >> >> >Have you explored the JMX tree to see if there is any info to
> tell
> >> you
> >> >> how
> >> >> >> >full your topics are?  Obviously what information is available
> will
> >> >> vary
> >> >> >> >based on what version you're running, so that might not help, but
> >> you
> >> >> >> >should definitely check.
> >> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com>
> >> wrote:
> >> >> >> >
> >> >> >> >>
> >> >> >> >> Hi Tim, thanks for response.
> >> >> >> >>
> >> >> >> >> Flow control is enabled, and it's configured to fail if out of
> >> >> memory.
> >> >> >> As
> >> >> >> >> noted below, the log lines in this version don't tell us which
> >> limit
> >> >> >> we're
> >> >> >> >> exceeding, so we're running half blind :(.  Knowing average
> topic
> >> >> size
> >> >> >> >> would be helpful, but having individual topic size would be
> good.
> >> >> Right
> >> >> >> >> now, looking at the producer side to see if there is some way
> to
> >> >> track.
> >> >> >> >>
> >> >> >> >> Raised the topic limit to 20mb and still had an issue.  It's
> very
> >> hit
> >> >> >> and
> >> >> >> >> miss.  I can run for a week without issue, then when I get hit
> >> hard,
> >> >> it
> >> >> >> >> falls over.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim
> Bain" <
> >> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >> >>
> >> >> >> >> >Later versions give a few addition stats (such as average
> message
> >> >> size)
> >> >> >> >> via
> >> >> >> >> >JMX, but that won't help you till that upgrade in production
> is
> >> >> >> complete.
> >> >> >> >> >
> >> >> >> >> >Do you have producer flow control enabled?  The error you're
> >> getting
> >> >> >> >> >doesn't match what I remember it being the last time I hit
> it, so
> >> >> I'm
> >> >> >> >> >assuming you don't.  PFC gives log lines that at least tell
> you
> >> >> exactly
> >> >> >> >> >which limit you ran into, plus it'll avoid losing any messages
> >> (but
> >> >> >> it'll
> >> >> >> >> >"hang" producers till messages are consumed), so you could
> enable
> >> >> it to
> >> >> >> >> >better understand what's going on.
> >> >> >> >> >
> >> >> >> >> >Tim
> >> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <disrael@liveops.com
> >
> >> >> wrote:
> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >> Hello All,
> >> >> >> >> >>
> >> >> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
> >> >> seeing
> >> >> >> >> >> exceptions like this:
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> Async error occurred: javax.jms.ResourceAllocationException:
> >> Usage
> >> >> >> >> Manager
> >> >> >> >> >> memory limit reached
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> I can't tell if this is because I am exceeding the
> configured
> >> >> amount
> >> >> >> of
> >> >> >> >> >> memory in SystemUsage, or if I am exceeding the amount of
> >> memory
> >> >> >> >> configured
> >> >> >> >> >> per topic.
> >> >> >> >> >>
> >> >> >> >> >> I am using only Topics right now, and I had the memory limit
> >> set
> >> >> to
> >> >> >> >> 10mb.
> >> >> >> >> >> The error doesn't direct me in any direction for this.  I am
> >> >> using an
> >> >> >> >> old
> >> >> >> >> >> version of AMQ (First step was to request upgrade to latest
> >> >> version.
> >> >> >> >> It's
> >> >> >> >> >> in the works, but it might be a week or two before it's
> >> completed
> >> >> in
> >> >> >> >> >> production) and I see changes in the source that give more
> >> details
> >> >> >> when
> >> >> >> >> >> throwing this exception.  Is there some historical record or
> >> log
> >> >> of
> >> >> >> >> >> Topics?  What I'd really like is to be able to see how often
> >> each
> >> >> >> Topic
> >> >> >> >> >> gets and distributes a message and how big that message was.
> >> The
> >> >> >> >> dashboard
> >> >> >> >> >> and Jconsole give me some information, but because Topics
> are
> >> >> >> delivered
> >> >> >> >> >> then released, I don't have any information beyond how many
> >> were
> >> >> >> >> enqueued
> >> >> >> >> >> and delivered.
> >> >> >> >> >>
> >> >> >> >> >> Is there any such animal available that would help me with
> >> this?
> >> >> Or
> >> >> >> >> >> suggestions how to approach?  Any help is appreciated.
> Thanks.
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.





On 8/21/15, 5:28 AM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>You can tell if a consumer is slower than the rest via JMX.  Look at each
>subscription for the topic and see if any of them has lots of pending
>messages while the rest do not.  

Is that the PendingQueueSize on the JMX console?

Thanks.


>If you discover that to be the case,
>configure a slow consumer abortion strategy to kick them off and throw away
>their pending messages.  (Note that this only works for topics, and only
>for non-durable subscriptions, but I think both of those apply to your
>scenario.)
>On Aug 21, 2015 12:30 AM, "Daniel Israel" <di...@liveops.com> wrote:
>
>>
>>
>>
>>
>>
>> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >The broker can't discard the message till all subscribers have consumed
>> it,
>>
>> This is good information, thanks!  I am wondering if there is a way to
>> detect this?
>>
>> >so a single slow consumer will result in increased memory/store usage and
>> >eventually PFC.  You could set up a slow consumer abortion strategy if
>> >you're worried about that possibility.
>> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:
>> >
>> >> Oooh, that's a good idea to add another subber.  I may be able to try
>> that.
>> >>
>> >> Also, could something on the client side cause a backup?  Or will the
>> >> broker just fall through if a client hangs?
>> >>
>> >>
>> >>
>> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >>
>> >> >Since this is happening in production, I assume it's not acceptable to
>> >> >attach a debugger and set a breakpoint on the line where the exception
>> is
>> >> >thrown.  If you could, that would let you step into the
>> >> >MemoryUsage.isFull() call and see what's going on, but it'll hang the
>> >> >broker when it happens and I assume that won't fly.
>> >> >
>> >> >You could probably figure out the size of the messages just by
>> creating a
>> >> >test app that just logs the size of each message it receives and
>> having it
>> >> >become another subscriber on the topic.  It won't tell you the rate at
>> >> >which your consumers consume, but you'll know the input rates at least.
>> >> >
>> >> >If you're worried that your topics are filling up because your
>> consumers
>> >> >are falling behind, you could force them to disconnect (and throw away
>> >> >their pending messages) if they get too far behind.
>> >> >http://activemq.apache.org/slow-consumer-handling.html  Just make sure
>> >> the
>> >> >limit you set is smaller than the PFC per-destination limit, or you'll
>> >> >never hit it.
>> >> >
>> >> >Tim
>> >> >
>> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com>
>> >> wrote:
>> >> >
>> >> >>
>> >> >> Forgot to attach.  This is log message:
>> >> >>
>> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
>> >> >>     - Async error occurred: javax.jms.ResourceAllocationException:
>> Usage
>> >> >> Manager memory limit reached
>> >> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
>> >> reached
>> >> >>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>> >> >>     at
>> >> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>> >> >>     at
>> >> >>
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>> >> >>     at java.lang.Thread.run(Thread.java:662)
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >>
>> >> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0)
>> look
>> >> >> >like the one in this page (
>> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/),
>> which
>> >> >> give
>> >> >> >lots of information about what limit is being hit.  I guess that
>> >> detailed
>> >> >> >info must have been added in a version after the one you're using.
>> >> What
>> >> >> >version is that, anyway?
>> >> >> >
>> >> >> >Can you post the full log message?
>> >> >> >
>> >> >> >Have you explored the JMX tree to see if there is any info to tell
>> you
>> >> how
>> >> >> >full your topics are?  Obviously what information is available will
>> >> vary
>> >> >> >based on what version you're running, so that might not help, but
>> you
>> >> >> >should definitely check.
>> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com>
>> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Hi Tim, thanks for response.
>> >> >> >>
>> >> >> >> Flow control is enabled, and it's configured to fail if out of
>> >> memory.
>> >> >> As
>> >> >> >> noted below, the log lines in this version don't tell us which
>> limit
>> >> >> we're
>> >> >> >> exceeding, so we're running half blind :(.  Knowing average topic
>> >> size
>> >> >> >> would be helpful, but having individual topic size would be good.
>> >> Right
>> >> >> >> now, looking at the producer side to see if there is some way to
>> >> track.
>> >> >> >>
>> >> >> >> Raised the topic limit to 20mb and still had an issue.  It's very
>> hit
>> >> >> and
>> >> >> >> miss.  I can run for a week without issue, then when I get hit
>> hard,
>> >> it
>> >> >> >> falls over.
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >> >>
>> >> >> >> >Later versions give a few addition stats (such as average message
>> >> size)
>> >> >> >> via
>> >> >> >> >JMX, but that won't help you till that upgrade in production is
>> >> >> complete.
>> >> >> >> >
>> >> >> >> >Do you have producer flow control enabled?  The error you're
>> getting
>> >> >> >> >doesn't match what I remember it being the last time I hit it, so
>> >> I'm
>> >> >> >> >assuming you don't.  PFC gives log lines that at least tell you
>> >> exactly
>> >> >> >> >which limit you ran into, plus it'll avoid losing any messages
>> (but
>> >> >> it'll
>> >> >> >> >"hang" producers till messages are consumed), so you could enable
>> >> it to
>> >> >> >> >better understand what's going on.
>> >> >> >> >
>> >> >> >> >Tim
>> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com>
>> >> wrote:
>> >> >> >> >
>> >> >> >> >>
>> >> >> >> >> Hello All,
>> >> >> >> >>
>> >> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
>> >> seeing
>> >> >> >> >> exceptions like this:
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Async error occurred: javax.jms.ResourceAllocationException:
>> Usage
>> >> >> >> Manager
>> >> >> >> >> memory limit reached
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I can't tell if this is because I am exceeding the configured
>> >> amount
>> >> >> of
>> >> >> >> >> memory in SystemUsage, or if I am exceeding the amount of
>> memory
>> >> >> >> configured
>> >> >> >> >> per topic.
>> >> >> >> >>
>> >> >> >> >> I am using only Topics right now, and I had the memory limit
>> set
>> >> to
>> >> >> >> 10mb.
>> >> >> >> >> The error doesn't direct me in any direction for this.  I am
>> >> using an
>> >> >> >> old
>> >> >> >> >> version of AMQ (First step was to request upgrade to latest
>> >> version.
>> >> >> >> It's
>> >> >> >> >> in the works, but it might be a week or two before it's
>> completed
>> >> in
>> >> >> >> >> production) and I see changes in the source that give more
>> details
>> >> >> when
>> >> >> >> >> throwing this exception.  Is there some historical record or
>> log
>> >> of
>> >> >> >> >> Topics?  What I'd really like is to be able to see how often
>> each
>> >> >> Topic
>> >> >> >> >> gets and distributes a message and how big that message was.
>> The
>> >> >> >> dashboard
>> >> >> >> >> and Jconsole give me some information, but because Topics are
>> >> >> delivered
>> >> >> >> >> then released, I don't have any information beyond how many
>> were
>> >> >> >> enqueued
>> >> >> >> >> and delivered.
>> >> >> >> >>
>> >> >> >> >> Is there any such animal available that would help me with
>> this?
>> >> Or
>> >> >> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
>> >> >> >> >>
>> >> >> >> >>
>> >> >>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

You can tell if a consumer is slower than the rest via JMX.  Look at each
subscription for the topic and see if any of them has lots of pending
messages while the rest do not.  If you discover that to be the case,
configure a slow consumer abortion strategy to kick them off and throw away
their pending messages.  (Note that this only works for topics, and only
for non-durable subscriptions, but I think both of those apply to your
scenario.)
On Aug 21, 2015 12:30 AM, "Daniel Israel" <di...@liveops.com> wrote:

>
>
>
>
>
> On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >The broker can't discard the message till all subscribers have consumed
> it,
>
> This is good information, thanks!  I am wondering if there is a way to
> detect this?
>
> >so a single slow consumer will result in increased memory/store usage and
> >eventually PFC.  You could set up a slow consumer abortion strategy if
> >you're worried about that possibility.
> >On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >
> >> Oooh, that's a good idea to add another subber.  I may be able to try
> that.
> >>
> >> Also, could something on the client side cause a backup?  Or will the
> >> broker just fall through if a client hangs?
> >>
> >>
> >>
> >> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >Since this is happening in production, I assume it's not acceptable to
> >> >attach a debugger and set a breakpoint on the line where the exception
> is
> >> >thrown.  If you could, that would let you step into the
> >> >MemoryUsage.isFull() call and see what's going on, but it'll hang the
> >> >broker when it happens and I assume that won't fly.
> >> >
> >> >You could probably figure out the size of the messages just by
> creating a
> >> >test app that just logs the size of each message it receives and
> having it
> >> >become another subscriber on the topic.  It won't tell you the rate at
> >> >which your consumers consume, but you'll know the input rates at least.
> >> >
> >> >If you're worried that your topics are filling up because your
> consumers
> >> >are falling behind, you could force them to disconnect (and throw away
> >> >their pending messages) if they get too far behind.
> >> >http://activemq.apache.org/slow-consumer-handling.html  Just make sure
> >> the
> >> >limit you set is smaller than the PFC per-destination limit, or you'll
> >> >never hit it.
> >> >
> >> >Tim
> >> >
> >> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com>
> >> wrote:
> >> >
> >> >>
> >> >> Forgot to attach.  This is log message:
> >> >>
> >> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
> >> >>     - Async error occurred: javax.jms.ResourceAllocationException:
> Usage
> >> >> Manager memory limit reached
> >> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
> >> reached
> >> >>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> >> >>     at
> >> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> >> >>     at
> >> >>
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> >> >>     at java.lang.Thread.run(Thread.java:662)
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >>
> >> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0)
> look
> >> >> >like the one in this page (
> >> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/),
> which
> >> >> give
> >> >> >lots of information about what limit is being hit.  I guess that
> >> detailed
> >> >> >info must have been added in a version after the one you're using.
> >> What
> >> >> >version is that, anyway?
> >> >> >
> >> >> >Can you post the full log message?
> >> >> >
> >> >> >Have you explored the JMX tree to see if there is any info to tell
> you
> >> how
> >> >> >full your topics are?  Obviously what information is available will
> >> vary
> >> >> >based on what version you're running, so that might not help, but
> you
> >> >> >should definitely check.
> >> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com>
> wrote:
> >> >> >
> >> >> >>
> >> >> >> Hi Tim, thanks for response.
> >> >> >>
> >> >> >> Flow control is enabled, and it's configured to fail if out of
> >> memory.
> >> >> As
> >> >> >> noted below, the log lines in this version don't tell us which
> limit
> >> >> we're
> >> >> >> exceeding, so we're running half blind :(.  Knowing average topic
> >> size
> >> >> >> would be helpful, but having individual topic size would be good.
> >> Right
> >> >> >> now, looking at the producer side to see if there is some way to
> >> track.
> >> >> >>
> >> >> >> Raised the topic limit to 20mb and still had an issue.  It's very
> hit
> >> >> and
> >> >> >> miss.  I can run for a week without issue, then when I get hit
> hard,
> >> it
> >> >> >> falls over.
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >> >>
> >> >> >> >Later versions give a few addition stats (such as average message
> >> size)
> >> >> >> via
> >> >> >> >JMX, but that won't help you till that upgrade in production is
> >> >> complete.
> >> >> >> >
> >> >> >> >Do you have producer flow control enabled?  The error you're
> getting
> >> >> >> >doesn't match what I remember it being the last time I hit it, so
> >> I'm
> >> >> >> >assuming you don't.  PFC gives log lines that at least tell you
> >> exactly
> >> >> >> >which limit you ran into, plus it'll avoid losing any messages
> (but
> >> >> it'll
> >> >> >> >"hang" producers till messages are consumed), so you could enable
> >> it to
> >> >> >> >better understand what's going on.
> >> >> >> >
> >> >> >> >Tim
> >> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com>
> >> wrote:
> >> >> >> >
> >> >> >> >>
> >> >> >> >> Hello All,
> >> >> >> >>
> >> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
> >> seeing
> >> >> >> >> exceptions like this:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Async error occurred: javax.jms.ResourceAllocationException:
> Usage
> >> >> >> Manager
> >> >> >> >> memory limit reached
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> I can't tell if this is because I am exceeding the configured
> >> amount
> >> >> of
> >> >> >> >> memory in SystemUsage, or if I am exceeding the amount of
> memory
> >> >> >> configured
> >> >> >> >> per topic.
> >> >> >> >>
> >> >> >> >> I am using only Topics right now, and I had the memory limit
> set
> >> to
> >> >> >> 10mb.
> >> >> >> >> The error doesn't direct me in any direction for this.  I am
> >> using an
> >> >> >> old
> >> >> >> >> version of AMQ (First step was to request upgrade to latest
> >> version.
> >> >> >> It's
> >> >> >> >> in the works, but it might be a week or two before it's
> completed
> >> in
> >> >> >> >> production) and I see changes in the source that give more
> details
> >> >> when
> >> >> >> >> throwing this exception.  Is there some historical record or
> log
> >> of
> >> >> >> >> Topics?  What I'd really like is to be able to see how often
> each
> >> >> Topic
> >> >> >> >> gets and distributes a message and how big that message was.
> The
> >> >> >> dashboard
> >> >> >> >> and Jconsole give me some information, but because Topics are
> >> >> delivered
> >> >> >> >> then released, I don't have any information beyond how many
> were
> >> >> >> enqueued
> >> >> >> >> and delivered.
> >> >> >> >>
> >> >> >> >> Is there any such animal available that would help me with
> this?
> >> Or
> >> >> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
> >> >> >> >>
> >> >> >> >>
> >> >>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.





On 8/20/15, 9:56 PM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>The broker can't discard the message till all subscribers have consumed it,

This is good information, thanks!  I am wondering if there is a way to detect this?

>so a single slow consumer will result in increased memory/store usage and
>eventually PFC.  You could set up a slow consumer abortion strategy if
>you're worried about that possibility.
>On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:
>
>> Oooh, that's a good idea to add another subber.  I may be able to try that.
>>
>> Also, could something on the client side cause a backup?  Or will the
>> broker just fall through if a client hangs?
>>
>>
>>
>> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >Since this is happening in production, I assume it's not acceptable to
>> >attach a debugger and set a breakpoint on the line where the exception is
>> >thrown.  If you could, that would let you step into the
>> >MemoryUsage.isFull() call and see what's going on, but it'll hang the
>> >broker when it happens and I assume that won't fly.
>> >
>> >You could probably figure out the size of the messages just by creating a
>> >test app that just logs the size of each message it receives and having it
>> >become another subscriber on the topic.  It won't tell you the rate at
>> >which your consumers consume, but you'll know the input rates at least.
>> >
>> >If you're worried that your topics are filling up because your consumers
>> >are falling behind, you could force them to disconnect (and throw away
>> >their pending messages) if they get too far behind.
>> >http://activemq.apache.org/slow-consumer-handling.html  Just make sure
>> the
>> >limit you set is smaller than the PFC per-destination limit, or you'll
>> >never hit it.
>> >
>> >Tim
>> >
>> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com>
>> wrote:
>> >
>> >>
>> >> Forgot to attach.  This is log message:
>> >>
>> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
>> >>     - Async error occurred: javax.jms.ResourceAllocationException: Usage
>> >> Manager memory limit reached
>> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
>> reached
>> >>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
>> >>     at
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
>> >>     at
>> >>
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
>> >>     at
>> >>
>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>> >>     at
>> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>> >>     at
>> >>
>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>> >>     at
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>> >>     at
>> >>
>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
>> >>     at
>> >>
>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>> >>     at
>> >>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
>> >>     at
>> >>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
>> >>     at
>> >>
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>> >>     at
>> >>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>> >>     at
>> >>
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>> >>     at
>> >>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>> >>     at
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>> >>     at
>> >>
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>> >>     at java.lang.Thread.run(Thread.java:662)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >>
>> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
>> >> >like the one in this page (
>> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which
>> >> give
>> >> >lots of information about what limit is being hit.  I guess that
>> detailed
>> >> >info must have been added in a version after the one you're using.
>> What
>> >> >version is that, anyway?
>> >> >
>> >> >Can you post the full log message?
>> >> >
>> >> >Have you explored the JMX tree to see if there is any info to tell you
>> how
>> >> >full your topics are?  Obviously what information is available will
>> vary
>> >> >based on what version you're running, so that might not help, but you
>> >> >should definitely check.
>> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
>> >> >
>> >> >>
>> >> >> Hi Tim, thanks for response.
>> >> >>
>> >> >> Flow control is enabled, and it's configured to fail if out of
>> memory.
>> >> As
>> >> >> noted below, the log lines in this version don't tell us which limit
>> >> we're
>> >> >> exceeding, so we're running half blind :(.  Knowing average topic
>> size
>> >> >> would be helpful, but having individual topic size would be good.
>> Right
>> >> >> now, looking at the producer side to see if there is some way to
>> track.
>> >> >>
>> >> >> Raised the topic limit to 20mb and still had an issue.  It's very hit
>> >> and
>> >> >> miss.  I can run for a week without issue, then when I get hit hard,
>> it
>> >> >> falls over.
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >> >>
>> >> >> >Later versions give a few addition stats (such as average message
>> size)
>> >> >> via
>> >> >> >JMX, but that won't help you till that upgrade in production is
>> >> complete.
>> >> >> >
>> >> >> >Do you have producer flow control enabled?  The error you're getting
>> >> >> >doesn't match what I remember it being the last time I hit it, so
>> I'm
>> >> >> >assuming you don't.  PFC gives log lines that at least tell you
>> exactly
>> >> >> >which limit you ran into, plus it'll avoid losing any messages (but
>> >> it'll
>> >> >> >"hang" producers till messages are consumed), so you could enable
>> it to
>> >> >> >better understand what's going on.
>> >> >> >
>> >> >> >Tim
>> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com>
>> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Hello All,
>> >> >> >>
>> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
>> seeing
>> >> >> >> exceptions like this:
>> >> >> >>
>> >> >> >>
>> >> >> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
>> >> >> Manager
>> >> >> >> memory limit reached
>> >> >> >>
>> >> >> >>
>> >> >> >> I can't tell if this is because I am exceeding the configured
>> amount
>> >> of
>> >> >> >> memory in SystemUsage, or if I am exceeding the amount of memory
>> >> >> configured
>> >> >> >> per topic.
>> >> >> >>
>> >> >> >> I am using only Topics right now, and I had the memory limit set
>> to
>> >> >> 10mb.
>> >> >> >> The error doesn't direct me in any direction for this.  I am
>> using an
>> >> >> old
>> >> >> >> version of AMQ (First step was to request upgrade to latest
>> version.
>> >> >> It's
>> >> >> >> in the works, but it might be a week or two before it's completed
>> in
>> >> >> >> production) and I see changes in the source that give more details
>> >> when
>> >> >> >> throwing this exception.  Is there some historical record or log
>> of
>> >> >> >> Topics?  What I'd really like is to be able to see how often each
>> >> Topic
>> >> >> >> gets and distributes a message and how big that message was.  The
>> >> >> dashboard
>> >> >> >> and Jconsole give me some information, but because Topics are
>> >> delivered
>> >> >> >> then released, I don't have any information beyond how many were
>> >> >> enqueued
>> >> >> >> and delivered.
>> >> >> >>
>> >> >> >> Is there any such animal available that would help me with this?
>> Or
>> >> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
>> >> >> >>
>> >> >> >>
>> >>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

The broker can't discard the message till all subscribers have consumed it,
so a single slow consumer will result in increased memory/store usage and
eventually PFC.  You could set up a slow consumer abortion strategy if
you're worried about that possibility.
On Aug 20, 2015 10:13 AM, "Daniel Israel" <di...@liveops.com> wrote:

> Oooh, that's a good idea to add another subber.  I may be able to try that.
>
> Also, could something on the client side cause a backup?  Or will the
> broker just fall through if a client hangs?
>
>
>
> On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >Since this is happening in production, I assume it's not acceptable to
> >attach a debugger and set a breakpoint on the line where the exception is
> >thrown.  If you could, that would let you step into the
> >MemoryUsage.isFull() call and see what's going on, but it'll hang the
> >broker when it happens and I assume that won't fly.
> >
> >You could probably figure out the size of the messages just by creating a
> >test app that just logs the size of each message it receives and having it
> >become another subscriber on the topic.  It won't tell you the rate at
> >which your consumers consume, but you'll know the input rates at least.
> >
> >If you're worried that your topics are filling up because your consumers
> >are falling behind, you could force them to disconnect (and throw away
> >their pending messages) if they get too far behind.
> >http://activemq.apache.org/slow-consumer-handling.html  Just make sure
> the
> >limit you set is smaller than the PFC per-destination limit, or you'll
> >never hit it.
> >
> >Tim
> >
> >On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com>
> wrote:
> >
> >>
> >> Forgot to attach.  This is log message:
> >>
> >> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
> >>     - Async error occurred: javax.jms.ResourceAllocationException: Usage
> >> Manager memory limit reached
> >> javax.jms.ResourceAllocationException: Usage Manager memory limit
> reached
> >>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
> >>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
> >>     at
> >>
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
> >>     at
> >>
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> >>     at
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> >>     at
> >>
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> >>     at
> >>
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> >>     at
> >>
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
> >>     at
> >>
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> >>     at
> >>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
> >>     at
> >>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
> >>     at
> >>
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> >>     at
> >>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
> >>     at
> >>
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
> >>     at
> >>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> >>     at
> >>
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> >>     at
> >>
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> >>     at java.lang.Thread.run(Thread.java:662)
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
> >> >like the one in this page (
> >> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which
> >> give
> >> >lots of information about what limit is being hit.  I guess that
> detailed
> >> >info must have been added in a version after the one you're using.
> What
> >> >version is that, anyway?
> >> >
> >> >Can you post the full log message?
> >> >
> >> >Have you explored the JMX tree to see if there is any info to tell you
> how
> >> >full your topics are?  Obviously what information is available will
> vary
> >> >based on what version you're running, so that might not help, but you
> >> >should definitely check.
> >> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >> >
> >> >>
> >> >> Hi Tim, thanks for response.
> >> >>
> >> >> Flow control is enabled, and it's configured to fail if out of
> memory.
> >> As
> >> >> noted below, the log lines in this version don't tell us which limit
> >> we're
> >> >> exceeding, so we're running half blind :(.  Knowing average topic
> size
> >> >> would be helpful, but having individual topic size would be good.
> Right
> >> >> now, looking at the producer side to see if there is some way to
> track.
> >> >>
> >> >> Raised the topic limit to 20mb and still had an issue.  It's very hit
> >> and
> >> >> miss.  I can run for a week without issue, then when I get hit hard,
> it
> >> >> falls over.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >> >>
> >> >> >Later versions give a few addition stats (such as average message
> size)
> >> >> via
> >> >> >JMX, but that won't help you till that upgrade in production is
> >> complete.
> >> >> >
> >> >> >Do you have producer flow control enabled?  The error you're getting
> >> >> >doesn't match what I remember it being the last time I hit it, so
> I'm
> >> >> >assuming you don't.  PFC gives log lines that at least tell you
> exactly
> >> >> >which limit you ran into, plus it'll avoid losing any messages (but
> >> it'll
> >> >> >"hang" producers till messages are consumed), so you could enable
> it to
> >> >> >better understand what's going on.
> >> >> >
> >> >> >Tim
> >> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com>
> wrote:
> >> >> >
> >> >> >>
> >> >> >> Hello All,
> >> >> >>
> >> >> >> I am tracking down a memory issue in AMQ.  I am time to time
> seeing
> >> >> >> exceptions like this:
> >> >> >>
> >> >> >>
> >> >> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
> >> >> Manager
> >> >> >> memory limit reached
> >> >> >>
> >> >> >>
> >> >> >> I can't tell if this is because I am exceeding the configured
> amount
> >> of
> >> >> >> memory in SystemUsage, or if I am exceeding the amount of memory
> >> >> configured
> >> >> >> per topic.
> >> >> >>
> >> >> >> I am using only Topics right now, and I had the memory limit set
> to
> >> >> 10mb.
> >> >> >> The error doesn't direct me in any direction for this.  I am
> using an
> >> >> old
> >> >> >> version of AMQ (First step was to request upgrade to latest
> version.
> >> >> It's
> >> >> >> in the works, but it might be a week or two before it's completed
> in
> >> >> >> production) and I see changes in the source that give more details
> >> when
> >> >> >> throwing this exception.  Is there some historical record or log
> of
> >> >> >> Topics?  What I'd really like is to be able to see how often each
> >> Topic
> >> >> >> gets and distributes a message and how big that message was.  The
> >> >> dashboard
> >> >> >> and Jconsole give me some information, but because Topics are
> >> delivered
> >> >> >> then released, I don't have any information beyond how many were
> >> >> enqueued
> >> >> >> and delivered.
> >> >> >>
> >> >> >> Is there any such animal available that would help me with this?
> Or
> >> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
> >> >> >>
> >> >> >>
> >>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.

Oooh, that's a good idea to add another subber.  I may be able to try that.  

Also, could something on the client side cause a backup?  Or will the broker just fall through if a client hangs?



On 8/20/15, 7:18 AM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>Since this is happening in production, I assume it's not acceptable to
>attach a debugger and set a breakpoint on the line where the exception is
>thrown.  If you could, that would let you step into the
>MemoryUsage.isFull() call and see what's going on, but it'll hang the
>broker when it happens and I assume that won't fly.
>
>You could probably figure out the size of the messages just by creating a
>test app that just logs the size of each message it receives and having it
>become another subscriber on the topic.  It won't tell you the rate at
>which your consumers consume, but you'll know the input rates at least.
>
>If you're worried that your topics are filling up because your consumers
>are falling behind, you could force them to disconnect (and throw away
>their pending messages) if they get too far behind.
>http://activemq.apache.org/slow-consumer-handling.html  Just make sure the
>limit you set is smaller than the PFC per-destination limit, or you'll
>never hit it.
>
>Tim
>
>On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com> wrote:
>
>>
>> Forgot to attach.  This is log message:
>>
>> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
>>     - Async error occurred: javax.jms.ResourceAllocationException: Usage
>> Manager memory limit reached
>> javax.jms.ResourceAllocationException: Usage Manager memory limit reached
>>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
>>     at
>> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
>>     at
>> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
>>     at
>> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>>     at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>>     at
>> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>>     at
>> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>>     at
>> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
>>     at
>> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>>     at
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
>>     at
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
>>     at
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>>     at
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>>     at
>> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>>     at
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>>     at
>> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>>     at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>>     at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>>
>>
>>
>>
>>
>> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
>> >like the one in this page (
>> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which
>> give
>> >lots of information about what limit is being hit.  I guess that detailed
>> >info must have been added in a version after the one you're using.  What
>> >version is that, anyway?
>> >
>> >Can you post the full log message?
>> >
>> >Have you explored the JMX tree to see if there is any info to tell you how
>> >full your topics are?  Obviously what information is available will vary
>> >based on what version you're running, so that might not help, but you
>> >should definitely check.
>> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
>> >
>> >>
>> >> Hi Tim, thanks for response.
>> >>
>> >> Flow control is enabled, and it's configured to fail if out of memory.
>> As
>> >> noted below, the log lines in this version don't tell us which limit
>> we're
>> >> exceeding, so we're running half blind :(.  Knowing average topic size
>> >> would be helpful, but having individual topic size would be good.  Right
>> >> now, looking at the producer side to see if there is some way to track.
>> >>
>> >> Raised the topic limit to 20mb and still had an issue.  It's very hit
>> and
>> >> miss.  I can run for a week without issue, then when I get hit hard, it
>> >> falls over.
>> >>
>> >>
>> >>
>> >>
>> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>> >>
>> >> >Later versions give a few addition stats (such as average message size)
>> >> via
>> >> >JMX, but that won't help you till that upgrade in production is
>> complete.
>> >> >
>> >> >Do you have producer flow control enabled?  The error you're getting
>> >> >doesn't match what I remember it being the last time I hit it, so I'm
>> >> >assuming you don't.  PFC gives log lines that at least tell you exactly
>> >> >which limit you ran into, plus it'll avoid losing any messages (but
>> it'll
>> >> >"hang" producers till messages are consumed), so you could enable it to
>> >> >better understand what's going on.
>> >> >
>> >> >Tim
>> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
>> >> >
>> >> >>
>> >> >> Hello All,
>> >> >>
>> >> >> I am tracking down a memory issue in AMQ.  I am time to time seeing
>> >> >> exceptions like this:
>> >> >>
>> >> >>
>> >> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
>> >> Manager
>> >> >> memory limit reached
>> >> >>
>> >> >>
>> >> >> I can't tell if this is because I am exceeding the configured amount
>> of
>> >> >> memory in SystemUsage, or if I am exceeding the amount of memory
>> >> configured
>> >> >> per topic.
>> >> >>
>> >> >> I am using only Topics right now, and I had the memory limit set to
>> >> 10mb.
>> >> >> The error doesn't direct me in any direction for this.  I am using an
>> >> old
>> >> >> version of AMQ (First step was to request upgrade to latest version.
>> >> It's
>> >> >> in the works, but it might be a week or two before it's completed in
>> >> >> production) and I see changes in the source that give more details
>> when
>> >> >> throwing this exception.  Is there some historical record or log of
>> >> >> Topics?  What I'd really like is to be able to see how often each
>> Topic
>> >> >> gets and distributes a message and how big that message was.  The
>> >> dashboard
>> >> >> and Jconsole give me some information, but because Topics are
>> delivered
>> >> >> then released, I don't have any information beyond how many were
>> >> enqueued
>> >> >> and delivered.
>> >> >>
>> >> >> Is there any such animal available that would help me with this?  Or
>> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
>> >> >>
>> >> >>
>>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

Since this is happening in production, I assume it's not acceptable to
attach a debugger and set a breakpoint on the line where the exception is
thrown.  If you could, that would let you step into the
MemoryUsage.isFull() call and see what's going on, but it'll hang the
broker when it happens and I assume that won't fly.

You could probably figure out the size of the messages just by creating a
test app that just logs the size of each message it receives and having it
become another subscriber on the topic.  It won't tell you the rate at
which your consumers consume, but you'll know the input rates at least.

If you're worried that your topics are filling up because your consumers
are falling behind, you could force them to disconnect (and throw away
their pending messages) if they get too far behind.
http://activemq.apache.org/slow-consumer-handling.html  Just make sure the
limit you set is smaller than the PFC per-destination limit, or you'll
never hit it.

Tim

On Wed, Aug 19, 2015 at 11:37 AM, Daniel Israel <di...@liveops.com> wrote:

>
> Forgot to attach.  This is log message:
>
> 2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service
>     - Async error occurred: javax.jms.ResourceAllocationException: Usage
> Manager memory limit reached
> javax.jms.ResourceAllocationException: Usage Manager memory limit reached
>     at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
>     at
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
>     at
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
>     at
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
>     at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
>     at
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
>     at
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
>     at
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
>     at
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
>     at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
>     at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
>     at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>     at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
>     at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
>     at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
>     at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>     at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
>     at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>
>
>
>
> On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
> >like the one in this page (
> >http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which
> give
> >lots of information about what limit is being hit.  I guess that detailed
> >info must have been added in a version after the one you're using.  What
> >version is that, anyway?
> >
> >Can you post the full log message?
> >
> >Have you explored the JMX tree to see if there is any info to tell you how
> >full your topics are?  Obviously what information is available will vary
> >based on what version you're running, so that might not help, but you
> >should definitely check.
> >On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
> >
> >>
> >> Hi Tim, thanks for response.
> >>
> >> Flow control is enabled, and it's configured to fail if out of memory.
> As
> >> noted below, the log lines in this version don't tell us which limit
> we're
> >> exceeding, so we're running half blind :(.  Knowing average topic size
> >> would be helpful, but having individual topic size would be good.  Right
> >> now, looking at the producer side to see if there is some way to track.
> >>
> >> Raised the topic limit to 20mb and still had an issue.  It's very hit
> and
> >> miss.  I can run for a week without issue, then when I get hit hard, it
> >> falls over.
> >>
> >>
> >>
> >>
> >> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> >> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
> >>
> >> >Later versions give a few addition stats (such as average message size)
> >> via
> >> >JMX, but that won't help you till that upgrade in production is
> complete.
> >> >
> >> >Do you have producer flow control enabled?  The error you're getting
> >> >doesn't match what I remember it being the last time I hit it, so I'm
> >> >assuming you don't.  PFC gives log lines that at least tell you exactly
> >> >which limit you ran into, plus it'll avoid losing any messages (but
> it'll
> >> >"hang" producers till messages are consumed), so you could enable it to
> >> >better understand what's going on.
> >> >
> >> >Tim
> >> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
> >> >
> >> >>
> >> >> Hello All,
> >> >>
> >> >> I am tracking down a memory issue in AMQ.  I am time to time seeing
> >> >> exceptions like this:
> >> >>
> >> >>
> >> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
> >> Manager
> >> >> memory limit reached
> >> >>
> >> >>
> >> >> I can't tell if this is because I am exceeding the configured amount
> of
> >> >> memory in SystemUsage, or if I am exceeding the amount of memory
> >> configured
> >> >> per topic.
> >> >>
> >> >> I am using only Topics right now, and I had the memory limit set to
> >> 10mb.
> >> >> The error doesn't direct me in any direction for this.  I am using an
> >> old
> >> >> version of AMQ (First step was to request upgrade to latest version.
> >> It's
> >> >> in the works, but it might be a week or two before it's completed in
> >> >> production) and I see changes in the source that give more details
> when
> >> >> throwing this exception.  Is there some historical record or log of
> >> >> Topics?  What I'd really like is to be able to see how often each
> Topic
> >> >> gets and distributes a message and how big that message was.  The
> >> dashboard
> >> >> and Jconsole give me some information, but because Topics are
> delivered
> >> >> then released, I don't have any information beyond how many were
> >> enqueued
> >> >> and delivered.
> >> >>
> >> >> Is there any such animal available that would help me with this?  Or
> >> >> suggestions how to approach?  Any help is appreciated.  Thanks.
> >> >>
> >> >>
>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.

Forgot to attach.  This is log message:

2013-03-30 22:34:42,824 [.96.47.33:34886] WARN  Service                        - Async error occurred: javax.jms.ResourceAllocationException: Usage Manager memory limit reached
javax.jms.ResourceAllocationException: Usage Manager memory limit reached
    at org.apache.activemq.broker.region.Topic.send(Topic.java:293)
    at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
    at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
    at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
    at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
    at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
    at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
    at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
    at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
    at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
    at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
    at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
    at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
    at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
    at java.lang.Thread.run(Thread.java:662)








On 8/19/15, 6:53 AM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
>like the one in this page (
>http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which give
>lots of information about what limit is being hit.  I guess that detailed
>info must have been added in a version after the one you're using.  What
>version is that, anyway?
>
>Can you post the full log message?
>
>Have you explored the JMX tree to see if there is any info to tell you how
>full your topics are?  Obviously what information is available will vary
>based on what version you're running, so that might not help, but you
>should definitely check.
>On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:
>
>>
>> Hi Tim, thanks for response.
>>
>> Flow control is enabled, and it's configured to fail if out of memory.  As
>> noted below, the log lines in this version don't tell us which limit we're
>> exceeding, so we're running half blind :(.  Knowing average topic size
>> would be helpful, but having individual topic size would be good.  Right
>> now, looking at the producer side to see if there is some way to track.
>>
>> Raised the topic limit to 20mb and still had an issue.  It's very hit and
>> miss.  I can run for a week without issue, then when I get hit hard, it
>> falls over.
>>
>>
>>
>>
>> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
>> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>>
>> >Later versions give a few addition stats (such as average message size)
>> via
>> >JMX, but that won't help you till that upgrade in production is complete.
>> >
>> >Do you have producer flow control enabled?  The error you're getting
>> >doesn't match what I remember it being the last time I hit it, so I'm
>> >assuming you don't.  PFC gives log lines that at least tell you exactly
>> >which limit you ran into, plus it'll avoid losing any messages (but it'll
>> >"hang" producers till messages are consumed), so you could enable it to
>> >better understand what's going on.
>> >
>> >Tim
>> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
>> >
>> >>
>> >> Hello All,
>> >>
>> >> I am tracking down a memory issue in AMQ.  I am time to time seeing
>> >> exceptions like this:
>> >>
>> >>
>> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
>> Manager
>> >> memory limit reached
>> >>
>> >>
>> >> I can't tell if this is because I am exceeding the configured amount of
>> >> memory in SystemUsage, or if I am exceeding the amount of memory
>> configured
>> >> per topic.
>> >>
>> >> I am using only Topics right now, and I had the memory limit set to
>> 10mb.
>> >> The error doesn't direct me in any direction for this.  I am using an
>> old
>> >> version of AMQ (First step was to request upgrade to latest version.
>> It's
>> >> in the works, but it might be a week or two before it's completed in
>> >> production) and I see changes in the source that give more details when
>> >> throwing this exception.  Is there some historical record or log of
>> >> Topics?  What I'd really like is to be able to see how often each Topic
>> >> gets and distributes a message and how big that message was.  The
>> dashboard
>> >> and Jconsole give me some information, but because Topics are delivered
>> >> then released, I don't have any information beyond how many were
>> enqueued
>> >> and delivered.
>> >>
>> >> Is there any such animal available that would help me with this?  Or
>> >> suggestions how to approach?  Any help is appreciated.  Thanks.
>> >>
>> >>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

Hmm, the error messages I'm used to seeing (from 5.8.0 and 5.10.0) look
like the one in this page (
http://blogs.sourceallies.com/2014/10/activemq-memory-tuning/), which give
lots of information about what limit is being hit.  I guess that detailed
info must have been added in a version after the one you're using.  What
version is that, anyway?

Can you post the full log message?

Have you explored the JMX tree to see if there is any info to tell you how
full your topics are?  Obviously what information is available will vary
based on what version you're running, so that might not help, but you
should definitely check.
On Aug 18, 2015 10:18 AM, "Daniel Israel" <di...@liveops.com> wrote:

>
> Hi Tim, thanks for response.
>
> Flow control is enabled, and it's configured to fail if out of memory.  As
> noted below, the log lines in this version don't tell us which limit we're
> exceeding, so we're running half blind :(.  Knowing average topic size
> would be helpful, but having individual topic size would be good.  Right
> now, looking at the producer side to see if there is some way to track.
>
> Raised the topic limit to 20mb and still had an issue.  It's very hit and
> miss.  I can run for a week without issue, then when I get hit hard, it
> falls over.
>
>
>
>
> On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <
> tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:
>
> >Later versions give a few addition stats (such as average message size)
> via
> >JMX, but that won't help you till that upgrade in production is complete.
> >
> >Do you have producer flow control enabled?  The error you're getting
> >doesn't match what I remember it being the last time I hit it, so I'm
> >assuming you don't.  PFC gives log lines that at least tell you exactly
> >which limit you ran into, plus it'll avoid losing any messages (but it'll
> >"hang" producers till messages are consumed), so you could enable it to
> >better understand what's going on.
> >
> >Tim
> >On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
> >
> >>
> >> Hello All,
> >>
> >> I am tracking down a memory issue in AMQ.  I am time to time seeing
> >> exceptions like this:
> >>
> >>
> >> Async error occurred: javax.jms.ResourceAllocationException: Usage
> Manager
> >> memory limit reached
> >>
> >>
> >> I can't tell if this is because I am exceeding the configured amount of
> >> memory in SystemUsage, or if I am exceeding the amount of memory
> configured
> >> per topic.
> >>
> >> I am using only Topics right now, and I had the memory limit set to
> 10mb.
> >> The error doesn't direct me in any direction for this.  I am using an
> old
> >> version of AMQ (First step was to request upgrade to latest version.
> It's
> >> in the works, but it might be a week or two before it's completed in
> >> production) and I see changes in the source that give more details when
> >> throwing this exception.  Is there some historical record or log of
> >> Topics?  What I'd really like is to be able to see how often each Topic
> >> gets and distributes a message and how big that message was.  The
> dashboard
> >> and Jconsole give me some information, but because Topics are delivered
> >> then released, I don't have any information beyond how many were
> enqueued
> >> and delivered.
> >>
> >> Is there any such animal available that would help me with this?  Or
> >> suggestions how to approach?  Any help is appreciated.  Thanks.
> >>
> >>

Re: User Memory Issues

Posted by Daniel Israel <di...@liveops.com>.

Hi Tim, thanks for response.  

Flow control is enabled, and it's configured to fail if out of memory.  As noted below, the log lines in this version don't tell us which limit we're exceeding, so we're running half blind :(.  Knowing average topic size would be helpful, but having individual topic size would be good.  Right now, looking at the producer side to see if there is some way to track.

Raised the topic limit to 20mb and still had an issue.  It's very hit and miss.  I can run for a week without issue, then when I get hit hard, it falls over. 




On 8/18/15, 5:58 AM, "tbain98@gmail.com on behalf of Tim Bain" <tbain98@gmail.com on behalf of tbain@alumni.duke.edu> wrote:

>Later versions give a few addition stats (such as average message size) via
>JMX, but that won't help you till that upgrade in production is complete.
>
>Do you have producer flow control enabled?  The error you're getting
>doesn't match what I remember it being the last time I hit it, so I'm
>assuming you don't.  PFC gives log lines that at least tell you exactly
>which limit you ran into, plus it'll avoid losing any messages (but it'll
>"hang" producers till messages are consumed), so you could enable it to
>better understand what's going on.
>
>Tim
>On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:
>
>>
>> Hello All,
>>
>> I am tracking down a memory issue in AMQ.  I am time to time seeing
>> exceptions like this:
>>
>>
>> Async error occurred: javax.jms.ResourceAllocationException: Usage Manager
>> memory limit reached
>>
>>
>> I can't tell if this is because I am exceeding the configured amount of
>> memory in SystemUsage, or if I am exceeding the amount of memory configured
>> per topic.
>>
>> I am using only Topics right now, and I had the memory limit set to 10mb.
>> The error doesn't direct me in any direction for this.  I am using an old
>> version of AMQ (First step was to request upgrade to latest version.  It's
>> in the works, but it might be a week or two before it's completed in
>> production) and I see changes in the source that give more details when
>> throwing this exception.  Is there some historical record or log of
>> Topics?  What I'd really like is to be able to see how often each Topic
>> gets and distributes a message and how big that message was.  The dashboard
>> and Jconsole give me some information, but because Topics are delivered
>> then released, I don't have any information beyond how many were enqueued
>> and delivered.
>>
>> Is there any such animal available that would help me with this?  Or
>> suggestions how to approach?  Any help is appreciated.  Thanks.
>>
>>

Re: User Memory Issues

Posted by Tim Bain <tb...@alumni.duke.edu>.

Later versions give a few addition stats (such as average message size) via
JMX, but that won't help you till that upgrade in production is complete.

Do you have producer flow control enabled?  The error you're getting
doesn't match what I remember it being the last time I hit it, so I'm
assuming you don't.  PFC gives log lines that at least tell you exactly
which limit you ran into, plus it'll avoid losing any messages (but it'll
"hang" producers till messages are consumed), so you could enable it to
better understand what's going on.

Tim
On Aug 17, 2015 1:45 PM, "Daniel Israel" <di...@liveops.com> wrote:

>
> Hello All,
>
> I am tracking down a memory issue in AMQ.  I am time to time seeing
> exceptions like this:
>
>
> Async error occurred: javax.jms.ResourceAllocationException: Usage Manager
> memory limit reached
>
>
> I can't tell if this is because I am exceeding the configured amount of
> memory in SystemUsage, or if I am exceeding the amount of memory configured
> per topic.
>
> I am using only Topics right now, and I had the memory limit set to 10mb.
> The error doesn't direct me in any direction for this.  I am using an old
> version of AMQ (First step was to request upgrade to latest version.  It's
> in the works, but it might be a week or two before it's completed in
> production) and I see changes in the source that give more details when
> throwing this exception.  Is there some historical record or log of
> Topics?  What I'd really like is to be able to see how often each Topic
> gets and distributes a message and how big that message was.  The dashboard
> and Jconsole give me some information, but because Topics are delivered
> then released, I don't have any information beyond how many were enqueued
> and delivered.
>
> Is there any such animal available that would help me with this?  Or
> suggestions how to approach?  Any help is appreciated.  Thanks.
>
>