Broker not releasing memory after restart

[hfridhi <ha...@gmail.com>]

Hello guys,
I know it could be shocking problem but it happened to me and I'm confused
too.
I started a broker to which I'm intensively sending messages through WS and
consuming them by a camel embedded route.
At a certain time, I'm facing "gc overhead limit exceeded java".
I tried to restart the broker and follow its memory using visualVM and I got
this 
<http://activemq.2283324.n4.nabble.com/file/t379890/Screenshot_2020-04-14_at_00.png> 
This means that even without any call, memory is already at a high level and
calling GC manually have no considerable effect.

PS: Below is my Java configuration of AMQ



Thank you for you help.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Broker not releasing memory after restart

[Tim Bain <tb...@alumni.duke.edu>]

Yes, it would be helpful to have your exact configuration, and to know the
version of ActiveMQ that you are using.

Unfortunately I won't have time to try to reproduce this until the weekend.
But in the meantime would you please try the following to see if we can
narrow in on what aspect of the broker is the one using the memory?

1. Remove your Camel routes from your activemq.xml and restart, and see if
the steady-state heap utilization changes.
2. Change activemq.xml to point to an empty or non-existent data directory
and restart, and see if the steady-state heap utilization changes.
3. If neither of those show any reduction in heap usage, could you capture
a CPU sampling from JVisualVM as I described in my first post from
yesterday?

Thanks,
Tim

On Tue, Apr 14, 2020, 7:06 AM hfridhi <ha...@gmail.com> wrote:

> I agree with you the fact that we have to give the needed memory and do a
> good sizing in order to avoid such problems.
> However, my question is quite simple how it is possible that broker uses
> around 1,3G after 18 seconds knowing that there is no message in queues.
> <
> http://activemq.2283324.n4.nabble.com/file/t379890/VisualVM_ActiveMQ_Uptime_20200414.jpg>
>
> PLease let me know if you require any configuration file.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Broker not releasing memory after restart

[hfridhi <ha...@gmail.com>]

I agree with you the fact that we have to give the needed memory and do a
good sizing in order to avoid such problems.
However, my question is quite simple how it is possible that broker uses
around 1,3G after 18 seconds knowing that there is no message in queues.
<http://activemq.2283324.n4.nabble.com/file/t379890/VisualVM_ActiveMQ_Uptime_20200414.jpg> 
PLease let me know if you require any configuration file.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Broker not releasing memory after restart

[Tim Bain <tb...@alumni.duke.edu>]

Also, increasing the heap isn't necessarily a desperate measure for
desperate times. Rather, it's right-sizing the compute resources allocated
to the broker, so long as the heap usage is normal and expected.

From your question, clearly your initial assumption is that this heap usage
is not normal or expected. But there are definitely many people who have
found that the right-size heap utilization of their broker was well above 2
GB based on the load and usage patterns they subjected the broker to, so my
initial assumption is that this is normal/correct and that you need to
adjust your configuration (and your expectations) accordingly.

As you can see from the email I just wrote, clearly I'm happy to help you
explore whether perhaps there is a memory leak or some other problem
causing the memory usage, because it's always possible that there is a bug,
and if so then we're grateful for the help identifying it. But you also
need to understand that there is no expectation that a heavily-used broker
would never use more heap than an empty on straight out of the box, so
"increase the heap" may turn out to be the solution, not a hack.

Tim

On Tue, Apr 14, 2020, 5:40 AM Tim Bain <tb...@alumni.duke.edu> wrote:

> Do you have a significant number of unconsumed persistent messages?
> Especially on many queues/topics? That would produce an increase in the
> amount of RAM relative to an empty broker, because the broker will pull a
> certain number of the messages into memory to be ready to dispatch them to
> consumers without a disk read.
>
> Also, I've never used embedded Camel routes, so I'm not sure what to
> expect from them in terms of memory usage.
>
> One other technique you could use is to capture a CPU sample via JVisualVM
> during the time period when the memory usage is increasing, to determine
> which broker activities are likely causing the increase.
>
> Tim
>
> On Tue, Apr 14, 2020, 2:33 AM hfridhi <ha...@gmail.com> wrote:
>
>> Hello Tim,
>> Thank you for your quick reply.
>> I agree with that increasing JVM's heap is a desperate measures in
>> desperate
>> times but my question is why even after a restart memory is quickly
>> consumed
>> (a window of 2 minutes).
>> <
>> http://activemq.2283324.n4.nabble.com/file/t379890/VisualVM_ActiveMQ_20200414.jpg>
>>
>>
>> PS: yesterday I was passing around 508K message through the broker. But
>> why
>> is possible that after restarting, JVM's heap is reached within 2 minutes
>> without any activity?
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>

Re: Broker not releasing memory after restart

[Tim Bain <tb...@alumni.duke.edu>]

Do you have a significant number of unconsumed persistent messages?
Especially on many queues/topics? That would produce an increase in the
amount of RAM relative to an empty broker, because the broker will pull a
certain number of the messages into memory to be ready to dispatch them to
consumers without a disk read.

Also, I've never used embedded Camel routes, so I'm not sure what to expect
from them in terms of memory usage.

One other technique you could use is to capture a CPU sample via JVisualVM
during the time period when the memory usage is increasing, to determine
which broker activities are likely causing the increase.

Tim

On Tue, Apr 14, 2020, 2:33 AM hfridhi <ha...@gmail.com> wrote:

> Hello Tim,
> Thank you for your quick reply.
> I agree with that increasing JVM's heap is a desperate measures in
> desperate
> times but my question is why even after a restart memory is quickly
> consumed
> (a window of 2 minutes).
> <
> http://activemq.2283324.n4.nabble.com/file/t379890/VisualVM_ActiveMQ_20200414.jpg>
>
>
> PS: yesterday I was passing around 508K message through the broker. But why
> is possible that after restarting, JVM's heap is reached within 2 minutes
> without any activity?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

Re: Broker not releasing memory after restart

[hfridhi <ha...@gmail.com>]

Hello Tim,
Thank you for your quick reply.
I agree with that increasing JVM's heap is a desperate measures in desperate
times but my question is why even after a restart memory is quickly consumed
(a window of 2 minutes).
<http://activemq.2283324.n4.nabble.com/file/t379890/VisualVM_ActiveMQ_20200414.jpg> 

PS: yesterday I was passing around 508K message through the broker. But why
is possible that after restarting, JVM's heap is reached within 2 minutes
without any activity?



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Broker not releasing memory after restart

[Tim Bain <tb...@alumni.duke.edu>]

The simplest path forward is to increase your JVM's heap, say to 3GB or
4GB. As long as the broker's steady-state utilization (after triggering a
full GC) remains relatively constant over time (more precisely, the steady
state utilization doesn't rise consistently over time), then increasing the
heap isn't a problem.

The only concern would be if the steady-state utilization rises over time
(indicating a possible memory leak), in which case increasing the heap is
simply prolonging the inevitable and not solving the problem. But as long
as you're not seeing that afterwards, increasing the heap is a reasonable
solution.

Tim

On Mon, Apr 13, 2020, 4:29 PM hfridhi <ha...@gmail.com> wrote:

> Hello guys,
> I know it could be shocking problem but it happened to me and I'm confused
> too.
> I started a broker to which I'm intensively sending messages through WS and
> consuming them by a camel embedded route.
> At a certain time, I'm facing "gc overhead limit exceeded java".
> I tried to restart the broker and follow its memory using visualVM and I
> got
> this
> <
> http://activemq.2283324.n4.nabble.com/file/t379890/Screenshot_2020-04-14_at_00.png>
>
> This means that even without any call, memory is already at a high level
> and
> calling GC manually have no considerable effect.
>
> PS: Below is my Java configuration of AMQ
>
>
>
> Thank you for you help.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>