You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by sainath <sk...@ra.rockwell.com.INVALID> on 2019/11/25 08:46:44 UTC
OutOfMemoryError in activeMQ and applcation gets stopped
We have LiveTransfer application that use ActiveMQ to transfer data from
Production database to operational Database
Application is getting issue for every 50+ days i.e. OutOfMemoryError in
activeMQ and LT gets stopped. After restarting the AMQ services, it will
resume the transfer.
----
INFO | jvm 1 | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
heap space
INFO | jvm 1 | 2019/09/06 00:56:34 | Dumping heap to
D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
INFO | jvm 1 | 2019/09/06 00:56:58 | Heap dump file created [4259842078
bytes in 23.626 secs]
----
Due to this issue we found data missing between Production database to
operational Database.
Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
using JConsole and found memory is consuming more than i.e. 2.8 GB
This system is only dedicated for AMQ and Live transfer
AactiveMQ version-5.14.1
JDK version : 1.8 Update 60
Could you please help to provide the root cause and fix on the issue.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by sainath <sk...@ra.rockwell.com.INVALID>.
We have tried with java Update 181 and it still ran out of memory. Could you
please help us to provide some suggestion on this.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi,
What ActiveMQ did you test ?
Regards
JB
> Le 2 juin 2020 à 17:26, sainath <sk...@ra.rockwell.com.INVALID> a écrit :
>
> We tried just JMX which made a big difference but it seems that advisories
> still had some effect and decided to be safe and disabled both. If someone
> is able to read heap dumps we would be more than happy to collect them to
> verify what if any objects are being retained.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by sainath <sk...@ra.rockwell.com.INVALID>.
We tried just JMX which made a big difference but it seems that advisories
still had some effect and decided to be safe and disabled both. If someone
is able to read heap dumps we would be more than happy to collect them to
verify what if any objects are being retained.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by Tim Bain <tb...@alumni.duke.edu>.
Does disabling one but not both of those things also fix the problem? Or is
the problem only eliminated if both are disabled?
Tim
On Mon, Jun 1, 2020, 7:44 AM sainath <sk...@ra.rockwell.com.invalid>
wrote:
> Hi All,
>
> We disabled JMX and advisories, this seems to fix the memory issue.
>
> In the ActiveMQ.xml file we
>
> Added these flags useJmx="false" and advisorySupport="false"
>
> eg:
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="false"
> advisorySupport="false">
>
> For any clients we added jms.watchTopicAdvisories=false
>
> eg:
>
> loaderBrokerURL=failover\:(tcp\://localhost\:61616)?jms.watchTopicAdvisories=false
>
> Has anyone had to do this or understand why this would help? Is there a
> bug
> with these components that’s causing a memory leak?
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by sainath <sk...@ra.rockwell.com.INVALID>.
Hi All,
We disabled JMX and advisories, this seems to fix the memory issue.
In the ActiveMQ.xml file we
Added these flags useJmx="false" and advisorySupport="false"
eg:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}" useJmx="false"
advisorySupport="false">
For any clients we added jms.watchTopicAdvisories=false
eg:
loaderBrokerURL=failover\:(tcp\://localhost\:61616)?jms.watchTopicAdvisories=false
Has anyone had to do this or understand why this would help? Is there a bug
with these components that’s causing a memory leak?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by Tim Bain <tb...@alumni.duke.edu>.
Are you by any chance using CMS as your GC strategy? It's got a nasty
shortcoming where because it doesn't compact Old Gen, it's possible to have
heap fragmentation that means it's not possible to allocate contiguous
space for a large object even though there's still space free. If so, you
should consider moving to G1, though if you were going to switch to it I'd
recommend you upgrade to a recent Java 8 JRE. There were some G1 bugs
addressed over the course of development of Java 8, and I'm not sure that
Update 60 would be recent enough to have all of them.
Ultimately, though, any shortcomings of the garbage collector as space gets
tight might just be hastening a problem you'd have hit anyway after a
little more time. So Chris's questions about exactly what the memory was
being used for remain relevant.
Tim
On Tue, Nov 26, 2019 at 8:49 PM sainath <sk...@ra.rockwell.com.invalid>
wrote:
> Hi All,
>
> Yes, we have configured the limits on memory usage in the broker settings.
>
> We also checked the heap dump and found that still 500 MB free space
> available. So what could you be the reason for OutOfMemoryError in
> activeMQ
>
> Also we are monitoring the AMQ queues and there is no pending message.
>
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by sainath <sk...@ra.rockwell.com.INVALID>.
Hi All,
Yes, we have configured the limits on memory usage in the broker settings.
We also checked the heap dump and found that still 500 MB free space
available. So what could you be the reason for OutOfMemoryError in activeMQ
Also we are monitoring the AMQ queues and there is no pending message.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by Tim Bain <tb...@alumni.duke.edu>.
Are your consumers keeping up with your producers? One easy way to run out
of memory is to produce messages faster than you consume them.
Tim
On Mon, Nov 25, 2019, 11:16 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Have you configured limits on memory usage in the broker settings? You need
> to look at a heap dump and see exactly where the memory is being used (if
> it's messages on a destination, in prefetch etc).
>
> Some general info about OOM and how to prevent it:
> https://activemq.apache.org/javalangoutofmemory.html
>
> On Mon, Nov 25, 2019 at 10:38 AM sainath <sk...@ra.rockwell.com.invalid>
> wrote:
>
> > We have LiveTransfer application that use ActiveMQ to transfer data from
> > Production database to operational Database
> >
> > Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> > activeMQ and LT gets stopped. After restarting the AMQ services, it will
> > resume the transfer.
> >
> > ----
> > INFO | jvm 1 | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> > heap space
> > INFO | jvm 1 | 2019/09/06 00:56:34 | Dumping heap to
> > D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> > INFO | jvm 1 | 2019/09/06 00:56:58 | Heap dump file created
> [4259842078
> > bytes in 23.626 secs]
> > ----
> >
> > Due to this issue we found data missing between Production database to
> > operational Database.
> >
> > Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> > using JConsole and found memory is consuming more than i.e. 2.8 GB
> >
> > This system is only dedicated for AMQ and Live transfer
> >
> > AactiveMQ version-5.14.1
> >
> > JDK version : 1.8 Update 60
> >
> > Could you please help to provide the root cause and fix on the issue.
> >
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Re: OutOfMemoryError in activeMQ and applcation gets stopped
Posted by Christopher Shannon <ch...@gmail.com>.
Have you configured limits on memory usage in the broker settings? You need
to look at a heap dump and see exactly where the memory is being used (if
it's messages on a destination, in prefetch etc).
Some general info about OOM and how to prevent it:
https://activemq.apache.org/javalangoutofmemory.html
On Mon, Nov 25, 2019 at 10:38 AM sainath <sk...@ra.rockwell.com.invalid>
wrote:
> We have LiveTransfer application that use ActiveMQ to transfer data from
> Production database to operational Database
>
> Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> activeMQ and LT gets stopped. After restarting the AMQ services, it will
> resume the transfer.
>
> ----
> INFO | jvm 1 | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> heap space
> INFO | jvm 1 | 2019/09/06 00:56:34 | Dumping heap to
> D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> INFO | jvm 1 | 2019/09/06 00:56:58 | Heap dump file created [4259842078
> bytes in 23.626 secs]
> ----
>
> Due to this issue we found data missing between Production database to
> operational Database.
>
> Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> using JConsole and found memory is consuming more than i.e. 2.8 GB
>
> This system is only dedicated for AMQ and Live transfer
>
> AactiveMQ version-5.14.1
>
> JDK version : 1.8 Update 60
>
> Could you please help to provide the root cause and fix on the issue.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>