You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ealiass <al...@ericsson.com> on 2016/02/10 15:21:52 UTC

java.lang.OutOfMemoryError: GC overhead limit exceeded

Hi All,

I am having java.lang.OutOfMemoryError: GC overhead limit exceeded Error
after a while from starting the ActiveMQ on my system (2-3 weeks) start
seeing this issue on production.
 i had to restart the my ActiveMQ every while on production so i might lose
some messages coming in the time while the queue is restarting.

we are using the version:5.11.1
we have 1Gb of memory for the queue on production
there was similar issue on the forum and the answer was to increase the
memory to 2Gb, but this is not a real solution since we only delaying the
issue from happening   

find below the logs i got 

org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//172.16.19.0_40176\n","stream":"stdout","time":"2016-01-18T10:40:55.455379802Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52528 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:40:58.87142185Z"}
{"log":"ERROR | Could not accept connection : java.lang.Exception:
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stdout","time":"2016-01-18T10:41:00.229514014Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40222 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.994358557Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40195 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.99828402Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40193 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:06.424638043Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40374 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:09.107570289Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34082 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:11.89354178Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34072 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:13.548763654Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52575 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.788697766Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52628 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.795035469Z"}
{"log":"ERROR | Could not accept connection : java.lang.Exception:
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stdout","time":"2016-01-18T10:41:17.821256534Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40341 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:17.824982456Z"}
{"log":"Exception in thread \"ActiveMQ InactivityMonitor Worker\"
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stderr","time":"2016-01-18T10:41:22.453110848Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34040 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:22.459674904Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40300 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:26.945482864Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52777 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:30.77306023Z"}
{"log":"Exception in thread \"ActiveMQ BrokerService[localhost] Task-72110\"
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stderr","time":"2016-01-18T10:41:33.768910175Z"}
{"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stderr","time":"2016-01-18T10:41:34.812406659Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40283 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:37.718970346Z"}
{"log":" INFO | KahaDB: Recovering checkpoint thread after
death\n","stream":"stdout","time":"2016-01-18T10:41:39.163309356Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40528 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.018391792Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34235 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.019448416Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40485 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:45.595817225Z"}
{"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stderr","time":"2016-01-18T10:41:47.076826925Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34264 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.04356476Z"}
{"log":"Exception in thread \"ActiveMQ Transport Server:
stomp://0.0.0.0:61613?maximumConnections=1000\u0026wireFormat.maxFrameSize=104857600\"
java.lang.OutOfMemoryError: GC overhead limit
exceeded\n","stream":"stderr","time":"2016-01-18T10:41:50.047204344Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52760 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.049358006Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40487 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:51.04876469Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40453 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.941294862Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34209 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.943995696Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34253 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.257523013Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34206 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.25780067Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52713 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.147528061Z"}
{"log":"Exception in thread \"ActiveMQ Transport:
tcp:///172.16.90.0:34152@61616\" java.lang.OutOfMemoryError: GC overhead
limit exceeded\n","stream":"stderr","time":"2016-01-18T10:41:57.14816742Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52717 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.148887172Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40437 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.59718311Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.90.0:34176 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.597488872Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52671 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.598515011Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52693 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.585086858Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40433 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.586107233Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52633 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.988706309Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.19.0:40376 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.992193223Z"}
{"log":" INFO | KahaDB: Recovering checkpoint thread after
death\n","stream":"stdout","time":"2016-01-18T10:42:04.079519366Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52821 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.08123258Z"}
{"log":" WARN | Transport Connection to: tcp://172.16.33.0:52859 failed:
java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.084563477Z"}
{"log":" WARN | Failed to register MBean 

if anyone one have had the same issue or have any hint to guide me please
help would appreciate it :)

Thanks,




--
View this message in context: http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by HalfBloodPrince <Ab...@confluxsys.com>.
Hi @Tim Bain
Can you please tell me how to enable the PFC in ActiveMQ?
Can we pass some VM arguments to enable it? If Yes then which file should
have those VM arguments.?

Thank You 



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by HalfBloodPrince <Ab...@confluxsys.com>.
Hi Everyone,

Can anyone please help me to give some solution on below error?
I am getting this error after every 2-3 Hours. 
My activeMQ configuration is as follow.

ActiveMQ configuration is as follow:
 ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY -Xms1g -Xmx1g"

and activemq.xml configuration as follow:
<policyEntry queue=">" producerFlowControl="true" memoryLimit="180mb">    
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>


<systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="50" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
</systemUsage>

Tomcat Server Configration JVM Arguments:
-XX:+UseParallelOldGC -Xms3g -Xmx3g -XX:NewRatio=3 -XX:PermSize=256m
-XX:MaxPermSize=1024m

Exception in thread "ajp-nio-8009-ClientPoller-1" Exception in thread
"http-nio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "logback-1" java.lang.OutOfMemoryError: GC overhead
limit exceeded
05-Jun-2018 07:45:09.213 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run
Unexpected death of background thread
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
 java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.util.zip.ZipCoder.getBytes(ZipCoder.java:89)
        at java.util.zip.ZipFile.getEntry(ZipFile.java:316)
        at java.util.jar.JarFile.getEntry(JarFile.java:240)
        at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
        at
org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:98)
        at
org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
        at
org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
        at org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
        at
org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216)
        at
org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
        at
org.apache.catalina.loader.WebappClassLoaderBase.modified(WebappClassLoaderBase.java:691)
        at
org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:343)
        at
org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:287)
        at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5447)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1396)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1368)
        at java.lang.Thread.run(Thread.java:748)

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at java.util.zip.ZipCoder.getBytes(ZipCoder.java:89)
        at java.util.zip.ZipFile.getEntry(ZipFile.java:316)
        at java.util.jar.JarFile.getEntry(JarFile.java:240)
        at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
        at
org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntry(AbstractSingleArchiveResourceSet.java:98)
        at
org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:256)
        at
org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
        at org.apache.catalina.webresources.Cache.getResource(Cache.java:62)
        at
org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:216)
        at
org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
        at
org.apache.catalina.loader.WebappClassLoaderBase.modified(WebappClassLoaderBase.java:691)
        at
org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:343)
        at
org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:287)
        at
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5447)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1396)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1368)
        at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at
java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)
        at
org.apache.tomcat.util.net.NioEndpoint$Poller.timeout(NioEndpoint.java:933)
        at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:753)
        at java.lang.Thread.run(Thread.java:748)
Exception in thread "logback-1" java.lang.OutOfMemoryError: GC overhead
limit exceeded
05-Jun-2018 07:48:35.867 WARNING [ActiveMQ Connection Executor:
tcp://localhost/127.0.0.1:61616@123456]
org.springframework.jms.connection.SingleConnectionFactory.onException
Encountered a JMSException - resetting the underlying JMS Connection
 javax.jms.JMSException: java.io.EOFException
        at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
        at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1952)
        at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1971)
        at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
        at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
        at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
        at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
        at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:173)
        at
org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:345)
        at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:219)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
        at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
        ... 1 more


Can anyone please help me out from this error? I am getting this error after
every 1-2 Hr on server. Please let me know if I am missing something.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by Tim Bain <tb...@alumni.duke.edu>.
On Thu, Mar 15, 2018 at 1:45 AM, sathish.create@gmail.com <
sathish.create@gmail.com> wrote:

> Thanks a lot for your reply Tim,
>
> Below is the metrics details i grabbed from the server.
> ActiveMQ:
>         5.15.0
>         non-persistent
>         Running consumer -> singleton -> onMessage() -> submit callables ->
> callables send updated message to destination
>         messages -> 600/hour
>         messages size -> 8 byte
>         messages consume interval -> listener pools continuously as soon
> as the
> message arrives
> JVM:
>         -Xms1g
>         -Xmx3g
> Application:
>         -Xms4g
>         -Xmx4g
> GCTYPE:
>         ParallelGC
>
> Its only consumer gives issues in Broker service side.
>
> 1. Yes. i do have to see how to enable PFC. But just a thought since its
> only a consumer application do we need to to enable PFC? if so how do i
> enable for consumer client side using programming?
>

Your original message in this thread said " I am having
java.lang.OutOfMemoryError: GC overhead limit exceeded Error after a while
from starting the ActiveMQ." Are you telling me that that sentence referred
to a client rather than the broker?


> 2. Also seeing below warning and does that makes sense even if am using non
> persistence messages?
> WARN  BrokerService:2139 - Temporary Store limit is 51200 mb (current store
> usage is 0 mb). The data directory: /temp only has 2000 mb of usable space.
> - resetting to maximum available disk space: 2000 mb
>

Temp store is used to overflow to disk when the memory store gets full, so
yes, it's relevant when using non-persistent messages. In fact, it's *only*
relevant when using non-persistent messages. So that error message means
that you can only have 2 GB more non-persistent messages than whatever fits
in the memory store.


> 3. The consumer design is like below, am i doing something wrong in the
> design?
> Consumer Main -> inject Consumer (init->activeMQ producer:consumer,
> onMessage->call workerThread(consumer) ) -> workerThread (send message to
> consumer, producer)
>

I have no idea what you're trying to describe here. Maybe take another shot
at it, using sentences with enough words to convey unambiguous meaning,
instead of arrow diagrams...


> Expecting your valuable comments on this.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by "sathish.create@gmail.com" <sa...@gmail.com>.
Thanks a lot for your reply Tim,

Below is the metrics details i grabbed from the server.
ActiveMQ:
	5.15.0
	non-persistent
	Running consumer -> singleton -> onMessage() -> submit callables ->
callables send updated message to destination
	messages -> 600/hour
	messages size -> 8 byte
	messages consume interval -> listener pools continuously as soon as the
message arrives 
JVM:
	-Xms1g
	-Xmx3g
Application:
	-Xms4g
	-Xmx4g
GCTYPE:	
	ParallelGC 

Its only consumer gives issues in Broker service side.

1. Yes. i do have to see how to enable PFC. But just a thought since its
only a consumer application do we need to to enable PFC? if so how do i
enable for consumer client side using programming?

2. Also seeing below warning and does that makes sense even if am using non
persistence messages?
WARN  BrokerService:2139 - Temporary Store limit is 51200 mb (current store
usage is 0 mb). The data directory: /temp only has 2000 mb of usable space.
- resetting to maximum available disk space: 2000 mb

3. The consumer design is like below, am i doing something wrong in the
design?
Consumer Main -> inject Consumer (init->activeMQ producer:consumer,
onMessage->call workerThread(consumer) ) -> workerThread (send message to
consumer, producer)

Expecting your valuable comments on this.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by Tim Bain <tb...@alumni.duke.edu>.
1. http://activemq.apache.org/producer-flow-control.html, and yes, you
should enable it. You should protect the broker (by preventing messages
from over-filling its heap) first and foremost, and PFC does that.
2. When you're out of memory, it tends to manifest itself all over the
place. There's no direct causal relationship between the specific calls
that fail and the running out of memory; all those calls tell you is that
they're using memory, not that they're using an unreasonable amount of it
or failing to free it.

Things that are relevant in any OOM situation: what version of ActiveMQ,
what max heap size are you using, what GC strategy are you using, and
what's the usage pattern for your broker (how many messages/second, how
many queues/topics, persistent or non-persistent messages, how big are the
messages, how long do the messages sit on the broker before being consumed,
etc.)?

I'd start with enabling PFC, and allocating more heap to the broker process
if that's an option, and then I'd spend time characterizing what the
expected worst-case memory usage is based on how many messages you expect
to have in broker memory (non-persistent messages, plus persistent messages
paged in for dispatch to consumers, see
http://activemq.apache.org/message-cursors.html for an explanation) in the
worst-case scenario you need to support. It's possible that you simply
don't have the heap sized large enough to support the use-case you're
trying to perform.

Tim

On Wed, Mar 7, 2018 at 2:06 AM, sathish.create@gmail.com <
sathish.create@gmail.com> wrote:

> Hi artnaseef,
>
> i am facing same issue in my startup log after 5 days, similar to this,
>
> Exception in thread "ActiveMQ InactivityMonitor Worker" Exception in thread
> "pool-3-thread-121233" Exception in thread "pool-1-thread-1" Exception in
> thread "pool-3-thread-432434" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> Let me know if below options will help,
> 1. should i enable producer flow control?  if so how do i enable it?
> 2. getting this exception also along with this ,
>
> Exception in thread "ActiveMQ Transport Server:
> ssl://ip:port?socket.needClientAuth=true" java.lang.OutOfMemoryError: GC
> overhead limit exceeded
>  at sun.security.ssl.InputRecord.<init>(InputRecord.java:93)
>         at sun.security.ssl.AppInputStream.<init>(AppInputStream.java:50)
>         at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:640)
>         at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:524)
>         at
> sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:343)
>         at
> org.apache.activemq.transport.tcp.TcpTransportServer.
> doRunWithServerSocket(TcpTransportServer.java:403)
>         at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(
> TcpTransportServer.java:325)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by "sathish.create@gmail.com" <sa...@gmail.com>.
Hi artnaseef,

i am facing same issue in my startup log after 5 days, similar to this,

Exception in thread "ActiveMQ InactivityMonitor Worker" Exception in thread
"pool-3-thread-121233" Exception in thread "pool-1-thread-1" Exception in
thread "pool-3-thread-432434" java.lang.OutOfMemoryError: GC overhead limit
exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded

Let me know if below options will help,
1. should i enable producer flow control?  if so how do i enable it?
2. getting this exception also along with this , 

Exception in thread "ActiveMQ Transport Server:
ssl://ip:port?socket.needClientAuth=true" java.lang.OutOfMemoryError: GC
overhead limit exceeded
 at sun.security.ssl.InputRecord.<init>(InputRecord.java:93)
        at sun.security.ssl.AppInputStream.<init>(AppInputStream.java:50)
        at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:640)
        at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:524)
        at
sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:343)
        at
org.apache.activemq.transport.tcp.TcpTransportServer.doRunWithServerSocket(TcpTransportServer.java:403)
        at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:325)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by artnaseef <ar...@artnaseef.com>.
Check first for slow consumption.

Is producer-flow-control enabled on the broker?  And, if it is, what is the
per-queue memory limit and the overall system limit.

The most common cause of Out Of Memory on ActiveMQ is a broker without
producer-flow-control and slow consumption, which mimics a memory leak.



--
View this message in context: http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387p4707909.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by ealiass <al...@ericsson.com>.
i am using ParallelGC (the default garbage collector)

here is some information :

XX:InitialHeapSize=262927424 -XX:MaxHeapSize=4206838784
-XX:+PrintCommandLineFlags -XX:+UseCompressedOops -XX:+UseParallelGC





--
View this message in context: http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387p4707519.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Posted by Tim Bain <tb...@alumni.duke.edu>.
First, for a question about GC problems, you should probably tell us what
GC strategy you're using, and why you chose it.

Second, you made the claim that "to increase the memory to 2Gb ... only
delaying the issue from happening," which implies that you believe you're
running out of memory because of a memory leak rather than because you
sized your JVM too small.  Do you have any evidence of such a leak or that
you've properly sized your JVM and run out of memory, or are you just
making an assumption?  I've spent a couple months performance testing the
5.8.0 broker's memory usage with non-persistent messages that are consumed
immediately, and I can tell you for certain that there's no memory leak in
the code paths executed by that specific scenario (in that version).  It's
always possible that there's a leak in some other code that my test wasn't
executing, but even that is somewhat unlikely given that no one but you has
complained about it.  So although it's possible that there's a leak, I
think it's more likely that either you have more live objects than will fit
into the amount of heap you've given to your JVM (in which case increasing
the heap is the solution, and a real solution), or you're using the CMS GC
strategy and you're hitting the Old Gen fragmentation problem.

Tim
On Feb 10, 2016 10:52 AM, "ealiass" <al...@ericsson.com> wrote:

> Hi All,
>
> I am having java.lang.OutOfMemoryError: GC overhead limit exceeded Error
> after a while from starting the ActiveMQ on my system (2-3 weeks) start
> seeing this issue on production.
>  i had to restart the my ActiveMQ every while on production so i might lose
> some messages coming in the time while the queue is restarting.
>
> we are using the version:5.11.1
> we have 1Gb of memory for the queue on production
> there was similar issue on the forum and the answer was to increase the
> memory to 2Gb, but this is not a real solution since we only delaying the
> issue from happening
>
> find below the logs i got
>
>
> org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=openwire,connectionViewType=remoteAddress,connectionName=tcp_//172.16.19.0_40176\n","stream":"stdout","time":"2016-01-18T10:40:55.455379802Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52528 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:40:58.87142185Z"}
> {"log":"ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stdout","time":"2016-01-18T10:41:00.229514014Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40222 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.994358557Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40195 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:01.99828402Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40193 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:06.424638043Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40374 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:09.107570289Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34082 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:11.89354178Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34072 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:13.548763654Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52575 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.788697766Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52628 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:15.795035469Z"}
> {"log":"ERROR | Could not accept connection : java.lang.Exception:
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stdout","time":"2016-01-18T10:41:17.821256534Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40341 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:17.824982456Z"}
> {"log":"Exception in thread \"ActiveMQ InactivityMonitor Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:22.453110848Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34040 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:22.459674904Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40300 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:26.945482864Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52777 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:30.77306023Z"}
> {"log":"Exception in thread \"ActiveMQ BrokerService[localhost]
> Task-72110\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:33.768910175Z"}
> {"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:34.812406659Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40283 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:37.718970346Z"}
> {"log":" INFO | KahaDB: Recovering checkpoint thread after
> death\n","stream":"stdout","time":"2016-01-18T10:41:39.163309356Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40528 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.018391792Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34235 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:42.019448416Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40485 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:45.595817225Z"}
> {"log":"Exception in thread \"ActiveMQ Journal Checkpoint Worker\"
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:47.076826925Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34264 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.04356476Z"}
> {"log":"Exception in thread \"ActiveMQ Transport Server:
> stomp://
> 0.0.0.0:61613?maximumConnections=1000\u0026wireFormat.maxFrameSize=104857600\
> "
> java.lang.OutOfMemoryError: GC overhead limit
> exceeded\n","stream":"stderr","time":"2016-01-18T10:41:50.047204344Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52760 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:50.049358006Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40487 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:51.04876469Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40453 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.941294862Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34209 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:52.943995696Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34253 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.257523013Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34206 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:55.25780067Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52713 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.147528061Z"}
> {"log":"Exception in thread \"ActiveMQ Transport:
> tcp:///172.16.90.0:34152@61616\" java.lang.OutOfMemoryError: GC overhead
> limit exceeded\n","stream":"stderr","time":"2016-01-18T10:41:57.14816742Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52717 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:57.148887172Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40437 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.59718311Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.90.0:34176 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.597488872Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52671 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:41:58.598515011Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52693 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.585086858Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40433 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:00.586107233Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52633 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.988706309Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.19.0:40376 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:02.992193223Z"}
> {"log":" INFO | KahaDB: Recovering checkpoint thread after
> death\n","stream":"stdout","time":"2016-01-18T10:42:04.079519366Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52821 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.08123258Z"}
> {"log":" WARN | Transport Connection to: tcp://172.16.33.0:52859 failed:
>
> java.io.EOFException\n","stream":"stdout","time":"2016-01-18T10:42:04.084563477Z"}
> {"log":" WARN | Failed to register MBean
>
> if anyone one have had the same issue or have any hint to guide me please
> help would appreciate it :)
>
> Thanks,
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/java-lang-OutOfMemoryError-GC-overhead-limit-exceeded-tp4707387.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>