You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Craig Blake (JIRA)" <ji...@apache.org> on 2008/06/28 04:33:00 UTC
[jira] Created: (AMQ-1829) OOM under minimal load
OOM under minimal load
----------------------
Key: AMQ-1829
URL: https://issues.apache.org/activemq/browse/AMQ-1829
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.1.0
Environment: Grails w/ JMS Plugin, JDK 1.6, OSX Leopard
Reporter: Craig Blake
Priority: Critical
Under load of 70 messages per minute, ActiveMQ fails with an OutOfMemory exception after 1.5 hours. VM information available via JMX indicates that over 2,500 live threads exist at time of failure, with 116MB of memory committed out of 490MB available.
The netstat command shows several thousand connections in CLOSE_WAIT status at the time of failure.
WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=65276] throws
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:615)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:384)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:637)
Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:615)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
at org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
The application sees the following error:
javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1183)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1262)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:292)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
... 56 more
Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:85)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:70)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172)
... 60 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (AMQ-1829) OOM under minimal load
Posted by "Craig Blake (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Blake closed AMQ-1829.
----------------------------
Resolution: Cannot Reproduce
I found out that this problem was caused by another piece of software not properly closing connections to the JMX server.
> OOM under minimal load
> ----------------------
>
> Key: AMQ-1829
> URL: https://issues.apache.org/activemq/browse/AMQ-1829
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Grails w/ JMS Plugin, JDK 1.6, OSX Leopard
> Reporter: Craig Blake
> Priority: Critical
>
> Under load of 70 messages per minute, ActiveMQ fails with an OutOfMemory exception after 1.5 hours. VM information available via JMX indicates that over 2,500 live threads exist at time of failure, with 116MB of memory committed out of 490MB available.
> The netstat command shows several thousand connections in CLOSE_WAIT status at the time of failure.
> WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=65276] throws
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:615)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:384)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:637)
> Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:615)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> at org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> The application sees the following error:
> javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1183)
> at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1262)
> at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:292)
> at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> ... 56 more
> Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
> at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:85)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:70)
> at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172)
> ... 60 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1829) OOM under minimal load
Posted by "Craig Blake (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=43770#action_43770 ]
Craig Blake commented on AMQ-1829:
----------------------------------
After looking through the JMX console a bit, I found that most of the threads that build up and eventually crash the server look like this:
Name: JMX server connection timeout 7459
State: TIMED_WAITING on [I@6fd6306c
Total blocked: 0 Total waited: 4
Stack trace:
java.lang.Object.wait(Native Method)
com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
java.lang.Thread.run(Thread.java:637)
This appears to be similar to issue AMQ-1483 which was supposed to be fixed in ActiveMQ 5.1.0
> OOM under minimal load
> ----------------------
>
> Key: AMQ-1829
> URL: https://issues.apache.org/activemq/browse/AMQ-1829
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Grails w/ JMS Plugin, JDK 1.6, OSX Leopard
> Reporter: Craig Blake
> Priority: Critical
>
> Under load of 70 messages per minute, ActiveMQ fails with an OutOfMemory exception after 1.5 hours. VM information available via JMX indicates that over 2,500 live threads exist at time of failure, with 116MB of memory committed out of 490MB available.
> The netstat command shows several thousand connections in CLOSE_WAIT status at the time of failure.
> WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=65276] throws
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:615)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:384)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:637)
> Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:615)
> at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128)
> at org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103)
> at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> The application sees the following error:
> javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
> at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1183)
> at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1262)
> at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:292)
> at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
> at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:450)
> ... 56 more
> Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
> at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:85)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:70)
> at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
> at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172)
> ... 60 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.