You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ashish <as...@gmail.com> on 2011/09/21 21:44:18 UTC
ActiveMQ - 5.3.2 - connection taking too long to shutdown
*We are running ActiveMQ in JDBC master slave mode. A few days back we
observed the following log entries in the master log files:*
The connection to '/<ip>:<port>' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Task
.
.
.
ERROR | Could not accept connection : java.net.SocketException: Broken pipe
| org.apache.activemq.broker.TransportConnector | ActiveMQ Task
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at
org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
at
org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:85)
at
org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:914)
at
org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:230)
at
org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:190)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:254)
at
org.apache.activemq.transport.InactivityMonitor$4.run(InactivityMonitor.java:174)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
*some time after this exception was printed the master activemq went down
and slave became the master.
We then observed the following warnings/errors in the new masters log files.
These log entries started getting generated 10 minutes after the slave
became the master:
*
INFO | The connection to '/<ip>:<port>' is taking a long time to shutdown.
| org.apache.activemq.broker.TransportConnection | ActiveMQ Task
ERROR | Could not accept connection : java.net.SocketException: Broken pipe
| org.apache.activemq.broker.TransportConnector | ActiveMQ Task
WARN | cannot create async transport stopper thread.. not waiting for stop
to complete, reason: | org.apache.activemq.broker.TransportConnection |
ActiveMQ Transport: tcp:///<ip>:<port>
java.lang.OutOfMemoryError: GC overhead limit exceeded
*After a day or two after the slave became the master we observed the
following warnings in the current masters log files:*
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
| org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
WARN | Failed to unregister mbean:
org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
| org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
| org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
WARN | Failed to register MBean:
org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
| org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
WARN | Failed to unregister mbean:
org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
| org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
*What could be the reason for the occurrence of these warnings/errors?*
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-3-2-connection-taking-too-long-to-shutdown-tp3831196p3831196.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ - 5.3.2 - connection taking too long to shutdown
Posted by Torsten Mielke <to...@fusesource.com>.
Your broker is running out of JVM heap memory. From that moment on, anything can happen.
I suggest you try to figure out the reason for running out of memory in the first place.
Do you impose memory limits on each destination?
Do you configure for a system memory usage in your broker that is well under the JVM heap memory limit?
Also, 5.3.2 is rather old and hundreds of bugs have been fixed since (including a couple of memory leaks).
If possible, you might want to consider upgrading to the latest 5.5 version.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Sep 21, 2011, at 9:44 PM, ashish wrote:
> *We are running ActiveMQ in JDBC master slave mode. A few days back we
> observed the following log entries in the master log files:*
>
> The connection to '/<ip>:<port>' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Task
> .
> .
> .
>
> ERROR | Could not accept connection : java.net.SocketException: Broken pipe
> | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
>
>
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Unknown Source)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:85)
> at
> org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:914)
> at
> org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:230)
> at
> org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:190)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
> at
> org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:254)
> at
> org.apache.activemq.transport.InactivityMonitor$4.run(InactivityMonitor.java:174)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>
>
> *some time after this exception was printed the master activemq went down
> and slave became the master.
>
> We then observed the following warnings/errors in the new masters log files.
> These log entries started getting generated 10 minutes after the slave
> became the master:
>
> *
> INFO | The connection to '/<ip>:<port>' is taking a long time to shutdown.
> | org.apache.activemq.broker.TransportConnection | ActiveMQ Task
> ERROR | Could not accept connection : java.net.SocketException: Broken pipe
> | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> WARN | cannot create async transport stopper thread.. not waiting for stop
> to complete, reason: | org.apache.activemq.broker.TransportConnection |
> ActiveMQ Transport: tcp:///<ip>:<port>
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
>
> *After a day or two after the slave became the master we observed the
> following warnings in the current masters log files:*
>
> WARN | Failed to register MBean:
> org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
> | org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
> WARN | Failed to unregister mbean:
> org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
> | org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
> WARN | Failed to register MBean:
> org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
> | org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
> WARN | Failed to register MBean:
> org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
> | org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
> WARN | Failed to unregister mbean:
> org.apache.activemq:BrokerName=slave,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/<ip>_<port>
> | org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ Task
>
>
>
> *What could be the reason for the occurrence of these warnings/errors?*
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-3-2-connection-taking-too-long-to-shutdown-tp3831196p3831196.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.