You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by GaryG <gg...@us.ibm.com> on 2007/02/23 17:47:55 UTC
Too many open files exception on broker
I'm seeing tons of exceptions on the broker:
============
2007-02-23 07:15:48,528 [xxsxxx:61616] ERROR TransportConnector
- Could not accept connection : java.net.SocketException: Too many open
files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:153)
at java.lang.Thread.run(Thread.java:595)
============
I can't connect to it via JMX anymore, although some messages still seem to
be getting
through.
I'm also seeing lots of errors like this on the broker:
=========
2007-02-23 07:15:43,523 [34.186.85:33862] DEBUG Service
- Error occured while processing sync command:
javax.jms.InvalidClientIDException: Broker: amqDev2 - Client: wf4 already
connected
javax.jms.InvalidClientIDException: Broker: amqDev2 - Client: wf4 already
connected
at
org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:205)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:82)
at
org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:70)
at
org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:82)
at
org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:92)
at
org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:687)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:86)
at
org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:121)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:284)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Thread.java:595)
==========
However, looking at the logs on the client mentioned above "wf4", I'll see
an error like this:
-------------
2007-02-23 14:30:01,332 [AcitveMQ Connection Worker:
tcp://xxxxxxxxxxxxxxxxx/xxxxxxxxxxx:61616] ERROR jms_comm -
ActiveMQFactoryUtil: exception reconnecting: javax.jms.JMSException: Wire
format negociation timeout: peer did not send his wire format.
-----------------------
but then right before and right after such errors, messages are being
received and sent with no problem.
These errors started happening after about 2 days of running the broker with
broker usageMemory being set to 256MB, and broker JVM has 512MB. I know
last I checked memory use for the broker, according to JMX, was growing
steadily. We're using non-persistent messages, but it seems memory is not
beeing freed up.
A co-worker here recommended upping the open file limit before I bring up
the Broker JVM with these commands:
----------
ulimit -u unlimited
ulimit -n 90000
ulimit -s unlimited
-----------
Which, I'll try shortly, but what I don't understand, is why would AMQ even
be in such a state in the first place?
Is there a bug with this too many open files issue? Is there another fix
that is recommended?
--
View this message in context: http://www.nabble.com/Too-many-open-files-exception-on-broker-tf3279888s2354.html#a9122263
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Too many open files exception on broker
Posted by James Strachan <ja...@gmail.com>.
So the main error you're getting is that your OS is running out of
file descriptors - which can happen anyway (each socket connected is a
file descriptor as is any attempt to open a file etc.
If sockets get disconnected, clients reconnect but the broker can lag
detecting the old socket disconnecting (due to OS limitations
detecting dead sockets), so its always good to have plenty
Note one or two issues have been found with ActiveMQ not always
cleaning up great on clients that just die rather than disconnect, so
you could try trunk if you are seeing some kind of leaks (we'll
hopefully have 4.1.1 soon with these patches included)
On 2/23/07, GaryG <gg...@us.ibm.com> wrote:
>
> I'm seeing tons of exceptions on the broker:
>
> ============
> 2007-02-23 07:15:48,528 [xxsxxx:61616] ERROR TransportConnector
> - Could not accept connection : java.net.SocketException: Too many open
> files
> java.net.SocketException: Too many open files
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> at java.net.ServerSocket.implAccept(ServerSocket.java:450)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:153)
> at java.lang.Thread.run(Thread.java:595)
> ============
>
> I can't connect to it via JMX anymore, although some messages still seem to
> be getting
> through.
>
> I'm also seeing lots of errors like this on the broker:
>
> =========
> 2007-02-23 07:15:43,523 [34.186.85:33862] DEBUG Service
> - Error occured while processing sync command:
> javax.jms.InvalidClientIDException: Broker: amqDev2 - Client: wf4 already
> connected
> javax.jms.InvalidClientIDException: Broker: amqDev2 - Client: wf4 already
> connected
> at
> org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:205)
> at
> org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:82)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:70)
> at
> org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:82)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:92)
> at
> org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:687)
> at
> org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:86)
> at
> org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:121)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:284)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:177)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
> at java.lang.Thread.run(Thread.java:595)
> ==========
>
> However, looking at the logs on the client mentioned above "wf4", I'll see
> an error like this:
> -------------
> 2007-02-23 14:30:01,332 [AcitveMQ Connection Worker:
> tcp://xxxxxxxxxxxxxxxxx/xxxxxxxxxxx:61616] ERROR jms_comm -
> ActiveMQFactoryUtil: exception reconnecting: javax.jms.JMSException: Wire
> format negociation timeout: peer did not send his wire format.
> -----------------------
>
> but then right before and right after such errors, messages are being
> received and sent with no problem.
>
> These errors started happening after about 2 days of running the broker with
> broker usageMemory being set to 256MB, and broker JVM has 512MB. I know
> last I checked memory use for the broker, according to JMX, was growing
> steadily. We're using non-persistent messages, but it seems memory is not
> beeing freed up.
>
> A co-worker here recommended upping the open file limit before I bring up
> the Broker JVM with these commands:
> ----------
> ulimit -u unlimited
> ulimit -n 90000
> ulimit -s unlimited
> -----------
>
> Which, I'll try shortly, but what I don't understand, is why would AMQ even
> be in such a state in the first place?
>
> Is there a bug with this too many open files issue? Is there another fix
> that is recommended?
>
> --
> View this message in context: http://www.nabble.com/Too-many-open-files-exception-on-broker-tf3279888s2354.html#a9122263
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Too many open files exception on broker
Posted by "Christopher G. Stach II" <cg...@ldsys.net>.
GaryG wrote:
> A co-worker here recommended upping the open file limit before I bring up
> the Broker JVM with these commands:
> ----------
> ulimit -u unlimited
> ulimit -n 90000
> ulimit -s unlimited
> -----------
>
> Which, I'll try shortly, but what I don't understand, is why would AMQ even
> be in such a state in the first place?
>
> Is there a bug with this too many open files issue? Is there another fix
> that is recommended?
>
It's normal *nix resource limits. Just like the nohup thing, this
really doesn't have anything to do with AMQ specifically.
--
Christopher G. Stach II