ActiveMQ leaks file descriptors

[Johan Carlquist <jo...@su.se>]

We have two brokers configured in a network of brokers.
      <networkConnector
        networkTTL="-1"
        conduitSubscriptions="false"
        decreaseNetworkConsumerPriority="true"
        uri="static:(ssl://${activemq.networkPartner}:61616)"
        userName="${activemq.username}"
        password="${activemq.password}"
        />
    </networkConnectors>

Both stores the messages on local disk using KahaDB.
    <persistenceAdapter>
        <kahaDB directory="${activemq.data}/kahadb"/>
    </persistenceAdapter>

And they both run ActiveMQ-5.13.4 in java 8u45.

Clients are allowed to connect through openwires and amqps:
    <transportConnectors>
      <transportConnector name="nio+ssl" uri="nio+ssl://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
      <transportConnector name="amqp+ssl" uri="amqp+nio+ssl://0.0.0.0:5671?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
    </transportConnectors>

We have about 10 queues and 1 topic configured. The major part of these
messages are sent by Nagios that frequently checks that everything is
fine by sending and receiving messages on all accepted protocols.

Since we setup this configuration, ActiveMQ has leaked file descriptors
and we haven't been able to figure out the cause.

According to `lsof` there is a lot of pipes and anon_inodes open:
[...]
java    33212 root 6462w  FIFO                0,8      0t0 76388774 pipe
java    33212 root 6463r  FIFO                0,8      0t0 76326567 pipe
java    33212 root 6464w  FIFO                0,8      0t0 76326567 pipe
java    33212 root 6465u  0000                0,9        0     6827 anon_inode
java    33212 root 6466r  FIFO                0,8      0t0 76344481 pipe
java    33212 root 6467w  FIFO                0,8      0t0 76344481 pipe
java    33212 root 6468u  0000                0,9        0     6827 anon_inode
java    33212 root 6469u  0000                0,9        0     6827 anon_inode
java    33212 root 6470r  FIFO                0,8      0t0 76372935 pipe
java    33212 root 6471w  FIFO                0,8      0t0 76372935 pipe
java    33212 root 6472u  0000                0,9        0     6827 anon_inode
java    33212 root 6473r  FIFO                0,8      0t0 76514685 pipe
java    33212 root 6474r  FIFO                0,8      0t0 76435445 pipe
java    33212 root 6475r  FIFO                0,8      0t0 76406235 pipe
java    33212 root 6476w  FIFO                0,8      0t0 76406235 pipe
java    33212 root 6477u  0000                0,9        0     6827 anon_inode
java    33212 root 6478w  FIFO                0,8      0t0 76435445 pipe
[...]

Graph of open files growing over time:
https://imgur.com/a/mM7MC

Other ActiveMQ "too many open files" related issues seems to be about
sockets that hangs on "CLOSE_WAIT" but that doesn't seems to be the
problem here.

Any idea how to fix this?

--
jocar

Re: ActiveMQ leaks file descriptors

[Johan Carlquist <jo...@su.se>]

> On 4 Nov 2016, at 18:12, mlange <ml...@anwb.nl> wrote:
> 
> Could it be a (or more) messages staying on a queue, causing the logs not to
> be cleared?

No, all messages are consumed and nothing is stored on the server.

We recived two tips from otherside this list:
* Try with the latest JDK
* Disable NIO.

So I have tried them both:

* Disable NIO:
So we disabled NIO on one of the machines for both amqps and openwires
and the graphs speaks for them self, no more leaking file descriptors:
https://imgur.com/a/mEl4k

* Try with the latest JDK:
Upgraded the other server (with NIO enabled) to JDK 8u112 and after
staring at the graphs for a few hours we realised that newer java
didn't fixed the issue:
https://imgur.com/a/Y2WP5

Guess that the next step is to create an issue on
https://issues.apache.org/

--
jocar

Re: ActiveMQ leaks file descriptors

[mlange <ml...@anwb.nl>]

Could it be a (or more) messages staying on a queue, causing the logs not to
be cleared?




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-leaks-file-descriptors-tp4718793p4718954.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.