[jira] [Closed] (AMQ-6260) ActiveMQ 100% memory usage. Too many open files

["Christopher L. Shannon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christopher L. Shannon closed AMQ-6260.
---------------------------------------
    Resolution: Not A Problem

As you have pointed out this error means there are too many open files and needs to be fixed with ulimits and permissions.  There's really nothing that can be done in the code as far as I am aware.

I'm not entirely sure why it's giving the error if you are showing the total file counts lower than the set ulimit but I would suggest researching it a bit. I do know that ulimits can be set in several locations in Linux so may there's a system level configuration conflicting with something.

> ActiveMQ 100% memory usage. Too many open files
> -----------------------------------------------
>
>                 Key: AMQ-6260
>                 URL: https://issues.apache.org/jira/browse/AMQ-6260
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.11.3
>            Reporter: Pankaj Takawale
>
> Version: 5.11.3
> OS: Centos 6.5
> Persistence: JDBC  (Postgresql)
> ActiveMQ is running on 16 vCPU, 29 GB ram. It has around 700K pending messages left on one of the queues. This queue has around 240 to 360 consumers. 
> There are hundreds of other queues, but with less number of pending messages.
> Producer flow control is turned off.
> I'm getting "Too many open files" error. Opened files are much less than the overall limit. See below the details.
> Memory usage reaches to 100% after few minutes of restart. We experienced total 1+ million messages across multiple queues, but did not face the 100% memory or too many opened files at that time.
> Apr 19 08:04:12.232-ActiveMQ NIO Worker 2687 - ERROR TransportConnector   - deliverybroker                        - Could not accept connection : java.io.IOException: Too many open files
> lsof shows: total file counts 1724
> netsta -an|wc  : 374
> ulimit -a for a user that is running activemq process
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 240441
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 65536
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 1024
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> dmesg:
> Out of memory: Kill process 102774 (java) score 945 or sacrifice child
> Killed process 102774, UID 505, (java) total-vm:50435392kB, anon-rss:28966224kB, file-rss:20kB



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)