You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Yeggy Javadi <Ye...@falconstor.com.INVALID> on 2021/05/17 19:17:56 UTC
apache-tomcat-8.5.59 too many open files on Linux 8
Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
I was running the same application on Linux 7.6 with the same apache-tomcat version without seeing this issue. I saw that RedHat has remove apache-tomcat fom their Linux 8 distribution. Please help with throubleshooting and indicating if there is any changes in Linux 8 that can explain this
Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64
Web-based application using Postgres and elastic search databases
# ps -ef | grep tomcat
root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
# cat /proc/51798/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 256518 256518 processes
Max open files 262144 262144 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256518 256518 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Too may open files:
# lsof -p 51798 | wc -l
6679
Most of open files are TCP sockets:
java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
# lsof -p 51798 | grep "protocol: TCPv6" | wc -l
6413
Thanks
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Addendum :
Maybe to debug this more efficiently, you could look at this issue from the opposite side :
Earlier in the thread of messages, you said this :
1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
Maybe when you did this, you also changed the driver which tomcat is using to communicate
with Postgresql. And maybe the problem lies in that driver.
I mean that the driver is the piece of code which creates connections (using sockets) with
Postgresql. And usually, that works and you have a number of ESTABLISHED connections
(which are visible in the netstat output).
But what if, occasionally, the connection doesn't work, and the driver is not very clean
in handling this failing socket ?
Or maybe the issue is in the code which uses these connections ?
Have a look at this :
https://stackoverflow.com/questions/2225221/closing-database-connections-in-java/2225275#2225275
On 26.05.2021 11:12, André Warnier (tomcat/perl) wrote:
> Maybe I am missing something, but at first sight it looks like lsof, inside the container,
> can also not get more information about these "sock" things.
>
> I am running out of ideas.
> If you search in Google for precisely this :
>
> lsof "sock" and "protocol : TCP"
>
> there are a lot of links which discuss similar issues, and this over many years.
> (So it is not either a unique or a recent issue).
>
> They all seem to boil down to this : some *application* is opening sockets, but then not
> really using them (and not closing them).
>
> In your case, that probably means one of the webapps running under tomcat.
>
> It seems quite unlikely that this would be tomcat itself, because if that was the case,
> this tomcat users list would probably be swamped with requests such as yours; which it isn't.
> It is worth noting also, that among all these messages found in Google, I have not so far
> seen a single one which provides another explanation for those "sock" things.
>
> In your case, the problem is going to be in finding out *which* webapp does that, unless
> there are not many, and you can turn them off one-by-one selectively.
> (The difficulty is in part due to the fact that, to the OS, the whole of the JVM, tomcat
> and all the webapps look like one single process; and to lsof also).
>
> Maybe there is some type of logging available in tomcat, that would help finding out which
> application is creating sockets, and then never using or destroying them.
> But my personal competences do not run that far, so maybe someone else can help you here.
>
>
> On 26.05.2021 00:03, Yeggy Javadi wrote:
>> Hi,
>> Below is the nsenter output:
>>
>> # ps -ef | grep tomcat
>> root 165217 1 1 10:22 ? 00:05:33 /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m
>> -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs=
>> -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
>> -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>> root 167329 167268 0 18:00 pts/1 00:00:00 grep --color=auto tomcat
>>
>> # nsenter -t 165217 --net lsof -n -p 165217 |less
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 165217 root cwd DIR 8,2 4096 157664
>> /usr/local/freestor/bin
>> java 165217 root rtd DIR 8,3 4096 2 /
>> java 165217 root txt REG 8,2 8712 8913
>> /usr/local/jdk/jre1.8.0_271/bin/java
>> java 165217 root mem REG 8,2 113371 160881
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-jpa-2.1-api-1.0.0.Final.jar
>>
>> java 165217 root mem REG 8,2 147874 160802
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activemq-protobuf-1.1.jar
>> java 165217 root mem REG 8,2 391515 160836
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-queryparser-4.10.4.jar
>> java 165217 root mem REG 8,2 868615 160813
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-context-3.2.17.RELEASE.jar
>> java 165217 root mem REG 8,2 9711 160792
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.6.6.jar
>> java 165217 root mem REG 8,2 196573 160819
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-expression-3.2.17.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 97173 160843
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-misc-4.10.4.jar
>> java 165217 root mem REG 8,2 10074 160872
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-ext-1.11.jar
>> java 165217 root mem REG 8,2 62983 160861
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activation-1.1.jar
>> java 165217 root mem REG 8,2 371668 160812
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-security-core-3.2.9.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 645914 160754
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-entitymanager-4.3.5.Final.jar
>>
>> java 165217 root mem REG 8,2 427030 160753
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-envers-4.3.5.Final.jar
>> java 165217 root mem REG 8,2 256468 160829
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/barcode4j-2.0.jar
>> java 165217 root mem REG 8,2 489884 160846
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/log4j-1.2.17.jar
>> java 165217 root mem REG 8,2 29257 160798
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.7.jar
>> java 165217 root mem REG 8,2 422612 160783
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-awt-util-1.11.jar
>> java 165217 root mem REG 8,2 16519 160796
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/jcl-over-slf4j-1.7.7.jar
>> java 165217 root mem REG 8,2 322362 160851
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/cglib-nodep-2.2.jar
>> java 165217 root mem REG 8,2 864598 160848
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/castor-1.2.jar
>> java 165217 root mem REG 8,2 434678 160748
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang3-3.4.jar
>> java 165217 root mem REG 8,2 325196 160785
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-css-1.11.jar
>> java 165217 root mem REG 8,2 5230007 160755
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-core-4.3.5.Final.jar
>> java 165217 root mem REG 8,2 14041249 160763
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/elasticsearch-1.7.5.jar
>> java 165217 root mem REG 8,2 206385 160749
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-jms-3.2.17.RELEASE.jar
>> java 165217 root mem REG 8,2 64021 160837
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-join-4.10.4.jar
>> java 165217 root mem REG 8,2 714194 160828
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/javassist-3.18.1-GA.jar
>> java 165217 root mem REG 8,2 279193 160876
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang-2.5.jar
>> java 165217 root mem REG 8,2 75311 160826
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-commons-annotations-4.0.4.Final.jar
>>
>> java 165217 root mem REG 8,2 436303 160745
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/avro-1.7.7.jar
>> ................
>> java 165217 root 73w FIFO 0,13 0t0 16786125 pipe
>> java 165217 root 74u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 75r FIFO 0,13 0t0 16786126 pipe
>> java 165217 root 76w FIFO 0,13 0t0 16786126 pipe
>> java 165217 root 77u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 78u sock 0,9 0t0 16961335 protocol: TCPv6
>> java 165217 root 79r FIFO 0,13 0t0 16786127 pipe
>> java 165217 root 80w FIFO 0,13 0t0 16786127 pipe
>> java 165217 root 81u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 82r FIFO 0,13 0t0 16786128 pipe
>> java 165217 root 83w FIFO 0,13 0t0 16786128 pipe
>> java 165217 root 84u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 86r FIFO 0,13 0t0 16786129 pipe
>> java 165217 root 87w FIFO 0,13 0t0 16786129 pipe
>> java 165217 root 88u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 89r FIFO 0,13 0t0 16786130 pipe
>> java 165217 root 90w FIFO 0,13 0t0 16786130 pipe
>> java 165217 root 91u a_inode 0,14 0 10970 [eventpoll]
>> java 165217 root 93u IPv6 16786133 0t0 TCP 127.0.0.1:mxi
>> (LISTEN)
>> java 165217 root 94u sock 0,9 0t0 16963075 protocol: TCPv6
>> java 165217 root 96u sock 0,9 0t0 16824649 protocol: TCPv6
>> java 165217 root 97u IPv6 17051898 0t0 TCP
>> 10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)
>> java 165217 root 98u sock 0,9 0t0 16824673 protocol: TCPv6
>> java 165217 root 99u sock 0,9 0t0 16824711 protocol: TCPv6
>> java 165217 root 100u sock 0,9 0t0 16893245 protocol: TCPv6
>> java 165217 root 101u sock 0,9 0t0 16824385 protocol: TCPv6
>> java 165217 root 102u sock 0,9 0t0 16824264 protocol: TCPv6
>> java 165217 root 103u sock 0,9 0t0 16824278 protocol: TCPv6
>> java 165217 root 104u sock 0,9 0t0 16824279 protocol: TCPv6
>> java 165217 root 105u sock 0,9 0t0 16824063 protocol: TCPv6
>> java 165217 root 106u sock 0,9 0t0 16823982 protocol: TCPv6
>> java 165217 root 107u sock 0,9 0t0 16823959 protocol: TCPv6
>> java 165217 root 108r CHR 1,8 0t0 1041 /dev/random
>> java 165217 root 109u sock 0,9 0t0 16778709 protocol: TCPv6
>> java 165217 root 110w REG 8,6 25506176 1835053
>> /apps/apache-tomcat/logs/fms.log
>> java 165217 root 111w REG 8,6 919042 1835019
>> /apps/apache-tomcat/logs/obd.log
>> java 165217 root 112u sock 0,9 0t0 16823821 protocol: TCPv6
>> java 165217 root 113u sock 0,9 0t0 16823822 protocol: TCPv6
>> java 165217 root 114u sock 0,9 0t0 16823633 protocol: TCPv6
>> java 165217 root 115u sock 0,9 0t0 16823527 protocol: TCPv6
>> java 165217 root 116u sock 0,9 0t0 16823489 protocol: TCPv6
>> java 165217 root 117u sock 0,9 0t0 16823490 protocol: TCPv6
>> java 165217 root 118u sock 0,9 0t0 16823491 protocol: TCPv6
>> ..................
>>
>> Thanks
>>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Tuesday, May 25, 2021 12:42 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Hi.
>> The point is to try to figure out what these thousands of apparently "TCPv6" sockets
>> belonging to the tomcat process actually are, so that we can maybe begin to look at where
>> they may be coming from.
>> The trouble is, the lsof output so far did not really tell us what these "sock" things
>> might be.
>>
>> But there may be a clue here :
>> https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
>> (about when things run in a container).
>> Is that your case ?
>> And if so, could you run the lsof command in the container, as they suggest ?
>>
>> And the point of forcing a tomcat/JVM GC was this :
>> When you restart tomcat (actually the JVM which runs tomcat), the OS will clean up *all*
>> the file descriptors belonging to that process, including the "legitimate" ones shown by
>> netstat, and the "unknown" ones shown in addition by lsof.
>> Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd that are held
>> by objects which are discarded, but still sit on the heap awaiting a GC to really destroy
>> them. If your heap is very large, it may otherwise take a long while before such a GC
>> happens, and such sockets could accumulate.
>> One way to trigger a GC is through JMX, but it takes a bit of additional setup to make
>> that work. That's why I was asking if you had some method to do that.
>> (see : https://code.google.com/archive/p/jmxsh/)
>> But let's look at the lsof part first.
>>
>>
>>
>> On 24.05.2021 16:09, Yeggy Javadi wrote:
>>> Hi,
>>> I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat
>>> GC, I just restart it to reset the lsof to 0.
>>> Please see outputs below:
>>>
>>> # ps -ef | grep tomcat
>>> root 143152 1 0 May22 ? 00:26:44 /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server
>>> -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs=
>>> -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
>>> -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>>> root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto tomcat
>>>
>>> # lsof -p 143152 | wc -l
>>> 41043
>>>
>>> # lsof -p 143152 | grep "protocol: TCPv6"| wc -l
>>> 40487
>>>
>>> # netstat -p -a -n --inet6 | grep 143152
>>> tcp6 0 0 :::8443 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 :::443 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:8005 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 :::1099 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 :::80 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 :::36081 :::* LISTEN
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55446 ESTABLISHED
>>> 143152/java
>>> tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576 CLOSE_WAIT
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:53682 172.22.21.48:443 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576 CLOSE_WAIT
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:40574 172.22.21.47:443 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:53782 172.22.21.48:443 ESTABLISHED
>>> 143152/java
>>> tcp6 0 1 10.4.3.55:49808 10.12.3.78:443 SYN_SENT
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60226 172.22.22.192:443 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55444 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:40540 172.22.21.47:443 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432 ESTABLISHED
>>> 143152/java
>>> tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>> tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300 ESTABLISHED
>>> 143152/java
>>>
>>> # lsof -a -p 143152 -T s -i6
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>> java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
>>> java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
>>> java 143152 root 82u IPv6 13816325 0t0 TCP localhost.localdomain:mxi (LISTEN)
>>> java 143152 root 84u IPv6 13874431 0t0 TCP
>>> localhost.localdomain:52310->localhost.localdomain:postgres (ESTABLISHED)
>>> java 143152 root 86u IPv6 13848367 0t0 TCP
>>> localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
>>> java 143152 root 215u IPv6 13822102 0t0 TCP
>>> localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
>>> java 143152 root 216u IPv6 13823064 0t0 TCP
>>> localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
>>> java 143152 root 217u IPv6 13821107 0t0 TCP
>>> localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
>>> java 143152 root 269u IPv6 13824001 0t0 TCP
>>> localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
>>> java 143152 root 270u IPv6 13787907 0t0 TCP
>>> Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 271u IPv6 13787908 0t0 TCP
>>> Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 272u IPv6 13787909 0t0 TCP
>>> Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 273u IPv6 13787910 0t0 TCP
>>> Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 274u IPv6 13787911 0t0 TCP
>>> Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 275u IPv6 13787912 0t0 TCP
>>> Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 276u IPv6 13787913 0t0 TCP
>>> Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 277u IPv6 13787914 0t0 TCP
>>> Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 278u IPv6 13787915 0t0 TCP
>>> Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 279u IPv6 13787916 0t0 TCP
>>> Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 280u IPv6 13787917 0t0 TCP
>>> Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 281u IPv6 13787918 0t0 TCP
>>> Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 282u IPv6 13787919 0t0 TCP
>>> Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry (LISTEN)
>>> java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
>>> java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https (LISTEN)
>>> java 143152 root *740u IPv6 15607907 0t0 TCP
>>> Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
>>> java 143152 root *876u IPv6 15608779 0t0 TCP
>>> Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
>>> java 143152 root *881u IPv6 15614275 0t0 TCP
>>> Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
>>> java 143152 root *882u IPv6 15558639 0t0 TCP
>>> Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
>>> java 143152 root *067u IPv6 15602583 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
>>> java 143152 root *070u IPv6 15602586 0t0 TCP
>>> Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
>>> java 143152 root *073u IPv6 15602582 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
>>> java 143152 root *080u IPv6 15512947 0t0 TCP
>>> Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>>> java 143152 root *081u IPv6 15512948 0t0 TCP
>>> Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>>> java 143152 root *817u IPv6 15611323 0t0 TCP
>>> Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
>>> java 143152 root *818u IPv6 15602584 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
>>>
>>> Thank
>>>
>>> -----Original Message-----
>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>> Sent: Sunday, May 23, 2021 6:30 PM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Hi.
>>>
>>> I have no idea what lines like this are :
>>>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>>>
>>> There are obviously too many of them, for them to match with the sockets listed by
>>> netstat.
>>>
>>> The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
>>> ones marked as "LISTEN" in the netstat output.
>>> But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
>>>
>>> Could you redo a netstat as follows :
>>> # netstat -p -a -n --inet6 | grep 130244
>>>
>>> and can you also try this form of lsof :
>>> # lsof -a -p 130244 -T s -i6
>>>
>>> And finally (after copying the result of the above) : do you know how to trigger a GC
>>> (Garbage Collection) in your tomcat JVM ?
>>> (the point is to see if when a GC happens, these things disappear).
>>>
>>>
>>> On 22.05.2021 18:03, Yeggy Javadi wrote:
>>>> Here it is:
>>>>
>>>> # netstat -p -a --tcp | grep 130244
>>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 [::]:https [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 [::]:37537 [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 [::]:8009 [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 0 0 [::]:http [::]:* LISTEN
>>>> 130244/java
>>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT
>>>> 130244/java
>>>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED
>>>> 130244/java
>>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED
>>>> 130244/java
>>>>
>>>>
>>>> # lsof -p 130244
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>> java 130244 root cwd DIR 8,2 4096 157664
>>>> /usr/local/freestor/bin
>>>> java 130244 root rtd DIR 8,3 4096 2 /
>>>> java 130244 root txt REG 8,2 8712 8913
>>>> /usr/local/jdk/jre1.8.0_271/bin/java
>>>> java 130244 root mem REG 8,2 498864 9007
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>>>> java 130244 root mem REG 8,2 39176 9006
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>>>> java 130244 root mem REG 8,2 759184 8996
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>>>> java 130244 root mem REG 8,2 3559360 9139
>>>> /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>>> java 130244 root mem REG 8,2 99680 133076
>>>> /usr/lib64/libgcc_s-8-20191121.so.1
>>>> java 130244 root mem REG 8,2 3135658 9133
>>>> /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>>> java 130244 root mem REG 8,2 283368 8980
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>>>> java 130244 root mem REG 8,2 1907341 9138
>>>> /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>>> java 130244 root mem REG 8,2 285868 9126
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>>> java 130244 root mem REG 8,2 43191 9125
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>>> java 130244 root mem REG 8,2 283206 9127
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>>> java 130244 root mem REG 8,2 118047 9136
>>>> /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>>> java 130244 root mem REG 8,2 1179462 9128
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>>> java 130244 root mem REG 8,2 3860563 9129
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>>> java 130244 root mem REG 8,2 51984 8995
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>>>> java 130244 root mem REG 8,2 12373 8871
>>>> /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>>> java 130244 root mem REG 8,2 169952 8868
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>>> java 130244 root mem REG 8,2 168382 8867
>>>> /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>>> java 130244 root mem REG 8,2 244451 8862
>>>> /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>>> java 130244 root mem REG 8,2 148761 8881
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>>> java 130244 root mem REG 8,2 62004 8863
>>>> /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>>> java 130244 root mem REG 8,2 2450404 8864
>>>> /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>>> java 130244 root mem REG 8,2 34798 8873
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>>> java 130244 root mem REG 8,2 78087 8886
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>>> java 130244 root mem REG 8,2 89695 8877
>>>> /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>>> java 130244 root mem REG 8,2 599431 8884
>>>> /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>>> java 130244 root mem REG 8,2 174393 8870
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>>> java 130244 root mem REG 8,2 294856 8861
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>>> java 130244 root mem REG 8,2 173282 8880
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>>> java 130244 root mem REG 8,2 284036 8872
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>>> java 130244 root mem REG 8,2 72656 8865
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>>> java 130244 root mem REG 8,2 54667 8878
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>>> java 130244 root mem REG 8,2 1705863 8860
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>>> java 130244 root mem REG 8,2 38241 8882
>>>> /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>>> java 130244 root mem REG 8,2 106051 8869
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>>> java 130244 root mem REG 8,2 120073 8876
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>>> java 130244 root mem REG 8,2 228391 8879
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>>> java 130244 root mem REG 8,2 868336 8888
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>>> java 130244 root mem REG 8,2 215400 8874
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>>> java 130244 root mem REG 8,2 155094 8866
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>>> java 130244 root mem REG 8,2 26872 8885
>>>> /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>>> java 130244 root mem REG 8,2 113008 8989
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>>>> java 130244 root mem REG 8,2 93872 9003
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>>>> java 130244 root mem REG 8,2 337024 395971
>>>> /usr/lib/locale/en_US.utf8/LC_CTYPE
>>>> java 130244 root mem REG 8,2 2586930 390297
>>>> /usr/lib/locale/en_US.utf8/LC_COLLATE
>>>> java 130244 root mem REG 8,2 10673 8887
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>>> java 130244 root mem REG 8,2 38561 8875
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>>> java 130244 root mem REG 8,2 158590 8883
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>>> java 130244 root mem REG 8,2 51205 8857
>>>> /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>>> java 130244 root mem REG 8,2 25287 8835
>>>> /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>>> java 130244 root mem REG 8,2 36132 8837
>>>> /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>>> java 130244 root mem REG 8,2 54 395975
>>>> /usr/lib/locale/en_US.utf8/LC_NUMERIC
>>>> java 130244 root mem REG 8,2 3316 521695
>>>> /usr/lib/locale/en_US.utf8/LC_TIME
>>>> java 130244 root mem REG 8,2 286 521693
>>>> /usr/lib/locale/en_US.utf8/LC_MONETARY
>>>> java 130244 root mem REG 8,2 26998 130155
>>>> /usr/lib64/gconv/gconv-modules.cache
>>>> java 130244 root mem REG 8,2 67053209 9140
>>>> /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>>> java 130244 root mem REG 8,2 127016 8988
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>>>> java 130244 root mem REG 8,2 76880 133068
>>>> /usr/lib64/libnss_files-2.28.so
>>>> java 130244 root mem REG 8,3 9253600 129922
>>>> /var/lib/sss/mc/passwd
>>>> java 130244 root mem REG 8,2 46432 134002
>>>> /usr/lib64/libnss_sss.so.2
>>>> java 130244 root mem REG 8,2 231960 8986
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>>>> java 130244 root mem REG 8,2 66112 8975
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>>>> java 130244 root mem REG 8,2 68944 133091
>>>> /usr/lib64/librt-2.28.so
>>>> java 130244 root mem REG 8,2 2191808 133023
>>>> /usr/lib64/libm-2.28.so
>>>> java 130244 root mem REG 8,2 17108824 8969
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>>>> java 130244 root mem REG 8,2 3154704 129871
>>>> /usr/lib64/libc-2.28.so
>>>> java 130244 root mem REG 8,2 28968 133013
>>>> /usr/lib64/libdl-2.28.so
>>>> java 130244 root mem REG 8,2 108616 8973
>>>> /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>>>> java 130244 root mem REG 8,2 321552 133079
>>>> /usr/lib64/libpthread-2.28.so
>>>> java 130244 root mem REG 8,2 252280 129864
>>>> /usr/lib64/ld-2.28.so
>>>> java 130244 root mem REG 8,3 32768 18
>>>> /tmp/hsperfdata_root/130244
>>>> java 130244 root mem REG 8,2 57 395973
>>>> /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>>>> java 130244 root mem REG 8,2 34 390317
>>>> /usr/lib/locale/en_US.utf8/LC_PAPER
>>>> java 130244 root mem REG 8,2 77 395974
>>>> /usr/lib/locale/en_US.utf8/LC_NAME
>>>> java 130244 root mem REG 8,2 167 519390
>>>> /usr/lib/locale/en_US.utf8/LC_ADDRESS
>>>> java 130244 root mem REG 8,2 59 521694
>>>> /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>>>> java 130244 root mem REG 8,2 23 521692
>>>> /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>>>> java 130244 root mem REG 8,2 368 521691
>>>> /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>>>> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
>>>> java 130244 root 1w REG 8,6 394654 1835011
>>>> /apps/apache-tomcat/logs/catalina.out
>>>> java 130244 root 2w REG 8,6 394654 1835011
>>>> /apps/apache-tomcat/logs/catalina.out
>>>> java 130244 root 3r REG 8,3 9253600 129922
>>>> /var/lib/sss/mc/passwd
>>>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
>>>> java 130244 root 5r REG 8,2 67053209 9140
>>>> /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>>> java 130244 root 6r REG 8,2 36132 8837
>>>> /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>>> java 130244 root 7r REG 8,2 25287 8835
>>>> /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>>> java 130244 root 8r REG 8,2 51205 8857
>>>> /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>>> java 130244 root 9w REG 8,6 21883 1835065
>>>> /apps/apache-tomcat/logs/catalina.2021-05-20.log
>>>> java 130244 root 10w REG 8,6 761 1835066
>>>> /apps/apache-tomcat/logs/localhost.2021-05-20.log
>>>> java 130244 root 11w REG 8,6 0 1835067
>>>> /apps/apache-tomcat/logs/manager.2021-05-20.log
>>>> java 130244 root 12w REG 8,6 0 1835068
>>>> /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>>>> java 130244 root 13r REG 8,2 158590 8883
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>>> java 130244 root 14r REG 8,2 38561 8875
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>>> java 130244 root 15r REG 8,2 26872 8885
>>>> /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>>> java 130244 root 16r REG 8,2 155094 8866
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>>> java 130244 root 17r REG 8,2 215400 8874
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>>> java 130244 root 18r REG 8,2 868336 8888
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>>> java 130244 root 19r REG 8,2 228391 8879
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>>> java 130244 root 20r REG 8,2 120073 8876
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>>> java 130244 root 21r REG 8,2 106051 8869
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>>> java 130244 root 22r REG 8,2 38241 8882
>>>> /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>>> java 130244 root 23r REG 8,2 1705863 8860
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>>> java 130244 root 24r REG 8,2 54667 8878
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>>> java 130244 root 25r REG 8,2 72656 8865
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>>> java 130244 root 26r REG 8,2 284036 8872
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>>> java 130244 root 27r REG 8,2 10673 8887
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>>> java 130244 root 28r REG 8,2 173282 8880
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>>> java 130244 root 29r REG 8,2 294856 8861
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>>> java 130244 root 30r REG 8,2 174393 8870
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>>> java 130244 root 31r REG 8,2 599431 8884
>>>> /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>>> java 130244 root 32r REG 8,2 89695 8877
>>>> /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>>> java 130244 root 33r REG 8,2 78087 8886
>>>> /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>>> java 130244 root 34r REG 8,2 34798 8873
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>>> java 130244 root 35r REG 8,2 2450404 8864
>>>> /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>>> java 130244 root 36r REG 8,2 62004 8863
>>>> /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>>> java 130244 root 37r REG 8,2 148761 8881
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>>> java 130244 root 38r REG 8,2 244451 8862
>>>> /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>>> java 130244 root 39r REG 8,2 168382 8867
>>>> /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>>> java 130244 root 40r REG 8,2 169952 8868
>>>> /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>>> java 130244 root 41r REG 8,2 12373 8871
>>>> /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>>> java 130244 root 42r REG 8,2 1907341 9138
>>>> /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>>> java 130244 root 43r REG 8,2 3860563 9129
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
>>>> java 130244 root 45r REG 8,2 1179462 9128
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>>> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
>>>> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
>>>> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
>>>> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
>>>> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
>>>> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
>>>> java 130244 root 52r REG 8,2 118047 9136
>>>> /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>>> java 130244 root 53r REG 8,2 283206 9127
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>>> java 130244 root 54r REG 8,2 43191 9125
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>>> java 130244 root 55r REG 8,2 285868 9126
>>>> /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>>> java 130244 root 56r REG 8,2 3559360 9139
>>>> /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>>> java 130244 root 57r REG 8,2 3135658 9133
>>>> /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>>> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
>>>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
>>>> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
>>>> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
>>>> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
>>>> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
>>>> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
>>>> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
>>>> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
>>>> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
>>>> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 71w REG 8,6 342714 1835069
>>>> /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>>>> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
>>>> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
>>>> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
>>>> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
>>>> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>>>> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
>>>> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
>>>> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
>>>> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
>>>> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
>>>> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
>>>> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
>>>> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
>>>> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
>>>> java 130244 root 93u IPv6 11968853 0t0 TCP
>>>> localhost.localdomain:mxi (LISTEN)
>>>> java 130244 root 94u IPv6 12298916 0t0 TCP
>>>> localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>>>> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
>>>> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
>>>> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
>>>> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
>>>> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
>>>> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
>>>> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
>>>> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
>>>> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
>>>> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
>>>> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
>>>> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
>>>> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
>>>> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
>>>> java 130244 root 109w REG 8,6 19536380 1835053
>>>> /apps/apache-tomcat/logs/fms.log
>>>> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
>>>> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
>>>> java 130244 root 112w REG 8,6 835408 1835019
>>>> /apps/apache-tomcat/logs/obd.log
>>>> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
>>>> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
>>>> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
>>>> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
>>>> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
>>>> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
>>>> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
>>>> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
>>>> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
>>>> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
>>>> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
>>>> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
>>>> ...........................
>>>> Keep seeing the same line
>>>>
>>>> -----Original Message-----
>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>> Sent: Saturday, May 22, 2021 10:22 AM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Mmm. Nothing very special in that netstat output.
>>>> The sockets seen there look quite normal for tomcat, and there are not a lot.
>>>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>>>
>>>> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the
>>>> tomcat JVM process.
>>>> Where is the difference between the lsof count (19948) and the netstat count (25) ?
>>>> (genuine question, I'm puzzled too)
>>>>
>>>> Can you give an example of the "lsof -p 130244" output lines ?
>>>> (not all 19948 please, but enough to see some variety)
>>>>
>>>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>>>> Hi,
>>>>> Here its is:
>>>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>> 19948
>>>>>
>>>>> # netstat -p -a -6 --tcp | grep 130244
>>>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 [::]:https [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 [::]:37537 [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 [::]:8009 [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 0 0 [::]:http [::]:* LISTEN
>>>>> 130244/java
>>>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED
>>>>> 130244/java
>>>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED
>>>>> 130244/java
>>>>> Thanks
>>>>> -----Original Message-----
>>>>> From: Noelette Stout <no...@isu.edu>
>>>>> Sent: Friday, May 21, 2021 8:28 AM
>>>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> ss has all the same options as netstat
>>>>>
>>>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>>>> <aw...@ice-sa.com>
>>>>> wrote:
>>>>>
>>>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>>>
>>>>>> The next thing that you may want to do :
>>>>>>
>>>>>> > netstat -p -a -6 --tcp
>>>>>>
>>>>>> That is an alternative list of sockets, which also shows the "tcp state"
>>>>>> of the sockets.
>>>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>>>> the last column.
>>>>>> The type of thing you are looking for is the column which should show
>>>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>>>
>>>>>> The options above :
>>>>>> -p : show PID and program
>>>>>> -a : show all sockets states
>>>>>> -6 : only inet v6
>>>>>> --tcp : only TCP sockets
>>>>>>
>>>>>> "netstat" may not be on your system by default, and you may need to
>>>>>> install it.
>>>>>> An alternative is "ss", but I don't know the options.
>>>>>>
>>>>>>
>>>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>>>> Hi,
>>>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>>>> 7474
>>>>>> are for TCPv6 sockets:
>>>>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>>>>> protocol: TCPv6
>>>>>>>
>>>>>>> # ps -ef | grep tomcat
>>>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>>>>> tomcat
>>>>>>>
>>>>>>> #lsof -p 130244 | wc -l
>>>>>>> 8028
>>>>>>>
>>>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>>>> 7474
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Hi.
>>>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>>>> on
>>>>>> your system.
>>>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>>>> when
>>>>>> you look at the whole command-line.
>>>>>>>
>>>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>>>> relevant here.
>>>>>>>
>>>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>>>> files opened by the tomcat JVM.
>>>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>>>> "apache-tomcat"
>>>>>>> (or anything else unique in that line)
>>>>>>>
>>>>>>>
>>>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>>>> Hi Chris,
>>>>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>>>>
>>>>>>>> Below is the list of running processes on my server:
>>>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>>>> root 1 0 0 May07 ? 00:00:14
>>>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>>>> root 6 2 0 May07 ? 00:00:00
>>>>>> [kworker/0:0H-kblockd]
>>>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>>>> root 19 2 0 May07 ? 00:00:00
>>>>>> [kworker/1:0H-kblockd]
>>>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>>>> root 25 2 0 May07 ? 00:00:00
>>>>>> [kworker/2:0H-kblockd]
>>>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>>>> root 31 2 0 May07 ? 00:00:00
>>>>>> [kworker/3:0H-kblockd]
>>>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>>>> root 37 2 0 May07 ? 00:00:00
>>>>>> [kworker/4:0H-kblockd]
>>>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>>>> root 43 2 0 May07 ? 00:00:00
>>>>>> [kworker/5:0H-kblockd]
>>>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>>>> root 49 2 0 May07 ? 00:00:00
>>>>>> [kworker/6:0H-kblockd]
>>>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>>>> root 55 2 0 May07 ? 00:00:00
>>>>>> [kworker/7:0H-kblockd]
>>>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>>>> root 540 2 0 May07 ? 00:00:01
>>>>>> [kworker/7:1H-kblockd]
>>>>>>>> root 541 2 0 May07 ? 00:00:01
>>>>>> [kworker/5:1H-kblockd]
>>>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 613 2 0 May07 ? 00:00:01
>>>>>> [kworker/6:1H-kblockd]
>>>>>>>> root 650 2 0 May07 ? 00:00:01
>>>>>> [kworker/4:1H-kblockd]
>>>>>>>> root 659 2 0 May07 ? 00:00:02
>>>>>> [kworker/3:1H-kblockd]
>>>>>>>> root 660 2 0 May07 ? 00:00:02
>>>>>> [kworker/2:1H-kblockd]
>>>>>>>> root 662 2 0 May07 ? 00:00:01
>>>>>> [kworker/0:1H-kblockd]
>>>>>>>> root 679 1 0 May07 ? 00:00:03
>>>>>> /usr/lib/systemd/systemd-journald
>>>>>>>> root 682 2 0 May07 ? 00:00:02
>>>>>> [kworker/1:1H-kblockd]
>>>>>>>> root 716 1 0 May07 ? 00:00:01
>>>>>> /usr/lib/systemd/systemd-udevd
>>>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>>>> --systemd-activation --syslog-only
>>>>>>>> root 885 1 0 May07 ? 00:02:42
>>>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>>>> root 887 1 0 May07 ? 00:00:44
>>>>>> /usr/sbin/irqbalance --foreground
>>>>>>>> root 888 1 0 May07 ? 00:00:00
>>>>>> /usr/bin/VGAuthService -s
>>>>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>>>>> --logger=files
>>>>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>>>> --fill-watermark=0
>>>>>>>> root 946 890 0 May07 ? 00:00:06
>>>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>>>> --logger=files
>>>>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>>>>> -d2
>>>>>>>> root 962 890 0 May07 ? 00:00:12
>>>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>>>>> -D
>>>>>>>> root 970 1 0 May07 ? 00:00:02
>>>>>> /usr/lib/systemd/systemd-logind
>>>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>>>>> -n
>>>>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>>>> /usr/local/vtl/bin/sanidmgr
>>>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>>>> /usr/local/vtl/bin/iscliproxy
>>>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>>>> /usr/local/vtl/bin/ipstorlog
>>>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>>>> /usr/local/vtl/bin/snmpd
>>>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>>>> /usr/lib/systemd/systemd --user
>>>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>>>>> -- \u --noclear tty1 linux
>>>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>>>> -
>>>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>>>> h
>>>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>>>> p
>>>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>>>> -
>>>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>>>> a
>>>>>> 1,diffie-hellman-group14-sha1
>>>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>>> m
>>>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>>>> [kworker/4:1-events_power_efficient]
>>>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>>>> [kworker/4:2-cgroup_destroy]
>>>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>>>> [kworker/6:0-mm_percpu_wq]
>>>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>>>> [kworker/3:3-mm_percpu_wq]
>>>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>>>> [kworker/2:1-mm_percpu_wq]
>>>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>>>> -XX:+UseConcMarkSweepGC
>>>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>>>> t
>>>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>>>> -Des.default.path.conf=/etc/elasticsearch
>>>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>>>>> process
>>>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>>>> checkpointer process
>>>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>>>>> process
>>>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>>>> writer process
>>>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>>>> autovacuum launcher process
>>>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>>>> collector process
>>>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>>>> /usr/local/freestor/bin/fmsd
>>>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>>>> freestor 127.0.0.1(42660) idle
>>>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>>>>> freestor 127.0.0.1(42662) idle
>>>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>>>> freestor 127.0.0.1(42664) idle
>>>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>>>> [kworker/5:1-mm_percpu_wq]
>>>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>>>> [kworker/u16:2-events_unbound]
>>>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>>>> [kworker/u16:1-events_unbound]
>>>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>>>> [kworker/1:0-events_freezable]
>>>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -----Original Message-----
>>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>>>> To: users@tomcat.apache.org
>>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>>
>>>>>>>> Yeggy,
>>>>>>>>
>>>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>>>> Hi Chris,
>>>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>>>
>>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>>>> 9.3
>>>>>> to 9.6.
>>>>>>>>>
>>>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>>>> many
>>>>>> open files) error when the limit of open files for tomcat has reached
>>>>>> (262144 in my environment), for example, I can get a connectivity
>>>>>> error when pulling info from a server as below :
>>>>>>>>>
>>>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>>>
>>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>>>
>>>>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>>>>> maxThreads="150"
>>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>>>>> compression="on"
>>>>>>>>> compressionMinSize="2048"
>>>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>>>
>>>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>>>> redirectPort="443" /> ...........
>>>>>>>>> <Connector address="0.0.0.0" port="443"
>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>>>>
>>>>>>
>>>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>
>>>>>>>>> keystoreFile="conf/freestor"
>>>>>> keystorePass="freestor"/>
>>>>>>>>>
>>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>>>>
>>>>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>>>
>>>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>>>> Definitely
>>>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>>>
>>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>>> of
>>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>>> Different kernel versions count things differently.
>>>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>>>> files for that process
>>>>>>>>
>>>>>>>> Ok.
>>>>>>>>
>>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>>> [YJ] Here is the netstat output:
>>>>>>>>> Active Internet connections (w/o servers)
>>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>>>> State
>>>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>>>> ESTABLISHED
>>>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>>>> ESTABLISHED
>>>>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>>>>> ESTABLISHED
>>>>>>>>
>>>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>>>> not
>>>>>> what the whole OS is handling.
>>>>>>>>
>>>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>>>> the
>>>>>> process. But if you get "too many open files" being thrown inside the
>>>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>>>> process, then something else is going on. But remember to limit your
>>>>>> lsof to *just the JVM* process.
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>>>> To: users@tomcat.apache.org
>>>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>>>
>>>>>>>>> Yeggy,
>>>>>>>>>
>>>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>>>> protocol: TCPv6
>>>>>>>>>>
>>>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>>>> changes in Linux
>>>>>>>>>> 8 that can explain this
>>>>>>>>>>
>>>>>>>>>> Configuration:
>>>>>>>>>> apache-tomcat-8.5.59
>>>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # ps -ef | grep tomcat
>>>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>>>>> tomcat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # cat /proc/51798/limits
>>>>>>>>>> Limit Soft Limit Hard Limit
>>>>>> Units
>>>>>>>>>> Max cpu time unlimited unlimited
>>>>>> seconds
>>>>>>>>>> Max file size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max data size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max stack size 8388608 unlimited
>>>>>> bytes
>>>>>>>>>> Max core file size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max resident set unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max processes 256518 256518
>>>>>> processes
>>>>>>>>>> Max open files 262144 262144
>>>>>> files
>>>>>>>>>> Max locked memory 65536 65536
>>>>>> bytes
>>>>>>>>>> Max address space unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max file locks unlimited unlimited
>>>>>> locks
>>>>>>>>>> Max pending signals 256518 256518
>>>>>> signals
>>>>>>>>>> Max msgqueue size 819200 819200
>>>>>> bytes
>>>>>>>>>> Max nice priority 0 0
>>>>>>>>>> Max realtime priority 0 0
>>>>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Too may open files:
>>>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>>>> 6679
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Most of open files are TCP sockets:
>>>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>>>> protocol: TCPv6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>>>> 6413
>>>>>>>>>
>>>>>>>>> Some potentially dumb questions:
>>>>>>>>>
>>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>>>
>>>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>>>
>>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>>>
>>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>>> of
>>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>>> Different kernel versions count things differently.
>>>>>>>>>
>>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>>>
>>>>>>>>> -chris
>>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Noelette Stout
>>>>>
>>>>> ITS Enterprise Applications - Application Administrator - Senior
>>>>>
>>>>> Business Administration Building, Rm 109L
>>>>> 921 South 8th Ave 8037
>>>>> Idaho State University
>>>>> Pocatello ID 83209
>>>>> E-mail: stounoel "at" isu "dot" edu
>>>>> Desk: 208-282-2554
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
André,
On 5/26/21 05:12, André Warnier (tomcat/perl) wrote:
> Maybe I am missing something, but at first sight it looks like lsof,
> inside the container, can also not get more information about these
> "sock" things.
>
> I am running out of ideas.
> If you search in Google for precisely this :
>
> lsof "sock" and "protocol : TCP"
>
> there are a lot of links which discuss similar issues, and this over
> many years.
> (So it is not either a unique or a recent issue).
>
> They all seem to boil down to this : some *application* is opening
> sockets, but then not really using them (and not closing them).
>
> In your case, that probably means one of the webapps running under tomcat.
Maybe a database connection pool? Or a REST service that the server code
is calling-out to? Or an SMTP connection? Or a JMS queue? Or, or, or..
> It seems quite unlikely that this would be tomcat itself, because if
> that was the case, this tomcat users list would probably be swamped with
> requests such as yours; which it isn't.
+1
-chris
> On 26.05.2021 00:03, Yeggy Javadi wrote:
>> Hi,
>> Below is the nsenter output:
>>
>> # ps -ef | grep tomcat
>> root 165217 1 1 10:22 ? 00:05:33
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
>> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
>> -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>> root 167329 167268 0 18:00 pts/1 00:00:00 grep --color=auto
>> tomcat
>>
>> # nsenter -t 165217 --net lsof -n -p 165217 |less
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>> NODE NAME
>> java 165217 root cwd DIR 8,2 4096
>> 157664 /usr/local/freestor/bin
>> java 165217 root rtd DIR 8,3 4096 2 /
>> java 165217 root txt REG 8,2 8712
>> 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>> java 165217 root mem REG 8,2 113371
>> 160881
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-jpa-2.1-api-1.0.0.Final.jar
>>
>> java 165217 root mem REG 8,2 147874
>> 160802
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activemq-protobuf-1.1.jar
>>
>> java 165217 root mem REG 8,2 391515
>> 160836
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-queryparser-4.10.4.jar
>>
>> java 165217 root mem REG 8,2 868615
>> 160813
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-context-3.2.17.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 9711
>> 160792
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.6.6.jar
>>
>> java 165217 root mem REG 8,2 196573
>> 160819
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-expression-3.2.17.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 97173
>> 160843
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-misc-4.10.4.jar
>>
>> java 165217 root mem REG 8,2 10074
>> 160872
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-ext-1.11.jar
>>
>> java 165217 root mem REG 8,2 62983
>> 160861
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activation-1.1.jar
>>
>> java 165217 root mem REG 8,2 371668
>> 160812
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-security-core-3.2.9.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 645914
>> 160754
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-entitymanager-4.3.5.Final.jar
>>
>> java 165217 root mem REG 8,2 427030
>> 160753
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-envers-4.3.5.Final.jar
>>
>> java 165217 root mem REG 8,2 256468
>> 160829
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/barcode4j-2.0.jar
>>
>> java 165217 root mem REG 8,2 489884
>> 160846
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/log4j-1.2.17.jar
>> java 165217 root mem REG 8,2 29257
>> 160798
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.7.jar
>>
>> java 165217 root mem REG 8,2 422612
>> 160783
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-awt-util-1.11.jar
>>
>> java 165217 root mem REG 8,2 16519
>> 160796
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/jcl-over-slf4j-1.7.7.jar
>>
>> java 165217 root mem REG 8,2 322362
>> 160851
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/cglib-nodep-2.2.jar
>>
>> java 165217 root mem REG 8,2 864598
>> 160848
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/castor-1.2.jar
>> java 165217 root mem REG 8,2 434678
>> 160748
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang3-3.4.jar
>>
>> java 165217 root mem REG 8,2 325196
>> 160785
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-css-1.11.jar
>>
>> java 165217 root mem REG 8,2 5230007
>> 160755
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-core-4.3.5.Final.jar
>>
>> java 165217 root mem REG 8,2 14041249
>> 160763
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/elasticsearch-1.7.5.jar
>>
>> java 165217 root mem REG 8,2 206385
>> 160749
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-jms-3.2.17.RELEASE.jar
>>
>> java 165217 root mem REG 8,2 64021
>> 160837
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-join-4.10.4.jar
>>
>> java 165217 root mem REG 8,2 714194
>> 160828
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/javassist-3.18.1-GA.jar
>>
>> java 165217 root mem REG 8,2 279193
>> 160876
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang-2.5.jar
>>
>> java 165217 root mem REG 8,2 75311
>> 160826
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-commons-annotations-4.0.4.Final.jar
>>
>> java 165217 root mem REG 8,2 436303
>> 160745
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/avro-1.7.7.jar
>> ................
>> java 165217 root 73w FIFO 0,13 0t0
>> 16786125 pipe
>> java 165217 root 74u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 75r FIFO 0,13 0t0
>> 16786126 pipe
>> java 165217 root 76w FIFO 0,13 0t0
>> 16786126 pipe
>> java 165217 root 77u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 78u sock 0,9 0t0
>> 16961335 protocol: TCPv6
>> java 165217 root 79r FIFO 0,13 0t0
>> 16786127 pipe
>> java 165217 root 80w FIFO 0,13 0t0
>> 16786127 pipe
>> java 165217 root 81u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 82r FIFO 0,13 0t0
>> 16786128 pipe
>> java 165217 root 83w FIFO 0,13 0t0
>> 16786128 pipe
>> java 165217 root 84u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 86r FIFO 0,13 0t0
>> 16786129 pipe
>> java 165217 root 87w FIFO 0,13 0t0
>> 16786129 pipe
>> java 165217 root 88u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 89r FIFO 0,13 0t0
>> 16786130 pipe
>> java 165217 root 90w FIFO 0,13 0t0
>> 16786130 pipe
>> java 165217 root 91u a_inode 0,14 0
>> 10970 [eventpoll]
>> java 165217 root 93u IPv6 16786133 0t0
>> TCP 127.0.0.1:mxi (LISTEN)
>> java 165217 root 94u sock 0,9 0t0
>> 16963075 protocol: TCPv6
>> java 165217 root 96u sock 0,9 0t0
>> 16824649 protocol: TCPv6
>> java 165217 root 97u IPv6 17051898 0t0
>> TCP 10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)
>> java 165217 root 98u sock 0,9 0t0
>> 16824673 protocol: TCPv6
>> java 165217 root 99u sock 0,9 0t0
>> 16824711 protocol: TCPv6
>> java 165217 root 100u sock 0,9 0t0
>> 16893245 protocol: TCPv6
>> java 165217 root 101u sock 0,9 0t0
>> 16824385 protocol: TCPv6
>> java 165217 root 102u sock 0,9 0t0
>> 16824264 protocol: TCPv6
>> java 165217 root 103u sock 0,9 0t0
>> 16824278 protocol: TCPv6
>> java 165217 root 104u sock 0,9 0t0
>> 16824279 protocol: TCPv6
>> java 165217 root 105u sock 0,9 0t0
>> 16824063 protocol: TCPv6
>> java 165217 root 106u sock 0,9 0t0
>> 16823982 protocol: TCPv6
>> java 165217 root 107u sock 0,9 0t0
>> 16823959 protocol: TCPv6
>> java 165217 root 108r CHR 1,8 0t0
>> 1041 /dev/random
>> java 165217 root 109u sock 0,9 0t0
>> 16778709 protocol: TCPv6
>> java 165217 root 110w REG 8,6 25506176
>> 1835053 /apps/apache-tomcat/logs/fms.log
>> java 165217 root 111w REG 8,6 919042
>> 1835019 /apps/apache-tomcat/logs/obd.log
>> java 165217 root 112u sock 0,9 0t0
>> 16823821 protocol: TCPv6
>> java 165217 root 113u sock 0,9 0t0
>> 16823822 protocol: TCPv6
>> java 165217 root 114u sock 0,9 0t0
>> 16823633 protocol: TCPv6
>> java 165217 root 115u sock 0,9 0t0
>> 16823527 protocol: TCPv6
>> java 165217 root 116u sock 0,9 0t0
>> 16823489 protocol: TCPv6
>> java 165217 root 117u sock 0,9 0t0
>> 16823490 protocol: TCPv6
>> java 165217 root 118u sock 0,9 0t0
>> 16823491 protocol: TCPv6
>> ..................
>>
>> Thanks
>>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Tuesday, May 25, 2021 12:42 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Hi.
>> The point is to try to figure out what these thousands of apparently
>> "TCPv6" sockets
>> belonging to the tomcat process actually are, so that we can maybe
>> begin to look at where
>> they may be coming from.
>> The trouble is, the lsof output so far did not really tell us what
>> these "sock" things
>> might be.
>>
>> But there may be a clue here :
>> https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
>>
>> (about when things run in a container).
>> Is that your case ?
>> And if so, could you run the lsof command in the container, as they
>> suggest ?
>>
>> And the point of forcing a tomcat/JVM GC was this :
>> When you restart tomcat (actually the JVM which runs tomcat), the OS
>> will clean up *all*
>> the file descriptors belonging to that process, including the
>> "legitimate" ones shown by
>> netstat, and the "unknown" ones shown in addition by lsof.
>> Doing a GC, without stopping the JVM, would clean up *only* such
>> sockets/fd that are held
>> by objects which are discarded, but still sit on the heap awaiting a
>> GC to really destroy
>> them. If your heap is very large, it may otherwise take a long while
>> before such a GC
>> happens, and such sockets could accumulate.
>> One way to trigger a GC is through JMX, but it takes a bit of
>> additional setup to make
>> that work. That's why I was asking if you had some method to do that.
>> (see : https://code.google.com/archive/p/jmxsh/)
>> But let's look at the lsof part first.
>>
>>
>>
>> On 24.05.2021 16:09, Yeggy Javadi wrote:
>>> Hi,
>>> I restarted tomcat so PID has changed to 143152; I do not know how to
>>> trigger tomcat GC, I just restart it to reset the lsof to 0.
>>> Please see outputs below:
>>>
>>> # ps -ef | grep tomcat
>>> root 143152 1 0 May22 ? 00:26:44
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
>>> -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>> root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>
>>> # lsof -p 143152 | wc -l
>>> 41043
>>>
>>> # lsof -p 143152 | grep "protocol: TCPv6"| wc -l
>>> 40487
>>>
>>> # netstat -p -a -n --inet6 | grep 143152
>>> tcp6 0 0 :::8443 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 :::443 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 127.0.0.1:8005 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 :::1099 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 :::80 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 :::36081 :::*
>>> LISTEN 143152/java
>>> tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55446
>>> ESTABLISHED 143152/java
>>> tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576
>>> CLOSE_WAIT 143152/java
>>> tcp6 0 0 10.4.3.55:53682 172.22.21.48:443
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576
>>> CLOSE_WAIT 143152/java
>>> tcp6 0 0 10.4.3.55:40574 172.22.21.47:443
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:53782 172.22.21.48:443
>>> ESTABLISHED 143152/java
>>> tcp6 0 1 10.4.3.55:49808 10.12.3.78:443
>>> SYN_SENT 143152/java
>>> tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60226 172.22.22.192:443
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55444
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:40540 172.22.21.47:443
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432
>>> ESTABLISHED 143152/java
>>> tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>> tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300
>>> ESTABLISHED 143152/java
>>>
>>> # lsof -a -p 143152 -T s -i6
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>> java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
>>> java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
>>> java 143152 root 82u IPv6 13816325 0t0 TCP
>>> localhost.localdomain:mxi (LISTEN)
>>> java 143152 root 84u IPv6 13874431 0t0 TCP
>>> localhost.localdomain:52310->localhost.localdomain:postgres
>>> (ESTABLISHED)
>>> java 143152 root 86u IPv6 13848367 0t0 TCP
>>> localhost.localdomain:52312->localhost.localdomain:postgres
>>> (ESTABLISHED)
>>> java 143152 root 215u IPv6 13822102 0t0 TCP
>>> localhost.localdomain:48620->localhost.localdomain:postgres
>>> (ESTABLISHED)
>>> java 143152 root 216u IPv6 13823064 0t0 TCP
>>> localhost.localdomain:48624->localhost.localdomain:postgres
>>> (ESTABLISHED)
>>> java 143152 root 217u IPv6 13821107 0t0 TCP
>>> localhost.localdomain:48622->localhost.localdomain:postgres
>>> (ESTABLISHED)
>>> java 143152 root 269u IPv6 13824001 0t0 TCP
>>> localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
>>> java 143152 root 270u IPv6 13787907 0t0 TCP
>>> Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 271u IPv6 13787908 0t0 TCP
>>> Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 272u IPv6 13787909 0t0 TCP
>>> Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 273u IPv6 13787910 0t0 TCP
>>> Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 274u IPv6 13787911 0t0 TCP
>>> Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 275u IPv6 13787912 0t0 TCP
>>> Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 276u IPv6 13787913 0t0 TCP
>>> Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 277u IPv6 13787914 0t0 TCP
>>> Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 278u IPv6 13787915 0t0 TCP
>>> Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 279u IPv6 13787916 0t0 TCP
>>> Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 280u IPv6 13787917 0t0 TCP
>>> Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 281u IPv6 13787918 0t0 TCP
>>> Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 282u IPv6 13787919 0t0 TCP
>>> Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>>> java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry
>>> (LISTEN)
>>> java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
>>> java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https
>>> (LISTEN)
>>> java 143152 root *740u IPv6 15607907 0t0 TCP
>>> Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
>>> java 143152 root *876u IPv6 15608779 0t0 TCP
>>> Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
>>> java 143152 root *881u IPv6 15614275 0t0 TCP
>>> Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
>>> java 143152 root *882u IPv6 15558639 0t0 TCP
>>> Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
>>> java 143152 root *067u IPv6 15602583 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
>>> java 143152 root *070u IPv6 15602586 0t0 TCP
>>> Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
>>> java 143152 root *073u IPv6 15602582 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
>>> java 143152 root *080u IPv6 15512947 0t0 TCP
>>> Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>>> java 143152 root *081u IPv6 15512948 0t0 TCP
>>> Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>>> java 143152 root *817u IPv6 15611323 0t0 TCP
>>> Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
>>> java 143152 root *818u IPv6 15602584 0t0 TCP
>>> Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
>>>
>>> Thank
>>>
>>> -----Original Message-----
>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>> Sent: Sunday, May 23, 2021 6:30 PM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Hi.
>>>
>>> I have no idea what lines like this are :
>>>> java 130244 root 78u sock 0,9 0t0
>>>> 12154374 protocol: TCPv6
>>>
>>> There are obviously too many of them, for them to match with the
>>> sockets listed by netstat.
>>>
>>> The ones which in the lsof output, have "TYPE" set to "IPv6" seem to
>>> correspond to the
>>> ones marked as "LISTEN" in the netstat output.
>>> But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a
>>> mystery to me.
>>>
>>> Could you redo a netstat as follows :
>>> # netstat -p -a -n --inet6 | grep 130244
>>>
>>> and can you also try this form of lsof :
>>> # lsof -a -p 130244 -T s -i6
>>>
>>> And finally (after copying the result of the above) : do you know how
>>> to trigger a GC
>>> (Garbage Collection) in your tomcat JVM ?
>>> (the point is to see if when a GC happens, these things disappear).
>>>
>>>
>>> On 22.05.2021 18:03, Yeggy Javadi wrote:
>>>> Here it is:
>>>>
>>>> # netstat -p -a --tcp | grep 130244
>>>> tcp6 0 0 [::]:pcsync-https [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 [::]:https [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 [::]:37537 [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 localhost.localdoma:mxi [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 [::]:8009 [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 [::]:rmiregistry [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 0 0 [::]:http [::]:*
>>>> LISTEN 130244/java
>>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https
>>>> CLOSE_WAIT 130244/java
>>>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https
>>>> SYN_SENT 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https
>>>> SYN_SENT 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres
>>>> ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace
>>>> ESTABLISHED 130244/java
>>>>
>>>>
>>>> # lsof -p 130244
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF
>>>> NODE NAME
>>>> java 130244 root cwd DIR 8,2 4096
>>>> 157664 /usr/local/freestor/bin
>>>> java 130244 root rtd DIR 8,3
>>>> 4096 2 /
>>>> java 130244 root txt REG 8,2 8712
>>>> 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>>>> java 130244 root mem REG 8,2 498864
>>>> 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>>>> java 130244 root mem REG 8,2 39176
>>>> 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>>>> java 130244 root mem REG 8,2 759184
>>>> 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>>>> java 130244 root mem REG 8,2 3559360
>>>> 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>>> java 130244 root mem REG 8,2 99680
>>>> 133076 /usr/lib64/libgcc_s-8-20191121.so.1
>>>> java 130244 root mem REG 8,2 3135658
>>>> 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>>> java 130244 root mem REG 8,2 283368
>>>> 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>>>> java 130244 root mem REG 8,2 1907341
>>>> 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>>> java 130244 root mem REG 8,2 285868
>>>> 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>>> java 130244 root mem REG 8,2 43191
>>>> 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>>> java 130244 root mem REG 8,2 283206
>>>> 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>>> java 130244 root mem REG 8,2 118047
>>>> 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>>> java 130244 root mem REG 8,2 1179462
>>>> 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>>> java 130244 root mem REG 8,2 3860563
>>>> 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>>> java 130244 root mem REG 8,2 51984
>>>> 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>>>> java 130244 root mem REG 8,2 12373
>>>> 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>>> java 130244 root mem REG 8,2 169952
>>>> 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>>> java 130244 root mem REG 8,2 168382
>>>> 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>>> java 130244 root mem REG 8,2 244451
>>>> 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>>> java 130244 root mem REG 8,2 148761
>>>> 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>>> java 130244 root mem REG 8,2 62004
>>>> 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>>> java 130244 root mem REG 8,2 2450404
>>>> 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>>> java 130244 root mem REG 8,2 34798
>>>> 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>>> java 130244 root mem REG 8,2 78087
>>>> 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>>> java 130244 root mem REG 8,2 89695
>>>> 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>>> java 130244 root mem REG 8,2 599431
>>>> 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>>> java 130244 root mem REG 8,2 174393
>>>> 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>>> java 130244 root mem REG 8,2 294856
>>>> 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>>> java 130244 root mem REG 8,2 173282
>>>> 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>>> java 130244 root mem REG 8,2 284036
>>>> 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>>> java 130244 root mem REG 8,2 72656
>>>> 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>>> java 130244 root mem REG 8,2 54667
>>>> 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>>> java 130244 root mem REG 8,2 1705863
>>>> 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>>> java 130244 root mem REG 8,2 38241
>>>> 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>>> java 130244 root mem REG 8,2 106051
>>>> 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>>> java 130244 root mem REG 8,2 120073
>>>> 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>>> java 130244 root mem REG 8,2 228391
>>>> 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>>> java 130244 root mem REG 8,2 868336
>>>> 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>>> java 130244 root mem REG 8,2 215400
>>>> 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>>> java 130244 root mem REG 8,2 155094
>>>> 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>>> java 130244 root mem REG 8,2 26872
>>>> 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>>> java 130244 root mem REG 8,2 113008
>>>> 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>>>> java 130244 root mem REG 8,2 93872
>>>> 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>>>> java 130244 root mem REG 8,2 337024
>>>> 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
>>>> java 130244 root mem REG 8,2 2586930
>>>> 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
>>>> java 130244 root mem REG 8,2 10673
>>>> 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>>> java 130244 root mem REG 8,2 38561
>>>> 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>>> java 130244 root mem REG 8,2 158590
>>>> 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>>> java 130244 root mem REG 8,2 51205
>>>> 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>>> java 130244 root mem REG 8,2 25287
>>>> 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>>> java 130244 root mem REG 8,2 36132
>>>> 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>>> java 130244 root mem REG 8,2 54
>>>> 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
>>>> java 130244 root mem REG 8,2 3316
>>>> 521695 /usr/lib/locale/en_US.utf8/LC_TIME
>>>> java 130244 root mem REG 8,2 286
>>>> 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
>>>> java 130244 root mem REG 8,2 26998
>>>> 130155 /usr/lib64/gconv/gconv-modules.cache
>>>> java 130244 root mem REG 8,2 67053209
>>>> 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>>> java 130244 root mem REG 8,2 127016
>>>> 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>>>> java 130244 root mem REG 8,2 76880
>>>> 133068 /usr/lib64/libnss_files-2.28.so
>>>> java 130244 root mem REG 8,3 9253600
>>>> 129922 /var/lib/sss/mc/passwd
>>>> java 130244 root mem REG 8,2 46432
>>>> 134002 /usr/lib64/libnss_sss.so.2
>>>> java 130244 root mem REG 8,2 231960
>>>> 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>>>> java 130244 root mem REG 8,2 66112
>>>> 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>>>> java 130244 root mem REG 8,2 68944
>>>> 133091 /usr/lib64/librt-2.28.so
>>>> java 130244 root mem REG 8,2 2191808
>>>> 133023 /usr/lib64/libm-2.28.so
>>>> java 130244 root mem REG 8,2 17108824
>>>> 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>>>> java 130244 root mem REG 8,2 3154704
>>>> 129871 /usr/lib64/libc-2.28.so
>>>> java 130244 root mem REG 8,2 28968
>>>> 133013 /usr/lib64/libdl-2.28.so
>>>> java 130244 root mem REG 8,2 108616
>>>> 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>>>> java 130244 root mem REG 8,2 321552
>>>> 133079 /usr/lib64/libpthread-2.28.so
>>>> java 130244 root mem REG 8,2 252280
>>>> 129864 /usr/lib64/ld-2.28.so
>>>> java 130244 root mem REG 8,3 32768
>>>> 18 /tmp/hsperfdata_root/130244
>>>> java 130244 root mem REG 8,2 57
>>>> 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>>>> java 130244 root mem REG 8,2 34
>>>> 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
>>>> java 130244 root mem REG 8,2 77
>>>> 395974 /usr/lib/locale/en_US.utf8/LC_NAME
>>>> java 130244 root mem REG 8,2 167
>>>> 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
>>>> java 130244 root mem REG 8,2 59
>>>> 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>>>> java 130244 root mem REG 8,2 23
>>>> 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>>>> java 130244 root mem REG 8,2 368
>>>> 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>>>> java 130244 root 0r CHR 1,3 0t0
>>>> 1037 /dev/null
>>>> java 130244 root 1w REG 8,6 394654
>>>> 1835011 /apps/apache-tomcat/logs/catalina.out
>>>> java 130244 root 2w REG 8,6 394654
>>>> 1835011 /apps/apache-tomcat/logs/catalina.out
>>>> java 130244 root 3r REG 8,3 9253600
>>>> 129922 /var/lib/sss/mc/passwd
>>>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0
>>>> 11980864 type=STREAM
>>>> java 130244 root 5r REG 8,2 67053209
>>>> 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>>> java 130244 root 6r REG 8,2 36132
>>>> 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>>> java 130244 root 7r REG 8,2 25287
>>>> 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>>> java 130244 root 8r REG 8,2 51205
>>>> 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>>> java 130244 root 9w REG 8,6 21883
>>>> 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
>>>> java 130244 root 10w REG 8,6 761
>>>> 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
>>>> java 130244 root 11w REG 8,6 0
>>>> 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
>>>> java 130244 root 12w REG 8,6 0
>>>> 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>>>> java 130244 root 13r REG 8,2 158590
>>>> 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>>> java 130244 root 14r REG 8,2 38561
>>>> 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>>> java 130244 root 15r REG 8,2 26872
>>>> 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>>> java 130244 root 16r REG 8,2 155094
>>>> 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>>> java 130244 root 17r REG 8,2 215400
>>>> 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>>> java 130244 root 18r REG 8,2 868336
>>>> 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>>> java 130244 root 19r REG 8,2 228391
>>>> 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>>> java 130244 root 20r REG 8,2 120073
>>>> 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>>> java 130244 root 21r REG 8,2 106051
>>>> 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>>> java 130244 root 22r REG 8,2 38241
>>>> 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>>> java 130244 root 23r REG 8,2 1705863
>>>> 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>>> java 130244 root 24r REG 8,2 54667
>>>> 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>>> java 130244 root 25r REG 8,2 72656
>>>> 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>>> java 130244 root 26r REG 8,2 284036
>>>> 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>>> java 130244 root 27r REG 8,2 10673
>>>> 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>>> java 130244 root 28r REG 8,2 173282
>>>> 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>>> java 130244 root 29r REG 8,2 294856
>>>> 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>>> java 130244 root 30r REG 8,2 174393
>>>> 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>>> java 130244 root 31r REG 8,2 599431
>>>> 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>>> java 130244 root 32r REG 8,2 89695
>>>> 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>>> java 130244 root 33r REG 8,2 78087
>>>> 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>>> java 130244 root 34r REG 8,2 34798
>>>> 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>>> java 130244 root 35r REG 8,2 2450404
>>>> 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>>> java 130244 root 36r REG 8,2 62004
>>>> 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>>> java 130244 root 37r REG 8,2 148761
>>>> 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>>> java 130244 root 38r REG 8,2 244451
>>>> 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>>> java 130244 root 39r REG 8,2 168382
>>>> 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>>> java 130244 root 40r REG 8,2 169952
>>>> 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>>> java 130244 root 41r REG 8,2 12373
>>>> 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>>> java 130244 root 42r REG 8,2 1907341
>>>> 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>>> java 130244 root 43r REG 8,2 3860563
>>>> 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0
>>>> 11980871 type=STREAM
>>>> java 130244 root 45r REG 8,2 1179462
>>>> 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>>> java 130244 root 46r CHR 1,8 0t0
>>>> 1041 /dev/random
>>>> java 130244 root 47r CHR 1,9 0t0
>>>> 1042 /dev/urandom
>>>> java 130244 root 48r CHR 1,8 0t0
>>>> 1041 /dev/random
>>>> java 130244 root 49r CHR 1,8 0t0
>>>> 1041 /dev/random
>>>> java 130244 root 50r CHR 1,9 0t0
>>>> 1042 /dev/urandom
>>>> java 130244 root 51r CHR 1,9 0t0
>>>> 1042 /dev/urandom
>>>> java 130244 root 52r REG 8,2 118047
>>>> 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>>> java 130244 root 53r REG 8,2 283206
>>>> 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>>> java 130244 root 54r REG 8,2 43191
>>>> 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>>> java 130244 root 55r REG 8,2 285868
>>>> 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>>> java 130244 root 56r REG 8,2 3559360
>>>> 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>>> java 130244 root 57r REG 8,2 3135658
>>>> 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>>> java 130244 root 58u IPv6 11980876 0t0
>>>> TCP *:http (LISTEN)
>>>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0
>>>> 11980877 type=STREAM
>>>> java 130244 root 60r FIFO 0,13 0t0
>>>> 11980879 pipe
>>>> java 130244 root 61w FIFO 0,13 0t0
>>>> 11980879 pipe
>>>> java 130244 root 62u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 63u IPv6 11980880 0t0
>>>> TCP *:https (LISTEN)
>>>> java 130244 root 64r FIFO 0,13 0t0
>>>> 11980884 pipe
>>>> java 130244 root 65w FIFO 0,13 0t0
>>>> 11980884 pipe
>>>> java 130244 root 66u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 67u IPv6 11980885 0t0
>>>> TCP *:8009 (LISTEN)
>>>> java 130244 root 68r FIFO 0,13 0t0
>>>> 11980886 pipe
>>>> java 130244 root 69w FIFO 0,13 0t0
>>>> 11980886 pipe
>>>> java 130244 root 70u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 71w REG 8,6 342714
>>>> 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>>>> java 130244 root 72r FIFO 0,13 0t0
>>>> 11968845 pipe
>>>> java 130244 root 73w FIFO 0,13 0t0
>>>> 11968845 pipe
>>>> java 130244 root 74u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 75r FIFO 0,13 0t0
>>>> 11968846 pipe
>>>> java 130244 root 76w FIFO 0,13 0t0
>>>> 11968846 pipe
>>>> java 130244 root 77u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 78u sock 0,9 0t0
>>>> 12154374 protocol: TCPv6
>>>> java 130244 root 79r FIFO 0,13 0t0
>>>> 11968847 pipe
>>>> java 130244 root 80w FIFO 0,13 0t0
>>>> 11968847 pipe
>>>> java 130244 root 81u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 82r FIFO 0,13 0t0
>>>> 11968848 pipe
>>>> java 130244 root 83w FIFO 0,13 0t0
>>>> 11968848 pipe
>>>> java 130244 root 84u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 86r FIFO 0,13 0t0
>>>> 11968849 pipe
>>>> java 130244 root 87w FIFO 0,13 0t0
>>>> 11968849 pipe
>>>> java 130244 root 88u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 89r FIFO 0,13 0t0
>>>> 11968850 pipe
>>>> java 130244 root 90w FIFO 0,13 0t0
>>>> 11968850 pipe
>>>> java 130244 root 91u a_inode 0,14 0
>>>> 10970 [eventpoll]
>>>> java 130244 root 93u IPv6 11968853 0t0
>>>> TCP localhost.localdomain:mxi (LISTEN)
>>>> java 130244 root 94u IPv6 12298916 0t0
>>>> TCP localhost.localdomain:45734->localhost.localdomain:postgres
>>>> (ESTABLISHED)
>>>> java 130244 root 95u sock 0,9 0t0
>>>> 12017983 protocol: TCPv6
>>>> java 130244 root 96u sock 0,9 0t0
>>>> 11983086 protocol: TCPv6
>>>> java 130244 root 97u sock 0,9 0t0
>>>> 11983087 protocol: TCPv6
>>>> java 130244 root 98u sock 0,9 0t0
>>>> 11983088 protocol: TCPv6
>>>> java 130244 root 99u sock 0,9 0t0
>>>> 11983089 protocol: TCPv6
>>>> java 130244 root 100u sock 0,9 0t0
>>>> 11983090 protocol: TCPv6
>>>> java 130244 root 101u sock 0,9 0t0
>>>> 11983091 protocol: TCPv6
>>>> java 130244 root 102u sock 0,9 0t0
>>>> 11983092 protocol: TCPv6
>>>> java 130244 root 103u sock 0,9 0t0
>>>> 11969375 protocol: TCPv6
>>>> java 130244 root 104u sock 0,9 0t0
>>>> 11987339 protocol: TCPv6
>>>> java 130244 root 105u sock 0,9 0t0
>>>> 11987347 protocol: TCPv6
>>>> java 130244 root 106u sock 0,9 0t0
>>>> 11969384 protocol: TCPv6
>>>> java 130244 root 107u sock 0,9 0t0
>>>> 11987348 protocol: TCPv6
>>>> java 130244 root 108r CHR 1,8 0t0
>>>> 1041 /dev/random
>>>> java 130244 root 109w REG 8,6 19536380
>>>> 1835053 /apps/apache-tomcat/logs/fms.log
>>>> java 130244 root 110u sock 0,9 0t0
>>>> 11979291 protocol: TCPv6
>>>> java 130244 root 111u sock 0,9 0t0
>>>> 11979292 protocol: TCPv6
>>>> java 130244 root 112w REG 8,6 835408
>>>> 1835019 /apps/apache-tomcat/logs/obd.log
>>>> java 130244 root 113u sock 0,9 0t0
>>>> 11987349 protocol: TCPv6
>>>> java 130244 root 114u sock 0,9 0t0
>>>> 11987350 protocol: TCPv6
>>>> java 130244 root 115u sock 0,9 0t0
>>>> 11987351 protocol: TCPv6
>>>> java 130244 root 116u sock 0,9 0t0
>>>> 11987352 protocol: TCPv6
>>>> java 130244 root 117u sock 0,9 0t0
>>>> 11987353 protocol: TCPv6
>>>> java 130244 root 118u sock 0,9 0t0
>>>> 11987354 protocol: TCPv6
>>>> java 130244 root 119u sock 0,9 0t0
>>>> 11987355 protocol: TCPv6
>>>> java 130244 root 120u sock 0,9 0t0
>>>> 11987356 protocol: TCPv6
>>>> java 130244 root 121u sock 0,9 0t0
>>>> 11987357 protocol: TCPv6
>>>> java 130244 root 122u sock 0,9 0t0
>>>> 11987358 protocol: TCPv6
>>>> java 130244 root 123u sock 0,9 0t0
>>>> 11987359 protocol: TCPv6
>>>> java 130244 root 124u sock 0,9 0t0
>>>> 11987360 protocol: TCPv6
>>>> ...........................
>>>> Keep seeing the same line
>>>>
>>>> -----Original Message-----
>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>> Sent: Saturday, May 22, 2021 10:22 AM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Mmm. Nothing very special in that netstat output.
>>>> The sockets seen there look quite normal for tomcat, and there are
>>>> not a lot.
>>>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>>>
>>>> Looks like lsof is counting things which are not IPv6 TCP sockets
>>>> belonging to the tomcat JVM process.
>>>> Where is the difference between the lsof count (19948) and the
>>>> netstat count (25) ?
>>>> (genuine question, I'm puzzled too)
>>>>
>>>> Can you give an example of the "lsof -p 130244" output lines ?
>>>> (not all 19948 please, but enough to see some variety)
>>>>
>>>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>>>> Hi,
>>>>> Here its is:
>>>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>> 19948
>>>>>
>>>>> # netstat -p -a -6 --tcp | grep 130244
>>>>> tcp6 0 0 [::]:pcsync-https [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 [::]:https [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 [::]:37537 [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 localhost.localdoma:mxi [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 [::]:8009 [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 [::]:rmiregistry [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 0 0 [::]:http [::]:*
>>>>> LISTEN 130244/java
>>>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https
>>>>> CLOSE_WAIT 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798
>>>>> ESTABLISHED 130244/java
>>>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace
>>>>> ESTABLISHED 130244/java
>>>>> Thanks
>>>>> -----Original Message-----
>>>>> From: Noelette Stout <no...@isu.edu>
>>>>> Sent: Friday, May 21, 2021 8:28 AM
>>>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> ss has all the same options as netstat
>>>>>
>>>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>>>> <aw...@ice-sa.com>
>>>>> wrote:
>>>>>
>>>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>>>
>>>>>> The next thing that you may want to do :
>>>>>>
>>>>>> > netstat -p -a -6 --tcp
>>>>>>
>>>>>> That is an alternative list of sockets, which also shows the "tcp
>>>>>> state"
>>>>>> of the sockets.
>>>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>>>> the last column.
>>>>>> The type of thing you are looking for is the column which should show
>>>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>>>
>>>>>> The options above :
>>>>>> -p : show PID and program
>>>>>> -a : show all sockets states
>>>>>> -6 : only inet v6
>>>>>> --tcp : only TCP sockets
>>>>>>
>>>>>> "netstat" may not be on your system by default, and you may need to
>>>>>> install it.
>>>>>> An alternative is "ss", but I don't know the options.
>>>>>>
>>>>>>
>>>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>>>> Hi,
>>>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>>>> 7474
>>>>>> are for TCPv6 sockets:
>>>>>>> java 130244 root 7805u sock 0,9 0t0
>>>>>>> 12294251
>>>>>> protocol: TCPv6
>>>>>>>
>>>>>>> # ps -ef | grep tomcat
>>>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>>
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep
>>>>>>> --color=auto
>>>>>> tomcat
>>>>>>>
>>>>>>> #lsof -p 130244 | wc -l
>>>>>>> 8028
>>>>>>>
>>>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>>>> 7474
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Hi.
>>>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>>>> on
>>>>>> your system.
>>>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>>>> when
>>>>>> you look at the whole command-line.
>>>>>>>
>>>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>>>> relevant here.
>>>>>>>
>>>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>>>> files opened by the tomcat JVM.
>>>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>>>> "apache-tomcat"
>>>>>>> (or anything else unique in that line)
>>>>>>>
>>>>>>>
>>>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>>>> Hi Chris,
>>>>>>>> Please indicate how to show lsof or netstat to *just the JVM*
>>>>>>>> process.
>>>>>>>>
>>>>>>>> Below is the list of running processes on my server:
>>>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>>>> root 1 0 0 May07 ? 00:00:14
>>>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>>>> root 6 2 0 May07 ? 00:00:00
>>>>>> [kworker/0:0H-kblockd]
>>>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>>>> root 19 2 0 May07 ? 00:00:00
>>>>>> [kworker/1:0H-kblockd]
>>>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>>>> root 25 2 0 May07 ? 00:00:00
>>>>>> [kworker/2:0H-kblockd]
>>>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>>>> root 31 2 0 May07 ? 00:00:00
>>>>>> [kworker/3:0H-kblockd]
>>>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>>>> root 37 2 0 May07 ? 00:00:00
>>>>>> [kworker/4:0H-kblockd]
>>>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>>>> root 43 2 0 May07 ? 00:00:00
>>>>>> [kworker/5:0H-kblockd]
>>>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>>>> root 49 2 0 May07 ? 00:00:00
>>>>>> [kworker/6:0H-kblockd]
>>>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>>>> root 55 2 0 May07 ? 00:00:00
>>>>>> [kworker/7:0H-kblockd]
>>>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>>>> root 222 2 0 May07 ? 00:00:00
>>>>>>>> [acpi_thermal_pm]
>>>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>>>> root 540 2 0 May07 ? 00:00:01
>>>>>> [kworker/7:1H-kblockd]
>>>>>>>> root 541 2 0 May07 ? 00:00:01
>>>>>> [kworker/5:1H-kblockd]
>>>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>>>> root 568 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>>>> root 612 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 613 2 0 May07 ? 00:00:01
>>>>>> [kworker/6:1H-kblockd]
>>>>>>>> root 650 2 0 May07 ? 00:00:01
>>>>>> [kworker/4:1H-kblockd]
>>>>>>>> root 659 2 0 May07 ? 00:00:02
>>>>>> [kworker/3:1H-kblockd]
>>>>>>>> root 660 2 0 May07 ? 00:00:02
>>>>>> [kworker/2:1H-kblockd]
>>>>>>>> root 662 2 0 May07 ? 00:00:01
>>>>>> [kworker/0:1H-kblockd]
>>>>>>>> root 679 1 0 May07 ? 00:00:03
>>>>>> /usr/lib/systemd/systemd-journald
>>>>>>>> root 682 2 0 May07 ? 00:00:02
>>>>>> [kworker/1:1H-kblockd]
>>>>>>>> root 716 1 0 May07 ? 00:00:01
>>>>>> /usr/lib/systemd/systemd-udevd
>>>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>>>> root 724 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>>>> root 734 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>>>> root 739 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>>>> root 745 2 0 May07 ? 00:00:00
>>>>>>>> [ext4-rsv-conver]
>>>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>>>> --systemd-activation --syslog-only
>>>>>>>> root 885 1 0 May07 ? 00:02:42
>>>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>>>> root 887 1 0 May07 ? 00:00:44
>>>>>> /usr/sbin/irqbalance --foreground
>>>>>>>> root 888 1 0 May07 ? 00:00:00
>>>>>> /usr/bin/VGAuthService -s
>>>>>>>> root 889 1 0 May07 ? 00:10:28
>>>>>>>> /usr/bin/vmtoolsd
>>>>>>>> root 890 1 0 May07 ? 00:00:00
>>>>>>>> /usr/sbin/sssd -i
>>>>>> --logger=files
>>>>>>>> chrony 906 1 0 May07 ? 00:00:00
>>>>>>>> /usr/sbin/chronyd
>>>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>>>> --fill-watermark=0
>>>>>>>> root 946 890 0 May07 ? 00:00:06
>>>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>>>> --logger=files
>>>>>>>> root 954 1 0 May07 ? 00:00:00
>>>>>>>> /usr/sbin/iscsid -f
>>>>>> -d2
>>>>>>>> root 962 890 0 May07 ? 00:00:12
>>>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>>>> root 963 1 0 May07 ? 00:00:00
>>>>>>>> /usr/sbin/gssproxy
>>>>>> -D
>>>>>>>> root 970 1 0 May07 ? 00:00:02
>>>>>> /usr/lib/systemd/systemd-logind
>>>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>>>> root 998 1 0 May07 ? 00:00:33
>>>>>>>> /usr/sbin/rsyslogd
>>>>>> -n
>>>>>>>> root 1007 1 0 May07 ? 00:00:00
>>>>>>>> /usr/sbin/crond -n
>>>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>>>> /usr/local/vtl/bin/sanidmgr
>>>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>>>> root 6539 2 0 May07 ? 00:00:00
>>>>>>>> [fsnalias_worker]
>>>>>>>> root 6541 2 0 May07 ? 00:00:00
>>>>>>>> [fsnalias_check]
>>>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>>>> /usr/local/vtl/bin/iscliproxy
>>>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>>>> /usr/local/vtl/bin/ipstorlog
>>>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>>>> /usr/local/vtl/bin/snmpd
>>>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>>>> /usr/lib/systemd/systemd --user
>>>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty
>>>>>>>> -o -p
>>>>>> -- \u --noclear tty1 linux
>>>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root
>>>>>>>> [priv]
>>>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd:
>>>>>>>> root@pts/0
>>>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>>>> root 14261 1 0 May08 ? 00:00:00
>>>>>>>> /usr/sbin/sshd -D
>>>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>>>> -
>>>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>>>> h
>>>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>>>> p
>>>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>>>> -
>>>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>>>> a
>>>>>> 1,diffie-hellman-group14-sha1
>>>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>>> m
>>>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>>>> root 122905 2 0 May19 ? 00:00:27
>>>>>>>> [kworker/7:0-events]
>>>>>>>> root 126037 2 0 05:30 ? 00:00:00
>>>>>>>> [kworker/1:2-events]
>>>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>>>> [kworker/4:1-events_power_efficient]
>>>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>>>> [kworker/4:2-cgroup_destroy]
>>>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>>>> [kworker/6:0-mm_percpu_wq]
>>>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>>>> [kworker/3:3-mm_percpu_wq]
>>>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>>>> [kworker/2:1-mm_percpu_wq]
>>>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>>
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java
>>>>>>>> -Xms8g
>>>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>>>> -XX:+UseConcMarkSweepGC
>>>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>>>> t
>>>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>>>> -Des.default.path.conf=/etc/elasticsearch
>>>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres:
>>>>>>>> logger
>>>>>> process
>>>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>>>> checkpointer process
>>>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres:
>>>>>>>> writer
>>>>>> process
>>>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>>>> writer process
>>>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>>>> autovacuum launcher process
>>>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>>>> collector process
>>>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>>>> /usr/local/freestor/bin/fmsd
>>>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres:
>>>>>>>> postgres
>>>>>> freestor 127.0.0.1(42660) idle
>>>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres:
>>>>>>>> postgres
>>>>>> freestor 127.0.0.1(42662) idle
>>>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres:
>>>>>>>> postgres
>>>>>> freestor 127.0.0.1(42664) idle
>>>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>>>> [kworker/5:1-mm_percpu_wq]
>>>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>>>> [kworker/u16:2-events_unbound]
>>>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>>>> [kworker/u16:1-events_unbound]
>>>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>>>> root 131465 2 0 14:30 ? 00:00:00
>>>>>>>> [kworker/0:2-events]
>>>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>>>> root 131491 2 0 14:40 ? 00:00:00
>>>>>>>> [kworker/0:1-events]
>>>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>>>> [kworker/1:0-events_freezable]
>>>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root
>>>>>>>> [priv]
>>>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd:
>>>>>>>> root@pts/1
>>>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> -----Original Message-----
>>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>>>> To: users@tomcat.apache.org
>>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>>
>>>>>>>> Yeggy,
>>>>>>>>
>>>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>>>> Hi Chris,
>>>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>>>
>>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>>>> 9.3
>>>>>> to 9.6.
>>>>>>>>>
>>>>>>>>> 2. What is error you are seeing? A full stack trace would be
>>>>>>>>> helpful.
>>>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>>>> many
>>>>>> open files) error when the limit of open files for tomcat has reached
>>>>>> (262144 in my environment), for example, I can get a connectivity
>>>>>> error when pulling info from a server as below :
>>>>>>>>>
>>>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>>>>
>>>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>>>
>>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>>>
>>>>>>>>> <Connector address="0.0.0.0" port="80"
>>>>>>>>> protocol="HTTP/1.1"
>>>>>>>>> maxThreads="150"
>>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>>>> connectionTimeout="20000"
>>>>>>>>> disableUploadTimeout="true"
>>>>>>>>> compression="on"
>>>>>>>>> compressionMinSize="2048"
>>>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>>>
>>>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>>>> redirectPort="443" /> ...........
>>>>>>>>> <Connector address="0.0.0.0" port="443"
>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>> SSLEnabled="true"
>>>>>>>>> maxThreads="150" scheme="https"
>>>>>>>>> secure="true"
>>>>>>>>> clientAuth="false"
>>>>>>>>> sslEnabledProtocols="TLSv1.2"
>>>>>>>>>
>>>>>>
>>>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>
>>>>>>>>> keystoreFile="conf/freestor"
>>>>>> keystorePass="freestor"/>
>>>>>>>>>
>>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>>> <Connector address="0.0.0.0" port="8009"
>>>>>>>>> protocol="AJP/1.3"
>>>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>>>>
>>>>>>>> You have lots of settings on the HTTP connector but not the
>>>>>>>> HTTPS one.
>>>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>>>
>>>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>>>> Definitely
>>>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>>>
>>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>>> of
>>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>>> Different kernel versions count things differently.
>>>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>>>> files for that process
>>>>>>>>
>>>>>>>> Ok.
>>>>>>>>
>>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>>> [YJ] Here is the netstat output:
>>>>>>>>> Active Internet connections (w/o servers)
>>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>>>> State
>>>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>>>> ESTABLISHED
>>>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>>>> ESTABLISHED
>>>>>>>>> tcp 0 0 localhost.loca:postgres
>>>>>>>>> localhost.localdo:36846
>>>>>> ESTABLISHED
>>>>>>>>
>>>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>>>> not
>>>>>> what the whole OS is handling.
>>>>>>>>
>>>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>>>> the
>>>>>> process. But if you get "too many open files" being thrown inside the
>>>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>>>> process, then something else is going on. But remember to limit your
>>>>>> lsof to *just the JVM* process.
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>>>> To: users@tomcat.apache.org
>>>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>>>
>>>>>>>>> Yeggy,
>>>>>>>>>
>>>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle
>>>>>>>>>> Linux 8:
>>>>>>>>>> java 51798 root 6601u sock 0,9 0t0
>>>>>>>>>> 3959342
>>>>>> protocol: TCPv6
>>>>>>>>>>
>>>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>>>> changes in Linux
>>>>>>>>>> 8 that can explain this
>>>>>>>>>>
>>>>>>>>>> Configuration:
>>>>>>>>>> apache-tomcat-8.5.59
>>>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # ps -ef | grep tomcat
>>>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>>>> /usr/local/jre/bin/java
>>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>>>
>>>>>> properties
>>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>>> -Dignore.endorsed.dirs= -classpath
>>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep
>>>>>>>>>> --color=auto
>>>>>> tomcat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # cat /proc/51798/limits
>>>>>>>>>> Limit Soft Limit Hard Limit
>>>>>> Units
>>>>>>>>>> Max cpu time unlimited unlimited
>>>>>> seconds
>>>>>>>>>> Max file size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max data size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max stack size 8388608 unlimited
>>>>>> bytes
>>>>>>>>>> Max core file size unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max resident set unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max processes 256518 256518
>>>>>> processes
>>>>>>>>>> Max open files 262144 262144
>>>>>> files
>>>>>>>>>> Max locked memory 65536 65536
>>>>>> bytes
>>>>>>>>>> Max address space unlimited unlimited
>>>>>> bytes
>>>>>>>>>> Max file locks unlimited unlimited
>>>>>> locks
>>>>>>>>>> Max pending signals 256518 256518
>>>>>> signals
>>>>>>>>>> Max msgqueue size 819200 819200
>>>>>> bytes
>>>>>>>>>> Max nice priority 0 0
>>>>>>>>>> Max realtime priority 0 0
>>>>>>>>>> Max realtime timeout unlimited
>>>>>>>>>> unlimited us
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Too may open files:
>>>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>>>> 6679
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Most of open files are TCP sockets:
>>>>>>>>>> java 51798 root 6601u sock 0,9 0t0
>>>>>>>>>> 3959342
>>>>>> protocol: TCPv6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>>>> 6413
>>>>>>>>>
>>>>>>>>> Some potentially dumb questions:
>>>>>>>>>
>>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>>>
>>>>>>>>> 2. What is error you are seeing? A full stack trace would be
>>>>>>>>> helpful.
>>>>>>>>>
>>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>>>
>>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>>> of
>>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>>> Different kernel versions count things differently.
>>>>>>>>>
>>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>>>
>>>>>>>>> -chris
>>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Noelette Stout
>>>>>
>>>>> ITS Enterprise Applications - Application Administrator - Senior
>>>>>
>>>>> Business Administration Building, Rm 109L
>>>>> 921 South 8th Ave 8037
>>>>> Idaho State University
>>>>> Pocatello ID 83209
>>>>> E-mail: stounoel "at" isu "dot" edu
>>>>> Desk: 208-282-2554
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Maybe I am missing something, but at first sight it looks like lsof, inside the container,
can also not get more information about these "sock" things.
I am running out of ideas.
If you search in Google for precisely this :
lsof "sock" and "protocol : TCP"
there are a lot of links which discuss similar issues, and this over many years.
(So it is not either a unique or a recent issue).
They all seem to boil down to this : some *application* is opening sockets, but then not
really using them (and not closing them).
In your case, that probably means one of the webapps running under tomcat.
It seems quite unlikely that this would be tomcat itself, because if that was the case,
this tomcat users list would probably be swamped with requests such as yours; which it isn't.
It is worth noting also, that among all these messages found in Google, I have not so far
seen a single one which provides another explanation for those "sock" things.
In your case, the problem is going to be in finding out *which* webapp does that, unless
there are not many, and you can turn them off one-by-one selectively.
(The difficulty is in part due to the fact that, to the OS, the whole of the JVM, tomcat
and all the webapps look like one single process; and to lsof also).
Maybe there is some type of logging available in tomcat, that would help finding out which
application is creating sockets, and then never using or destroying them.
But my personal competences do not run that far, so maybe someone else can help you here.
On 26.05.2021 00:03, Yeggy Javadi wrote:
> Hi,
> Below is the nsenter output:
>
> # ps -ef | grep tomcat
> root 165217 1 1 10:22 ? 00:05:33 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 167329 167268 0 18:00 pts/1 00:00:00 grep --color=auto tomcat
>
> # nsenter -t 165217 --net lsof -n -p 165217 |less
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 165217 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
> java 165217 root rtd DIR 8,3 4096 2 /
> java 165217 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
> java 165217 root mem REG 8,2 113371 160881 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-jpa-2.1-api-1.0.0.Final.jar
> java 165217 root mem REG 8,2 147874 160802 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activemq-protobuf-1.1.jar
> java 165217 root mem REG 8,2 391515 160836 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-queryparser-4.10.4.jar
> java 165217 root mem REG 8,2 868615 160813 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-context-3.2.17.RELEASE.jar
> java 165217 root mem REG 8,2 9711 160792 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.6.6.jar
> java 165217 root mem REG 8,2 196573 160819 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-expression-3.2.17.RELEASE.jar
> java 165217 root mem REG 8,2 97173 160843 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-misc-4.10.4.jar
> java 165217 root mem REG 8,2 10074 160872 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-ext-1.11.jar
> java 165217 root mem REG 8,2 62983 160861 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activation-1.1.jar
> java 165217 root mem REG 8,2 371668 160812 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-security-core-3.2.9.RELEASE.jar
> java 165217 root mem REG 8,2 645914 160754 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-entitymanager-4.3.5.Final.jar
> java 165217 root mem REG 8,2 427030 160753 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-envers-4.3.5.Final.jar
> java 165217 root mem REG 8,2 256468 160829 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/barcode4j-2.0.jar
> java 165217 root mem REG 8,2 489884 160846 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/log4j-1.2.17.jar
> java 165217 root mem REG 8,2 29257 160798 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.7.jar
> java 165217 root mem REG 8,2 422612 160783 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-awt-util-1.11.jar
> java 165217 root mem REG 8,2 16519 160796 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/jcl-over-slf4j-1.7.7.jar
> java 165217 root mem REG 8,2 322362 160851 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/cglib-nodep-2.2.jar
> java 165217 root mem REG 8,2 864598 160848 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/castor-1.2.jar
> java 165217 root mem REG 8,2 434678 160748 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang3-3.4.jar
> java 165217 root mem REG 8,2 325196 160785 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-css-1.11.jar
> java 165217 root mem REG 8,2 5230007 160755 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-core-4.3.5.Final.jar
> java 165217 root mem REG 8,2 14041249 160763 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/elasticsearch-1.7.5.jar
> java 165217 root mem REG 8,2 206385 160749 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-jms-3.2.17.RELEASE.jar
> java 165217 root mem REG 8,2 64021 160837 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-join-4.10.4.jar
> java 165217 root mem REG 8,2 714194 160828 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/javassist-3.18.1-GA.jar
> java 165217 root mem REG 8,2 279193 160876 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang-2.5.jar
> java 165217 root mem REG 8,2 75311 160826 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-commons-annotations-4.0.4.Final.jar
> java 165217 root mem REG 8,2 436303 160745 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/avro-1.7.7.jar
> ................
> java 165217 root 73w FIFO 0,13 0t0 16786125 pipe
> java 165217 root 74u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 75r FIFO 0,13 0t0 16786126 pipe
> java 165217 root 76w FIFO 0,13 0t0 16786126 pipe
> java 165217 root 77u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 78u sock 0,9 0t0 16961335 protocol: TCPv6
> java 165217 root 79r FIFO 0,13 0t0 16786127 pipe
> java 165217 root 80w FIFO 0,13 0t0 16786127 pipe
> java 165217 root 81u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 82r FIFO 0,13 0t0 16786128 pipe
> java 165217 root 83w FIFO 0,13 0t0 16786128 pipe
> java 165217 root 84u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 86r FIFO 0,13 0t0 16786129 pipe
> java 165217 root 87w FIFO 0,13 0t0 16786129 pipe
> java 165217 root 88u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 89r FIFO 0,13 0t0 16786130 pipe
> java 165217 root 90w FIFO 0,13 0t0 16786130 pipe
> java 165217 root 91u a_inode 0,14 0 10970 [eventpoll]
> java 165217 root 93u IPv6 16786133 0t0 TCP 127.0.0.1:mxi (LISTEN)
> java 165217 root 94u sock 0,9 0t0 16963075 protocol: TCPv6
> java 165217 root 96u sock 0,9 0t0 16824649 protocol: TCPv6
> java 165217 root 97u IPv6 17051898 0t0 TCP 10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)
> java 165217 root 98u sock 0,9 0t0 16824673 protocol: TCPv6
> java 165217 root 99u sock 0,9 0t0 16824711 protocol: TCPv6
> java 165217 root 100u sock 0,9 0t0 16893245 protocol: TCPv6
> java 165217 root 101u sock 0,9 0t0 16824385 protocol: TCPv6
> java 165217 root 102u sock 0,9 0t0 16824264 protocol: TCPv6
> java 165217 root 103u sock 0,9 0t0 16824278 protocol: TCPv6
> java 165217 root 104u sock 0,9 0t0 16824279 protocol: TCPv6
> java 165217 root 105u sock 0,9 0t0 16824063 protocol: TCPv6
> java 165217 root 106u sock 0,9 0t0 16823982 protocol: TCPv6
> java 165217 root 107u sock 0,9 0t0 16823959 protocol: TCPv6
> java 165217 root 108r CHR 1,8 0t0 1041 /dev/random
> java 165217 root 109u sock 0,9 0t0 16778709 protocol: TCPv6
> java 165217 root 110w REG 8,6 25506176 1835053 /apps/apache-tomcat/logs/fms.log
> java 165217 root 111w REG 8,6 919042 1835019 /apps/apache-tomcat/logs/obd.log
> java 165217 root 112u sock 0,9 0t0 16823821 protocol: TCPv6
> java 165217 root 113u sock 0,9 0t0 16823822 protocol: TCPv6
> java 165217 root 114u sock 0,9 0t0 16823633 protocol: TCPv6
> java 165217 root 115u sock 0,9 0t0 16823527 protocol: TCPv6
> java 165217 root 116u sock 0,9 0t0 16823489 protocol: TCPv6
> java 165217 root 117u sock 0,9 0t0 16823490 protocol: TCPv6
> java 165217 root 118u sock 0,9 0t0 16823491 protocol: TCPv6
> ..................
>
> Thanks
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Tuesday, May 25, 2021 12:42 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Hi.
> The point is to try to figure out what these thousands of apparently "TCPv6" sockets
> belonging to the tomcat process actually are, so that we can maybe begin to look at where
> they may be coming from.
> The trouble is, the lsof output so far did not really tell us what these "sock" things
> might be.
>
> But there may be a clue here :
> https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
> (about when things run in a container).
> Is that your case ?
> And if so, could you run the lsof command in the container, as they suggest ?
>
> And the point of forcing a tomcat/JVM GC was this :
> When you restart tomcat (actually the JVM which runs tomcat), the OS will clean up *all*
> the file descriptors belonging to that process, including the "legitimate" ones shown by
> netstat, and the "unknown" ones shown in addition by lsof.
> Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd that are held
> by objects which are discarded, but still sit on the heap awaiting a GC to really destroy
> them. If your heap is very large, it may otherwise take a long while before such a GC
> happens, and such sockets could accumulate.
> One way to trigger a GC is through JMX, but it takes a bit of additional setup to make
> that work. That's why I was asking if you had some method to do that.
> (see : https://code.google.com/archive/p/jmxsh/)
> But let's look at the lsof part first.
>
>
>
> On 24.05.2021 16:09, Yeggy Javadi wrote:
>> Hi,
>> I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat GC, I just restart it to reset the lsof to 0.
>> Please see outputs below:
>>
>> # ps -ef | grep tomcat
>> root 143152 1 0 May22 ? 00:26:44 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>> root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto tomcat
>>
>> # lsof -p 143152 | wc -l
>> 41043
>>
>> # lsof -p 143152 | grep "protocol: TCPv6"| wc -l
>> 40487
>>
>> # netstat -p -a -n --inet6 | grep 143152
>> tcp6 0 0 :::8443 :::* LISTEN 143152/java
>> tcp6 0 0 :::443 :::* LISTEN 143152/java
>> tcp6 0 0 127.0.0.1:8005 :::* LISTEN 143152/java
>> tcp6 0 0 :::1099 :::* LISTEN 143152/java
>> tcp6 0 0 :::80 :::* LISTEN 143152/java
>> tcp6 0 0 :::36081 :::* LISTEN 143152/java
>> tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55446 ESTABLISHED 143152/java
>> tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576 CLOSE_WAIT 143152/java
>> tcp6 0 0 10.4.3.55:53682 172.22.21.48:443 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576 CLOSE_WAIT 143152/java
>> tcp6 0 0 10.4.3.55:40574 172.22.21.47:443 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:53782 172.22.21.48:443 ESTABLISHED 143152/java
>> tcp6 0 1 10.4.3.55:49808 10.12.3.78:443 SYN_SENT 143152/java
>> tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60226 172.22.22.192:443 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:80 10.197.255.10:55444 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:40540 172.22.21.47:443 ESTABLISHED 143152/java
>> tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432 ESTABLISHED 143152/java
>> tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300 ESTABLISHED 143152/java
>> tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300 ESTABLISHED 143152/java
>>
>> # lsof -a -p 143152 -T s -i6
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
>> java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
>> java 143152 root 82u IPv6 13816325 0t0 TCP localhost.localdomain:mxi (LISTEN)
>> java 143152 root 84u IPv6 13874431 0t0 TCP localhost.localdomain:52310->localhost.localdomain:postgres (ESTABLISHED)
>> java 143152 root 86u IPv6 13848367 0t0 TCP localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
>> java 143152 root 215u IPv6 13822102 0t0 TCP localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
>> java 143152 root 216u IPv6 13823064 0t0 TCP localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
>> java 143152 root 217u IPv6 13821107 0t0 TCP localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
>> java 143152 root 269u IPv6 13824001 0t0 TCP localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
>> java 143152 root 270u IPv6 13787907 0t0 TCP Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 271u IPv6 13787908 0t0 TCP Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 272u IPv6 13787909 0t0 TCP Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 273u IPv6 13787910 0t0 TCP Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 274u IPv6 13787911 0t0 TCP Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 275u IPv6 13787912 0t0 TCP Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 276u IPv6 13787913 0t0 TCP Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 277u IPv6 13787914 0t0 TCP Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 278u IPv6 13787915 0t0 TCP Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 279u IPv6 13787916 0t0 TCP Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 280u IPv6 13787917 0t0 TCP Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 281u IPv6 13787918 0t0 TCP Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 282u IPv6 13787919 0t0 TCP Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
>> java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry (LISTEN)
>> java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
>> java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https (LISTEN)
>> java 143152 root *740u IPv6 15607907 0t0 TCP Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
>> java 143152 root *876u IPv6 15608779 0t0 TCP Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
>> java 143152 root *881u IPv6 15614275 0t0 TCP Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
>> java 143152 root *882u IPv6 15558639 0t0 TCP Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
>> java 143152 root *067u IPv6 15602583 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
>> java 143152 root *070u IPv6 15602586 0t0 TCP Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
>> java 143152 root *073u IPv6 15602582 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
>> java 143152 root *080u IPv6 15512947 0t0 TCP Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>> java 143152 root *081u IPv6 15512948 0t0 TCP Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
>> java 143152 root *817u IPv6 15611323 0t0 TCP Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
>> java 143152 root *818u IPv6 15602584 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
>>
>> Thank
>>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Sunday, May 23, 2021 6:30 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Hi.
>>
>> I have no idea what lines like this are :
>>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>>
>> There are obviously too many of them, for them to match with the sockets listed by netstat.
>>
>> The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
>> ones marked as "LISTEN" in the netstat output.
>> But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
>>
>> Could you redo a netstat as follows :
>> # netstat -p -a -n --inet6 | grep 130244
>>
>> and can you also try this form of lsof :
>> # lsof -a -p 130244 -T s -i6
>>
>> And finally (after copying the result of the above) : do you know how to trigger a GC
>> (Garbage Collection) in your tomcat JVM ?
>> (the point is to see if when a GC happens, these things disappear).
>>
>>
>> On 22.05.2021 18:03, Yeggy Javadi wrote:
>>> Here it is:
>>>
>>> # netstat -p -a --tcp | grep 130244
>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>>
>>>
>>> # lsof -p 130244
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
>>> java 130244 root rtd DIR 8,3 4096 2 /
>>> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>>> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>>> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>>> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>>> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
>>> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>>> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>>> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>>> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>>> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
>>> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
>>> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
>>> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
>>> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
>>> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
>>> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>>> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
>>> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>>> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
>>> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>>> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>>> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
>>> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
>>> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>>> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
>>> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
>>> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>>> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
>>> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
>>> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
>>> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>>> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
>>> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
>>> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
>>> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>>> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>>> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>>> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
>>> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>>> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>>> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
>>> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>>> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>>> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>>> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>>> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
>>> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
>>> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
>>> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>>> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>>> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>>> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>>> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>>> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>>> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>>> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>>> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>>> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>>> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>>> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>>> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>>> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>>> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>>> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>>> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>>> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>>> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>>> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>>> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>>> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>>> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>>> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>>> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>>> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>>> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>>> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>>> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>>> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>>> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>>> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
>>> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>>> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
>>> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
>>> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
>>> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
>>> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
>>> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
>>> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>>> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>>> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>>> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>>> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>>> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>>> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
>>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
>>> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
>>> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
>>> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
>>> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
>>> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
>>> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
>>> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
>>> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
>>> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>>> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
>>> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
>>> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
>>> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
>>> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>>> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
>>> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
>>> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
>>> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
>>> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
>>> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
>>> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
>>> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
>>> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
>>> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
>>> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>>> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
>>> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
>>> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
>>> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
>>> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
>>> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
>>> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
>>> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
>>> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
>>> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
>>> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
>>> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
>>> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
>>> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
>>> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
>>> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
>>> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
>>> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
>>> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
>>> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
>>> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
>>> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
>>> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
>>> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
>>> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
>>> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
>>> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
>>> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
>>> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
>>> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
>>> ...........................
>>> Keep seeing the same line
>>>
>>> -----Original Message-----
>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>> Sent: Saturday, May 22, 2021 10:22 AM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Mmm. Nothing very special in that netstat output.
>>> The sockets seen there look quite normal for tomcat, and there are not a lot.
>>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>>
>>> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
>>> Where is the difference between the lsof count (19948) and the netstat count (25) ?
>>> (genuine question, I'm puzzled too)
>>>
>>> Can you give an example of the "lsof -p 130244" output lines ?
>>> (not all 19948 please, but enough to see some variety)
>>>
>>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>>> Hi,
>>>> Here its is:
>>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>> 19948
>>>>
>>>> # netstat -p -a -6 --tcp | grep 130244
>>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>>>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>>>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>>>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>>>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>>> Thanks
>>>> -----Original Message-----
>>>> From: Noelette Stout <no...@isu.edu>
>>>> Sent: Friday, May 21, 2021 8:28 AM
>>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> ss has all the same options as netstat
>>>>
>>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>>> <aw...@ice-sa.com>
>>>> wrote:
>>>>
>>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>>
>>>>> The next thing that you may want to do :
>>>>>
>>>>> > netstat -p -a -6 --tcp
>>>>>
>>>>> That is an alternative list of sockets, which also shows the "tcp state"
>>>>> of the sockets.
>>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>>> the last column.
>>>>> The type of thing you are looking for is the column which should show
>>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>>
>>>>> The options above :
>>>>> -p : show PID and program
>>>>> -a : show all sockets states
>>>>> -6 : only inet v6
>>>>> --tcp : only TCP sockets
>>>>>
>>>>> "netstat" may not be on your system by default, and you may need to
>>>>> install it.
>>>>> An alternative is "ss", but I don't know the options.
>>>>>
>>>>>
>>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>>> Hi,
>>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>>> 7474
>>>>> are for TCPv6 sockets:
>>>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>>>> protocol: TCPv6
>>>>>>
>>>>>> # ps -ef | grep tomcat
>>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>>> /usr/local/jre/bin/java
>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>> properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>> -Dignore.endorsed.dirs= -classpath
>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>>>> tomcat
>>>>>>
>>>>>> #lsof -p 130244 | wc -l
>>>>>> 8028
>>>>>>
>>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>>> 7474
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Hi.
>>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>>> on
>>>>> your system.
>>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>>> when
>>>>> you look at the whole command-line.
>>>>>>
>>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>>> relevant here.
>>>>>>
>>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>>> files opened by the tomcat JVM.
>>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>>> "apache-tomcat"
>>>>>> (or anything else unique in that line)
>>>>>>
>>>>>>
>>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>>> Hi Chris,
>>>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>>>
>>>>>>> Below is the list of running processes on my server:
>>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>>> root 1 0 0 May07 ? 00:00:14
>>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>>> root 6 2 0 May07 ? 00:00:00
>>>>> [kworker/0:0H-kblockd]
>>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>>> root 19 2 0 May07 ? 00:00:00
>>>>> [kworker/1:0H-kblockd]
>>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>>> root 25 2 0 May07 ? 00:00:00
>>>>> [kworker/2:0H-kblockd]
>>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>>> root 31 2 0 May07 ? 00:00:00
>>>>> [kworker/3:0H-kblockd]
>>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>>> root 37 2 0 May07 ? 00:00:00
>>>>> [kworker/4:0H-kblockd]
>>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>>> root 43 2 0 May07 ? 00:00:00
>>>>> [kworker/5:0H-kblockd]
>>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>>> root 49 2 0 May07 ? 00:00:00
>>>>> [kworker/6:0H-kblockd]
>>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>>> root 55 2 0 May07 ? 00:00:00
>>>>> [kworker/7:0H-kblockd]
>>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>>> root 540 2 0 May07 ? 00:00:01
>>>>> [kworker/7:1H-kblockd]
>>>>>>> root 541 2 0 May07 ? 00:00:01
>>>>> [kworker/5:1H-kblockd]
>>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 613 2 0 May07 ? 00:00:01
>>>>> [kworker/6:1H-kblockd]
>>>>>>> root 650 2 0 May07 ? 00:00:01
>>>>> [kworker/4:1H-kblockd]
>>>>>>> root 659 2 0 May07 ? 00:00:02
>>>>> [kworker/3:1H-kblockd]
>>>>>>> root 660 2 0 May07 ? 00:00:02
>>>>> [kworker/2:1H-kblockd]
>>>>>>> root 662 2 0 May07 ? 00:00:01
>>>>> [kworker/0:1H-kblockd]
>>>>>>> root 679 1 0 May07 ? 00:00:03
>>>>> /usr/lib/systemd/systemd-journald
>>>>>>> root 682 2 0 May07 ? 00:00:02
>>>>> [kworker/1:1H-kblockd]
>>>>>>> root 716 1 0 May07 ? 00:00:01
>>>>> /usr/lib/systemd/systemd-udevd
>>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>>> --systemd-activation --syslog-only
>>>>>>> root 885 1 0 May07 ? 00:02:42
>>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>>> root 887 1 0 May07 ? 00:00:44
>>>>> /usr/sbin/irqbalance --foreground
>>>>>>> root 888 1 0 May07 ? 00:00:00
>>>>> /usr/bin/VGAuthService -s
>>>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>>>> --logger=files
>>>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>>> --fill-watermark=0
>>>>>>> root 946 890 0 May07 ? 00:00:06
>>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>>> --logger=files
>>>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>>>> -d2
>>>>>>> root 962 890 0 May07 ? 00:00:12
>>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>>>> -D
>>>>>>> root 970 1 0 May07 ? 00:00:02
>>>>> /usr/lib/systemd/systemd-logind
>>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>>>> -n
>>>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>>> /usr/local/vtl/bin/sanidmgr
>>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>>> /usr/local/vtl/bin/iscliproxy
>>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>>> /usr/local/vtl/bin/ipstorlog
>>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>>> /usr/local/vtl/bin/snmpd
>>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>>> /usr/local/vtl/bin/ipstorsm
>>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>>> /usr/lib/systemd/systemd --user
>>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>>>> -- \u --noclear tty1 linux
>>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>>> -
>>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>>> h
>>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>>> p
>>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>>> -
>>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>>> a
>>>>> 1,diffie-hellman-group14-sha1
>>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>>> m
>>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>>> [kworker/4:1-events_power_efficient]
>>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>>> [kworker/4:2-cgroup_destroy]
>>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>>> [kworker/6:0-mm_percpu_wq]
>>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>>> [kworker/3:3-mm_percpu_wq]
>>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>>> [kworker/2:1-mm_percpu_wq]
>>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>>> /usr/local/jre/bin/java
>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>> properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>> -Dignore.endorsed.dirs= -classpath
>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>>> -XX:+UseConcMarkSweepGC
>>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>>> t
>>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>>> -Des.default.path.conf=/etc/elasticsearch
>>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>>>> process
>>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>>> checkpointer process
>>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>>>> process
>>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>>> writer process
>>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>>> autovacuum launcher process
>>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>>> collector process
>>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>>> /usr/local/freestor/bin/fmsd
>>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>>> freestor 127.0.0.1(42660) idle
>>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>>>> freestor 127.0.0.1(42662) idle
>>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>>> freestor 127.0.0.1(42664) idle
>>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>>> [kworker/5:1-mm_percpu_wq]
>>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>>> [kworker/u16:2-events_unbound]
>>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>>> [kworker/u16:1-events_unbound]
>>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>>> [kworker/1:0-events_freezable]
>>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>>
>>>>>>> Thanks
>>>>>>> -----Original Message-----
>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Yeggy,
>>>>>>>
>>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>>> Hi Chris,
>>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>>
>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>>> 9.3
>>>>> to 9.6.
>>>>>>>>
>>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>>> many
>>>>> open files) error when the limit of open files for tomcat has reached
>>>>> (262144 in my environment), for example, I can get a connectivity
>>>>> error when pulling info from a server as below :
>>>>>>>>
>>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>>
>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>>
>>>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>>>> maxThreads="150"
>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>>>> compression="on"
>>>>>>>> compressionMinSize="2048"
>>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>>
>>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>>> redirectPort="443" /> ...........
>>>>>>>> <Connector address="0.0.0.0" port="443"
>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>>>
>>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>>> keystoreFile="conf/freestor"
>>>>> keystorePass="freestor"/>
>>>>>>>>
>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>>>
>>>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>>
>>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>>> Definitely
>>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>>
>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>> of
>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>> Different kernel versions count things differently.
>>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>>> files for that process
>>>>>>>
>>>>>>> Ok.
>>>>>>>
>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>> [YJ] Here is the netstat output:
>>>>>>>> Active Internet connections (w/o servers)
>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>>> State
>>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>>> ESTABLISHED
>>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>>> ESTABLISHED
>>>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>>>> ESTABLISHED
>>>>>>>
>>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>>> not
>>>>> what the whole OS is handling.
>>>>>>>
>>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>>> the
>>>>> process. But if you get "too many open files" being thrown inside the
>>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>>> process, then something else is going on. But remember to limit your
>>>>> lsof to *just the JVM* process.
>>>>>>>
>>>>>>> -chris
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>>> To: users@tomcat.apache.org
>>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>>
>>>>>>>> Yeggy,
>>>>>>>>
>>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>>> Hi,
>>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>>> protocol: TCPv6
>>>>>>>>>
>>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>>> changes in Linux
>>>>>>>>> 8 that can explain this
>>>>>>>>>
>>>>>>>>> Configuration:
>>>>>>>>> apache-tomcat-8.5.59
>>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # ps -ef | grep tomcat
>>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>>> /usr/local/jre/bin/java
>>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>>> properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>>> -Dignore.endorsed.dirs= -classpath
>>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>>>> tomcat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # cat /proc/51798/limits
>>>>>>>>> Limit Soft Limit Hard Limit
>>>>> Units
>>>>>>>>> Max cpu time unlimited unlimited
>>>>> seconds
>>>>>>>>> Max file size unlimited unlimited
>>>>> bytes
>>>>>>>>> Max data size unlimited unlimited
>>>>> bytes
>>>>>>>>> Max stack size 8388608 unlimited
>>>>> bytes
>>>>>>>>> Max core file size unlimited unlimited
>>>>> bytes
>>>>>>>>> Max resident set unlimited unlimited
>>>>> bytes
>>>>>>>>> Max processes 256518 256518
>>>>> processes
>>>>>>>>> Max open files 262144 262144
>>>>> files
>>>>>>>>> Max locked memory 65536 65536
>>>>> bytes
>>>>>>>>> Max address space unlimited unlimited
>>>>> bytes
>>>>>>>>> Max file locks unlimited unlimited
>>>>> locks
>>>>>>>>> Max pending signals 256518 256518
>>>>> signals
>>>>>>>>> Max msgqueue size 819200 819200
>>>>> bytes
>>>>>>>>> Max nice priority 0 0
>>>>>>>>> Max realtime priority 0 0
>>>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Too may open files:
>>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>>> 6679
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Most of open files are TCP sockets:
>>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>>> protocol: TCPv6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>>> 6413
>>>>>>>>
>>>>>>>> Some potentially dumb questions:
>>>>>>>>
>>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>>
>>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>>
>>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>>
>>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>>> of
>>>>> the sub-processes which represent the threads of the main process?
>>>>>>>> Different kernel versions count things differently.
>>>>>>>>
>>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> --
>>>> Noelette Stout
>>>>
>>>> ITS Enterprise Applications - Application Administrator - Senior
>>>>
>>>> Business Administration Building, Rm 109L
>>>> 921 South 8th Ave 8037
>>>> Idaho State University
>>>> Pocatello ID 83209
>>>> E-mail: stounoel "at" isu "dot" edu
>>>> Desk: 208-282-2554
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi,
Below is the nsenter output:
# ps -ef | grep tomcat
root 165217 1 1 10:22 ? 00:05:33 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
root 167329 167268 0 18:00 pts/1 00:00:00 grep --color=auto tomcat
# nsenter -t 165217 --net lsof -n -p 165217 |less
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 165217 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
java 165217 root rtd DIR 8,3 4096 2 /
java 165217 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
java 165217 root mem REG 8,2 113371 160881 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-jpa-2.1-api-1.0.0.Final.jar
java 165217 root mem REG 8,2 147874 160802 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activemq-protobuf-1.1.jar
java 165217 root mem REG 8,2 391515 160836 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-queryparser-4.10.4.jar
java 165217 root mem REG 8,2 868615 160813 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-context-3.2.17.RELEASE.jar
java 165217 root mem REG 8,2 9711 160792 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.6.6.jar
java 165217 root mem REG 8,2 196573 160819 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-expression-3.2.17.RELEASE.jar
java 165217 root mem REG 8,2 97173 160843 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-misc-4.10.4.jar
java 165217 root mem REG 8,2 10074 160872 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-ext-1.11.jar
java 165217 root mem REG 8,2 62983 160861 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/activation-1.1.jar
java 165217 root mem REG 8,2 371668 160812 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-security-core-3.2.9.RELEASE.jar
java 165217 root mem REG 8,2 645914 160754 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-entitymanager-4.3.5.Final.jar
java 165217 root mem REG 8,2 427030 160753 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-envers-4.3.5.Final.jar
java 165217 root mem REG 8,2 256468 160829 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/barcode4j-2.0.jar
java 165217 root mem REG 8,2 489884 160846 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/log4j-1.2.17.jar
java 165217 root mem REG 8,2 29257 160798 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.7.jar
java 165217 root mem REG 8,2 422612 160783 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-awt-util-1.11.jar
java 165217 root mem REG 8,2 16519 160796 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/jcl-over-slf4j-1.7.7.jar
java 165217 root mem REG 8,2 322362 160851 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/cglib-nodep-2.2.jar
java 165217 root mem REG 8,2 864598 160848 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/castor-1.2.jar
java 165217 root mem REG 8,2 434678 160748 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang3-3.4.jar
java 165217 root mem REG 8,2 325196 160785 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/batik-css-1.11.jar
java 165217 root mem REG 8,2 5230007 160755 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-core-4.3.5.Final.jar
java 165217 root mem REG 8,2 14041249 160763 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/elasticsearch-1.7.5.jar
java 165217 root mem REG 8,2 206385 160749 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-jms-3.2.17.RELEASE.jar
java 165217 root mem REG 8,2 64021 160837 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/lucene-join-4.10.4.jar
java 165217 root mem REG 8,2 714194 160828 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/javassist-3.18.1-GA.jar
java 165217 root mem REG 8,2 279193 160876 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/commons-lang-2.5.jar
java 165217 root mem REG 8,2 75311 160826 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/hibernate-commons-annotations-4.0.4.Final.jar
java 165217 root mem REG 8,2 436303 160745 /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/avro-1.7.7.jar
................
java 165217 root 73w FIFO 0,13 0t0 16786125 pipe
java 165217 root 74u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 75r FIFO 0,13 0t0 16786126 pipe
java 165217 root 76w FIFO 0,13 0t0 16786126 pipe
java 165217 root 77u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 78u sock 0,9 0t0 16961335 protocol: TCPv6
java 165217 root 79r FIFO 0,13 0t0 16786127 pipe
java 165217 root 80w FIFO 0,13 0t0 16786127 pipe
java 165217 root 81u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 82r FIFO 0,13 0t0 16786128 pipe
java 165217 root 83w FIFO 0,13 0t0 16786128 pipe
java 165217 root 84u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 86r FIFO 0,13 0t0 16786129 pipe
java 165217 root 87w FIFO 0,13 0t0 16786129 pipe
java 165217 root 88u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 89r FIFO 0,13 0t0 16786130 pipe
java 165217 root 90w FIFO 0,13 0t0 16786130 pipe
java 165217 root 91u a_inode 0,14 0 10970 [eventpoll]
java 165217 root 93u IPv6 16786133 0t0 TCP 127.0.0.1:mxi (LISTEN)
java 165217 root 94u sock 0,9 0t0 16963075 protocol: TCPv6
java 165217 root 96u sock 0,9 0t0 16824649 protocol: TCPv6
java 165217 root 97u IPv6 17051898 0t0 TCP 10.4.3.55:60204->172.22.22.192:https (ESTABLISHED)
java 165217 root 98u sock 0,9 0t0 16824673 protocol: TCPv6
java 165217 root 99u sock 0,9 0t0 16824711 protocol: TCPv6
java 165217 root 100u sock 0,9 0t0 16893245 protocol: TCPv6
java 165217 root 101u sock 0,9 0t0 16824385 protocol: TCPv6
java 165217 root 102u sock 0,9 0t0 16824264 protocol: TCPv6
java 165217 root 103u sock 0,9 0t0 16824278 protocol: TCPv6
java 165217 root 104u sock 0,9 0t0 16824279 protocol: TCPv6
java 165217 root 105u sock 0,9 0t0 16824063 protocol: TCPv6
java 165217 root 106u sock 0,9 0t0 16823982 protocol: TCPv6
java 165217 root 107u sock 0,9 0t0 16823959 protocol: TCPv6
java 165217 root 108r CHR 1,8 0t0 1041 /dev/random
java 165217 root 109u sock 0,9 0t0 16778709 protocol: TCPv6
java 165217 root 110w REG 8,6 25506176 1835053 /apps/apache-tomcat/logs/fms.log
java 165217 root 111w REG 8,6 919042 1835019 /apps/apache-tomcat/logs/obd.log
java 165217 root 112u sock 0,9 0t0 16823821 protocol: TCPv6
java 165217 root 113u sock 0,9 0t0 16823822 protocol: TCPv6
java 165217 root 114u sock 0,9 0t0 16823633 protocol: TCPv6
java 165217 root 115u sock 0,9 0t0 16823527 protocol: TCPv6
java 165217 root 116u sock 0,9 0t0 16823489 protocol: TCPv6
java 165217 root 117u sock 0,9 0t0 16823490 protocol: TCPv6
java 165217 root 118u sock 0,9 0t0 16823491 protocol: TCPv6
..................
Thanks
-----Original Message-----
From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
Sent: Tuesday, May 25, 2021 12:42 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Hi.
The point is to try to figure out what these thousands of apparently "TCPv6" sockets
belonging to the tomcat process actually are, so that we can maybe begin to look at where
they may be coming from.
The trouble is, the lsof output so far did not really tell us what these "sock" things
might be.
But there may be a clue here :
https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
(about when things run in a container).
Is that your case ?
And if so, could you run the lsof command in the container, as they suggest ?
And the point of forcing a tomcat/JVM GC was this :
When you restart tomcat (actually the JVM which runs tomcat), the OS will clean up *all*
the file descriptors belonging to that process, including the "legitimate" ones shown by
netstat, and the "unknown" ones shown in addition by lsof.
Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd that are held
by objects which are discarded, but still sit on the heap awaiting a GC to really destroy
them. If your heap is very large, it may otherwise take a long while before such a GC
happens, and such sockets could accumulate.
One way to trigger a GC is through JMX, but it takes a bit of additional setup to make
that work. That's why I was asking if you had some method to do that.
(see : https://code.google.com/archive/p/jmxsh/)
But let's look at the lsof part first.
On 24.05.2021 16:09, Yeggy Javadi wrote:
> Hi,
> I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat GC, I just restart it to reset the lsof to 0.
> Please see outputs below:
>
> # ps -ef | grep tomcat
> root 143152 1 0 May22 ? 00:26:44 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto tomcat
>
> # lsof -p 143152 | wc -l
> 41043
>
> # lsof -p 143152 | grep "protocol: TCPv6"| wc -l
> 40487
>
> # netstat -p -a -n --inet6 | grep 143152
> tcp6 0 0 :::8443 :::* LISTEN 143152/java
> tcp6 0 0 :::443 :::* LISTEN 143152/java
> tcp6 0 0 127.0.0.1:8005 :::* LISTEN 143152/java
> tcp6 0 0 :::1099 :::* LISTEN 143152/java
> tcp6 0 0 :::80 :::* LISTEN 143152/java
> tcp6 0 0 :::36081 :::* LISTEN 143152/java
> tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:80 10.197.255.10:55446 ESTABLISHED 143152/java
> tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576 CLOSE_WAIT 143152/java
> tcp6 0 0 10.4.3.55:53682 172.22.21.48:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576 CLOSE_WAIT 143152/java
> tcp6 0 0 10.4.3.55:40574 172.22.21.47:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:53782 172.22.21.48:443 ESTABLISHED 143152/java
> tcp6 0 1 10.4.3.55:49808 10.12.3.78:443 SYN_SENT 143152/java
> tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60226 172.22.22.192:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:80 10.197.255.10:55444 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:40540 172.22.21.47:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300 ESTABLISHED 143152/java
>
> # lsof -a -p 143152 -T s -i6
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
> java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
> java 143152 root 82u IPv6 13816325 0t0 TCP localhost.localdomain:mxi (LISTEN)
> java 143152 root 84u IPv6 13874431 0t0 TCP localhost.localdomain:52310->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 86u IPv6 13848367 0t0 TCP localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 215u IPv6 13822102 0t0 TCP localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 216u IPv6 13823064 0t0 TCP localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 217u IPv6 13821107 0t0 TCP localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 269u IPv6 13824001 0t0 TCP localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
> java 143152 root 270u IPv6 13787907 0t0 TCP Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 271u IPv6 13787908 0t0 TCP Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 272u IPv6 13787909 0t0 TCP Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 273u IPv6 13787910 0t0 TCP Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 274u IPv6 13787911 0t0 TCP Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 275u IPv6 13787912 0t0 TCP Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 276u IPv6 13787913 0t0 TCP Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 277u IPv6 13787914 0t0 TCP Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 278u IPv6 13787915 0t0 TCP Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 279u IPv6 13787916 0t0 TCP Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 280u IPv6 13787917 0t0 TCP Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 281u IPv6 13787918 0t0 TCP Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 282u IPv6 13787919 0t0 TCP Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry (LISTEN)
> java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
> java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https (LISTEN)
> java 143152 root *740u IPv6 15607907 0t0 TCP Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
> java 143152 root *876u IPv6 15608779 0t0 TCP Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
> java 143152 root *881u IPv6 15614275 0t0 TCP Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
> java 143152 root *882u IPv6 15558639 0t0 TCP Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
> java 143152 root *067u IPv6 15602583 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
> java 143152 root *070u IPv6 15602586 0t0 TCP Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
> java 143152 root *073u IPv6 15602582 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
> java 143152 root *080u IPv6 15512947 0t0 TCP Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
> java 143152 root *081u IPv6 15512948 0t0 TCP Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
> java 143152 root *817u IPv6 15611323 0t0 TCP Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
> java 143152 root *818u IPv6 15602584 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
>
> Thank
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Sunday, May 23, 2021 6:30 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Hi.
>
> I have no idea what lines like this are :
>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>
> There are obviously too many of them, for them to match with the sockets listed by netstat.
>
> The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
> ones marked as "LISTEN" in the netstat output.
> But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
>
> Could you redo a netstat as follows :
> # netstat -p -a -n --inet6 | grep 130244
>
> and can you also try this form of lsof :
> # lsof -a -p 130244 -T s -i6
>
> And finally (after copying the result of the above) : do you know how to trigger a GC
> (Garbage Collection) in your tomcat JVM ?
> (the point is to see if when a GC happens, these things disappear).
>
>
> On 22.05.2021 18:03, Yeggy Javadi wrote:
>> Here it is:
>>
>> # netstat -p -a --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>
>>
>> # lsof -p 130244
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
>> java 130244 root rtd DIR 8,3 4096 2 /
>> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
>> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
>> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
>> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
>> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
>> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
>> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
>> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
>> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
>> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
>> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
>> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
>> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
>> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
>> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
>> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
>> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
>> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
>> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
>> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
>> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
>> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
>> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
>> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
>> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
>> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
>> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
>> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
>> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
>> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
>> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
>> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
>> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
>> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
>> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
>> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
>> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
>> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
>> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
>> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
>> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
>> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
>> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
>> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
>> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
>> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
>> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
>> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
>> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
>> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
>> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
>> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
>> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
>> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
>> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
>> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
>> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
>> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
>> ...........................
>> Keep seeing the same line
>>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Saturday, May 22, 2021 10:22 AM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Mmm. Nothing very special in that netstat output.
>> The sockets seen there look quite normal for tomcat, and there are not a lot.
>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>
>> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
>> Where is the difference between the lsof count (19948) and the netstat count (25) ?
>> (genuine question, I'm puzzled too)
>>
>> Can you give an example of the "lsof -p 130244" output lines ?
>> (not all 19948 please, but enough to see some variety)
>>
>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>> Hi,
>>> Here its is:
>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>> 19948
>>>
>>> # netstat -p -a -6 --tcp | grep 130244
>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>> Thanks
>>> -----Original Message-----
>>> From: Noelette Stout <no...@isu.edu>
>>> Sent: Friday, May 21, 2021 8:28 AM
>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> ss has all the same options as netstat
>>>
>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>> <aw...@ice-sa.com>
>>> wrote:
>>>
>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>
>>>> The next thing that you may want to do :
>>>>
>>>> > netstat -p -a -6 --tcp
>>>>
>>>> That is an alternative list of sockets, which also shows the "tcp state"
>>>> of the sockets.
>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>> the last column.
>>>> The type of thing you are looking for is the column which should show
>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>
>>>> The options above :
>>>> -p : show PID and program
>>>> -a : show all sockets states
>>>> -6 : only inet v6
>>>> --tcp : only TCP sockets
>>>>
>>>> "netstat" may not be on your system by default, and you may need to
>>>> install it.
>>>> An alternative is "ss", but I don't know the options.
>>>>
>>>>
>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>> Hi,
>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>> 7474
>>>> are for TCPv6 sockets:
>>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>>> protocol: TCPv6
>>>>>
>>>>> # ps -ef | grep tomcat
>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>>
>>>>> #lsof -p 130244 | wc -l
>>>>> 8028
>>>>>
>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>> 7474
>>>>>
>>>>> Thanks
>>>>>
>>>>> -----Original Message-----
>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Hi.
>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>> on
>>>> your system.
>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>> when
>>>> you look at the whole command-line.
>>>>>
>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>> relevant here.
>>>>>
>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>> files opened by the tomcat JVM.
>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>> "apache-tomcat"
>>>>> (or anything else unique in that line)
>>>>>
>>>>>
>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>>
>>>>>> Below is the list of running processes on my server:
>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>> root 1 0 0 May07 ? 00:00:14
>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>> root 6 2 0 May07 ? 00:00:00
>>>> [kworker/0:0H-kblockd]
>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>> root 19 2 0 May07 ? 00:00:00
>>>> [kworker/1:0H-kblockd]
>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>> root 25 2 0 May07 ? 00:00:00
>>>> [kworker/2:0H-kblockd]
>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>> root 31 2 0 May07 ? 00:00:00
>>>> [kworker/3:0H-kblockd]
>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>> root 37 2 0 May07 ? 00:00:00
>>>> [kworker/4:0H-kblockd]
>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>> root 43 2 0 May07 ? 00:00:00
>>>> [kworker/5:0H-kblockd]
>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>> root 49 2 0 May07 ? 00:00:00
>>>> [kworker/6:0H-kblockd]
>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>> root 55 2 0 May07 ? 00:00:00
>>>> [kworker/7:0H-kblockd]
>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>> root 540 2 0 May07 ? 00:00:01
>>>> [kworker/7:1H-kblockd]
>>>>>> root 541 2 0 May07 ? 00:00:01
>>>> [kworker/5:1H-kblockd]
>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 613 2 0 May07 ? 00:00:01
>>>> [kworker/6:1H-kblockd]
>>>>>> root 650 2 0 May07 ? 00:00:01
>>>> [kworker/4:1H-kblockd]
>>>>>> root 659 2 0 May07 ? 00:00:02
>>>> [kworker/3:1H-kblockd]
>>>>>> root 660 2 0 May07 ? 00:00:02
>>>> [kworker/2:1H-kblockd]
>>>>>> root 662 2 0 May07 ? 00:00:01
>>>> [kworker/0:1H-kblockd]
>>>>>> root 679 1 0 May07 ? 00:00:03
>>>> /usr/lib/systemd/systemd-journald
>>>>>> root 682 2 0 May07 ? 00:00:02
>>>> [kworker/1:1H-kblockd]
>>>>>> root 716 1 0 May07 ? 00:00:01
>>>> /usr/lib/systemd/systemd-udevd
>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>> --systemd-activation --syslog-only
>>>>>> root 885 1 0 May07 ? 00:02:42
>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>> root 887 1 0 May07 ? 00:00:44
>>>> /usr/sbin/irqbalance --foreground
>>>>>> root 888 1 0 May07 ? 00:00:00
>>>> /usr/bin/VGAuthService -s
>>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>>> --logger=files
>>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>> --fill-watermark=0
>>>>>> root 946 890 0 May07 ? 00:00:06
>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>> --logger=files
>>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>>> -d2
>>>>>> root 962 890 0 May07 ? 00:00:12
>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>>> -D
>>>>>> root 970 1 0 May07 ? 00:00:02
>>>> /usr/lib/systemd/systemd-logind
>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>>> -n
>>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>> /usr/local/vtl/bin/sanidmgr
>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>> /usr/local/vtl/bin/iscliproxy
>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>> /usr/local/vtl/bin/ipstorlog
>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>> /usr/local/vtl/bin/snmpd
>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>> /usr/lib/systemd/systemd --user
>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>>> -- \u --noclear tty1 linux
>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>> -
>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>> h
>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>> p
>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>> -
>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>> a
>>>> 1,diffie-hellman-group14-sha1
>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m
>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>> [kworker/4:1-events_power_efficient]
>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>> [kworker/4:2-cgroup_destroy]
>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>> [kworker/6:0-mm_percpu_wq]
>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>> [kworker/3:3-mm_percpu_wq]
>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>> [kworker/2:1-mm_percpu_wq]
>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>> -XX:+UseConcMarkSweepGC
>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>> t
>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>> -Des.default.path.conf=/etc/elasticsearch
>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>>> process
>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>> checkpointer process
>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>>> process
>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>> writer process
>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>> autovacuum launcher process
>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>> collector process
>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>> /usr/local/freestor/bin/fmsd
>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42660) idle
>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>>> freestor 127.0.0.1(42662) idle
>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42664) idle
>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>> [kworker/5:1-mm_percpu_wq]
>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>> [kworker/u16:2-events_unbound]
>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>> [kworker/u16:1-events_unbound]
>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>> [kworker/1:0-events_freezable]
>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>
>>>>>> Thanks
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>> Hi Chris,
>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>> 9.3
>>>> to 9.6.
>>>>>>>
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>> many
>>>> open files) error when the limit of open files for tomcat has reached
>>>> (262144 in my environment), for example, I can get a connectivity
>>>> error when pulling info from a server as below :
>>>>>>>
>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>
>>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>>> maxThreads="150"
>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>>> compression="on"
>>>>>>> compressionMinSize="2048"
>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>
>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>> redirectPort="443" /> ...........
>>>>>>> <Connector address="0.0.0.0" port="443"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>>
>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>> keystoreFile="conf/freestor"
>>>> keystorePass="freestor"/>
>>>>>>>
>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>>
>>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>
>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>> Definitely
>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>> files for that process
>>>>>>
>>>>>> Ok.
>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>> [YJ] Here is the netstat output:
>>>>>>> Active Internet connections (w/o servers)
>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>> State
>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>> ESTABLISHED
>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>> ESTABLISHED
>>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>>> ESTABLISHED
>>>>>>
>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>> not
>>>> what the whole OS is handling.
>>>>>>
>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>> the
>>>> process. But if you get "too many open files" being thrown inside the
>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>> process, then something else is going on. But remember to limit your
>>>> lsof to *just the JVM* process.
>>>>>>
>>>>>> -chris
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Yeggy,
>>>>>>>
>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>> Hi,
>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>>
>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>> changes in Linux
>>>>>>>> 8 that can explain this
>>>>>>>>
>>>>>>>> Configuration:
>>>>>>>> apache-tomcat-8.5.59
>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>
>>>>>>>>
>>>>>>>> # ps -ef | grep tomcat
>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>>>>>
>>>>>>>>
>>>>>>>> # cat /proc/51798/limits
>>>>>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>>>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>>>>>> Max file size unlimited unlimited
>>>> bytes
>>>>>>>> Max data size unlimited unlimited
>>>> bytes
>>>>>>>> Max stack size 8388608 unlimited
>>>> bytes
>>>>>>>> Max core file size unlimited unlimited
>>>> bytes
>>>>>>>> Max resident set unlimited unlimited
>>>> bytes
>>>>>>>> Max processes 256518 256518
>>>> processes
>>>>>>>> Max open files 262144 262144
>>>> files
>>>>>>>> Max locked memory 65536 65536
>>>> bytes
>>>>>>>> Max address space unlimited unlimited
>>>> bytes
>>>>>>>> Max file locks unlimited unlimited
>>>> locks
>>>>>>>> Max pending signals 256518 256518
>>>> signals
>>>>>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>>>>>> Max nice priority 0 0
>>>>>>>> Max realtime priority 0 0
>>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>>
>>>>>>>>
>>>>>>>> Too may open files:
>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>> 6679
>>>>>>>>
>>>>>>>>
>>>>>>>> Most of open files are TCP sockets:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>>
>>>>>>>>
>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>> 6413
>>>>>>>
>>>>>>> Some potentially dumb questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>
>>>>>>> -chris
>>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> --
>>> Noelette Stout
>>>
>>> ITS Enterprise Applications - Application Administrator - Senior
>>>
>>> Business Administration Building, Rm 109L
>>> 921 South 8th Ave 8037
>>> Idaho State University
>>> Pocatello ID 83209
>>> E-mail: stounoel "at" isu "dot" edu
>>> Desk: 208-282-2554
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Hi.
The point is to try to figure out what these thousands of apparently "TCPv6" sockets
belonging to the tomcat process actually are, so that we can maybe begin to look at where
they may be coming from.
The trouble is, the lsof output so far did not really tell us what these "sock" things
might be.
But there may be a clue here :
https://serverfault.com/questions/1000338/in-lsof-output-what-are-those-sock-lines
(about when things run in a container).
Is that your case ?
And if so, could you run the lsof command in the container, as they suggest ?
And the point of forcing a tomcat/JVM GC was this :
When you restart tomcat (actually the JVM which runs tomcat), the OS will clean up *all*
the file descriptors belonging to that process, including the "legitimate" ones shown by
netstat, and the "unknown" ones shown in addition by lsof.
Doing a GC, without stopping the JVM, would clean up *only* such sockets/fd that are held
by objects which are discarded, but still sit on the heap awaiting a GC to really destroy
them. If your heap is very large, it may otherwise take a long while before such a GC
happens, and such sockets could accumulate.
One way to trigger a GC is through JMX, but it takes a bit of additional setup to make
that work. That's why I was asking if you had some method to do that.
(see : https://code.google.com/archive/p/jmxsh/)
But let's look at the lsof part first.
On 24.05.2021 16:09, Yeggy Javadi wrote:
> Hi,
> I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat GC, I just restart it to reset the lsof to 0.
> Please see outputs below:
>
> # ps -ef | grep tomcat
> root 143152 1 0 May22 ? 00:26:44 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto tomcat
>
> # lsof -p 143152 | wc -l
> 41043
>
> # lsof -p 143152 | grep "protocol: TCPv6"| wc -l
> 40487
>
> # netstat -p -a -n --inet6 | grep 143152
> tcp6 0 0 :::8443 :::* LISTEN 143152/java
> tcp6 0 0 :::443 :::* LISTEN 143152/java
> tcp6 0 0 127.0.0.1:8005 :::* LISTEN 143152/java
> tcp6 0 0 :::1099 :::* LISTEN 143152/java
> tcp6 0 0 :::80 :::* LISTEN 143152/java
> tcp6 0 0 :::36081 :::* LISTEN 143152/java
> tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:80 10.197.255.10:55446 ESTABLISHED 143152/java
> tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576 CLOSE_WAIT 143152/java
> tcp6 0 0 10.4.3.55:53682 172.22.21.48:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576 CLOSE_WAIT 143152/java
> tcp6 0 0 10.4.3.55:40574 172.22.21.47:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:53782 172.22.21.48:443 ESTABLISHED 143152/java
> tcp6 0 1 10.4.3.55:49808 10.12.3.78:443 SYN_SENT 143152/java
> tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60226 172.22.22.192:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:80 10.197.255.10:55444 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:40540 172.22.21.47:443 ESTABLISHED 143152/java
> tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432 ESTABLISHED 143152/java
> tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300 ESTABLISHED 143152/java
> tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300 ESTABLISHED 143152/java
>
> # lsof -a -p 143152 -T s -i6
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
> java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
> java 143152 root 82u IPv6 13816325 0t0 TCP localhost.localdomain:mxi (LISTEN)
> java 143152 root 84u IPv6 13874431 0t0 TCP localhost.localdomain:52310->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 86u IPv6 13848367 0t0 TCP localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 215u IPv6 13822102 0t0 TCP localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 216u IPv6 13823064 0t0 TCP localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 217u IPv6 13821107 0t0 TCP localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
> java 143152 root 269u IPv6 13824001 0t0 TCP localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
> java 143152 root 270u IPv6 13787907 0t0 TCP Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 271u IPv6 13787908 0t0 TCP Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 272u IPv6 13787909 0t0 TCP Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 273u IPv6 13787910 0t0 TCP Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 274u IPv6 13787911 0t0 TCP Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 275u IPv6 13787912 0t0 TCP Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 276u IPv6 13787913 0t0 TCP Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 277u IPv6 13787914 0t0 TCP Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 278u IPv6 13787915 0t0 TCP Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 279u IPv6 13787916 0t0 TCP Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 280u IPv6 13787917 0t0 TCP Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 281u IPv6 13787918 0t0 TCP Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 282u IPv6 13787919 0t0 TCP Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
> java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry (LISTEN)
> java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
> java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https (LISTEN)
> java 143152 root *740u IPv6 15607907 0t0 TCP Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
> java 143152 root *876u IPv6 15608779 0t0 TCP Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
> java 143152 root *881u IPv6 15614275 0t0 TCP Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
> java 143152 root *882u IPv6 15558639 0t0 TCP Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
> java 143152 root *067u IPv6 15602583 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
> java 143152 root *070u IPv6 15602586 0t0 TCP Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
> java 143152 root *073u IPv6 15602582 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
> java 143152 root *080u IPv6 15512947 0t0 TCP Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
> java 143152 root *081u IPv6 15512948 0t0 TCP Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
> java 143152 root *817u IPv6 15611323 0t0 TCP Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
> java 143152 root *818u IPv6 15602584 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
>
> Thank
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Sunday, May 23, 2021 6:30 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Hi.
>
> I have no idea what lines like this are :
>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>
> There are obviously too many of them, for them to match with the sockets listed by netstat.
>
> The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
> ones marked as "LISTEN" in the netstat output.
> But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
>
> Could you redo a netstat as follows :
> # netstat -p -a -n --inet6 | grep 130244
>
> and can you also try this form of lsof :
> # lsof -a -p 130244 -T s -i6
>
> And finally (after copying the result of the above) : do you know how to trigger a GC
> (Garbage Collection) in your tomcat JVM ?
> (the point is to see if when a GC happens, these things disappear).
>
>
> On 22.05.2021 18:03, Yeggy Javadi wrote:
>> Here it is:
>>
>> # netstat -p -a --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>
>>
>> # lsof -p 130244
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
>> java 130244 root rtd DIR 8,3 4096 2 /
>> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
>> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
>> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
>> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
>> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
>> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
>> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
>> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
>> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
>> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
>> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
>> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
>> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
>> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
>> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
>> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
>> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
>> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
>> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
>> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
>> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
>> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
>> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
>> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
>> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
>> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
>> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
>> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
>> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
>> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
>> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
>> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
>> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
>> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
>> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
>> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
>> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
>> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
>> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
>> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
>> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
>> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
>> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
>> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
>> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
>> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
>> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
>> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
>> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
>> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
>> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
>> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
>> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
>> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
>> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
>> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
>> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
>> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
>> ...........................
>> Keep seeing the same line
>>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Saturday, May 22, 2021 10:22 AM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Mmm. Nothing very special in that netstat output.
>> The sockets seen there look quite normal for tomcat, and there are not a lot.
>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>
>> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
>> Where is the difference between the lsof count (19948) and the netstat count (25) ?
>> (genuine question, I'm puzzled too)
>>
>> Can you give an example of the "lsof -p 130244" output lines ?
>> (not all 19948 please, but enough to see some variety)
>>
>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>> Hi,
>>> Here its is:
>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>> 19948
>>>
>>> # netstat -p -a -6 --tcp | grep 130244
>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>> Thanks
>>> -----Original Message-----
>>> From: Noelette Stout <no...@isu.edu>
>>> Sent: Friday, May 21, 2021 8:28 AM
>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> ss has all the same options as netstat
>>>
>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>> <aw...@ice-sa.com>
>>> wrote:
>>>
>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>
>>>> The next thing that you may want to do :
>>>>
>>>> > netstat -p -a -6 --tcp
>>>>
>>>> That is an alternative list of sockets, which also shows the "tcp state"
>>>> of the sockets.
>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>> the last column.
>>>> The type of thing you are looking for is the column which should show
>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>
>>>> The options above :
>>>> -p : show PID and program
>>>> -a : show all sockets states
>>>> -6 : only inet v6
>>>> --tcp : only TCP sockets
>>>>
>>>> "netstat" may not be on your system by default, and you may need to
>>>> install it.
>>>> An alternative is "ss", but I don't know the options.
>>>>
>>>>
>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>> Hi,
>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>> 7474
>>>> are for TCPv6 sockets:
>>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>>> protocol: TCPv6
>>>>>
>>>>> # ps -ef | grep tomcat
>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>>
>>>>> #lsof -p 130244 | wc -l
>>>>> 8028
>>>>>
>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>> 7474
>>>>>
>>>>> Thanks
>>>>>
>>>>> -----Original Message-----
>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Hi.
>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>> on
>>>> your system.
>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>> when
>>>> you look at the whole command-line.
>>>>>
>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>> relevant here.
>>>>>
>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>> files opened by the tomcat JVM.
>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>> "apache-tomcat"
>>>>> (or anything else unique in that line)
>>>>>
>>>>>
>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>>
>>>>>> Below is the list of running processes on my server:
>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>> root 1 0 0 May07 ? 00:00:14
>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>> root 6 2 0 May07 ? 00:00:00
>>>> [kworker/0:0H-kblockd]
>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>> root 19 2 0 May07 ? 00:00:00
>>>> [kworker/1:0H-kblockd]
>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>> root 25 2 0 May07 ? 00:00:00
>>>> [kworker/2:0H-kblockd]
>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>> root 31 2 0 May07 ? 00:00:00
>>>> [kworker/3:0H-kblockd]
>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>> root 37 2 0 May07 ? 00:00:00
>>>> [kworker/4:0H-kblockd]
>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>> root 43 2 0 May07 ? 00:00:00
>>>> [kworker/5:0H-kblockd]
>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>> root 49 2 0 May07 ? 00:00:00
>>>> [kworker/6:0H-kblockd]
>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>> root 55 2 0 May07 ? 00:00:00
>>>> [kworker/7:0H-kblockd]
>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>> root 540 2 0 May07 ? 00:00:01
>>>> [kworker/7:1H-kblockd]
>>>>>> root 541 2 0 May07 ? 00:00:01
>>>> [kworker/5:1H-kblockd]
>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 613 2 0 May07 ? 00:00:01
>>>> [kworker/6:1H-kblockd]
>>>>>> root 650 2 0 May07 ? 00:00:01
>>>> [kworker/4:1H-kblockd]
>>>>>> root 659 2 0 May07 ? 00:00:02
>>>> [kworker/3:1H-kblockd]
>>>>>> root 660 2 0 May07 ? 00:00:02
>>>> [kworker/2:1H-kblockd]
>>>>>> root 662 2 0 May07 ? 00:00:01
>>>> [kworker/0:1H-kblockd]
>>>>>> root 679 1 0 May07 ? 00:00:03
>>>> /usr/lib/systemd/systemd-journald
>>>>>> root 682 2 0 May07 ? 00:00:02
>>>> [kworker/1:1H-kblockd]
>>>>>> root 716 1 0 May07 ? 00:00:01
>>>> /usr/lib/systemd/systemd-udevd
>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>> --systemd-activation --syslog-only
>>>>>> root 885 1 0 May07 ? 00:02:42
>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>> root 887 1 0 May07 ? 00:00:44
>>>> /usr/sbin/irqbalance --foreground
>>>>>> root 888 1 0 May07 ? 00:00:00
>>>> /usr/bin/VGAuthService -s
>>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>>> --logger=files
>>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>> --fill-watermark=0
>>>>>> root 946 890 0 May07 ? 00:00:06
>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>> --logger=files
>>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>>> -d2
>>>>>> root 962 890 0 May07 ? 00:00:12
>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>>> -D
>>>>>> root 970 1 0 May07 ? 00:00:02
>>>> /usr/lib/systemd/systemd-logind
>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>>> -n
>>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>> /usr/local/vtl/bin/sanidmgr
>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>> /usr/local/vtl/bin/iscliproxy
>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>> /usr/local/vtl/bin/ipstorlog
>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>> /usr/local/vtl/bin/snmpd
>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>> /usr/lib/systemd/systemd --user
>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>>> -- \u --noclear tty1 linux
>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>> -
>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>> h
>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>> p
>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>> -
>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>> a
>>>> 1,diffie-hellman-group14-sha1
>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m
>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>> [kworker/4:1-events_power_efficient]
>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>> [kworker/4:2-cgroup_destroy]
>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>> [kworker/6:0-mm_percpu_wq]
>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>> [kworker/3:3-mm_percpu_wq]
>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>> [kworker/2:1-mm_percpu_wq]
>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>> -XX:+UseConcMarkSweepGC
>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>> t
>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>> -Des.default.path.conf=/etc/elasticsearch
>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>>> process
>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>> checkpointer process
>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>>> process
>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>> writer process
>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>> autovacuum launcher process
>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>> collector process
>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>> /usr/local/freestor/bin/fmsd
>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42660) idle
>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>>> freestor 127.0.0.1(42662) idle
>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42664) idle
>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>> [kworker/5:1-mm_percpu_wq]
>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>> [kworker/u16:2-events_unbound]
>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>> [kworker/u16:1-events_unbound]
>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>> [kworker/1:0-events_freezable]
>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>
>>>>>> Thanks
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>> Hi Chris,
>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>> 9.3
>>>> to 9.6.
>>>>>>>
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>> many
>>>> open files) error when the limit of open files for tomcat has reached
>>>> (262144 in my environment), for example, I can get a connectivity
>>>> error when pulling info from a server as below :
>>>>>>>
>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>
>>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>>> maxThreads="150"
>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>>> compression="on"
>>>>>>> compressionMinSize="2048"
>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>
>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>> redirectPort="443" /> ...........
>>>>>>> <Connector address="0.0.0.0" port="443"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>>
>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>> keystoreFile="conf/freestor"
>>>> keystorePass="freestor"/>
>>>>>>>
>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>>
>>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>
>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>> Definitely
>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>> files for that process
>>>>>>
>>>>>> Ok.
>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>> [YJ] Here is the netstat output:
>>>>>>> Active Internet connections (w/o servers)
>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>> State
>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>> ESTABLISHED
>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>> ESTABLISHED
>>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>>> ESTABLISHED
>>>>>>
>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>> not
>>>> what the whole OS is handling.
>>>>>>
>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>> the
>>>> process. But if you get "too many open files" being thrown inside the
>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>> process, then something else is going on. But remember to limit your
>>>> lsof to *just the JVM* process.
>>>>>>
>>>>>> -chris
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Yeggy,
>>>>>>>
>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>> Hi,
>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>>
>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>> changes in Linux
>>>>>>>> 8 that can explain this
>>>>>>>>
>>>>>>>> Configuration:
>>>>>>>> apache-tomcat-8.5.59
>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>
>>>>>>>>
>>>>>>>> # ps -ef | grep tomcat
>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>>>>>
>>>>>>>>
>>>>>>>> # cat /proc/51798/limits
>>>>>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>>>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>>>>>> Max file size unlimited unlimited
>>>> bytes
>>>>>>>> Max data size unlimited unlimited
>>>> bytes
>>>>>>>> Max stack size 8388608 unlimited
>>>> bytes
>>>>>>>> Max core file size unlimited unlimited
>>>> bytes
>>>>>>>> Max resident set unlimited unlimited
>>>> bytes
>>>>>>>> Max processes 256518 256518
>>>> processes
>>>>>>>> Max open files 262144 262144
>>>> files
>>>>>>>> Max locked memory 65536 65536
>>>> bytes
>>>>>>>> Max address space unlimited unlimited
>>>> bytes
>>>>>>>> Max file locks unlimited unlimited
>>>> locks
>>>>>>>> Max pending signals 256518 256518
>>>> signals
>>>>>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>>>>>> Max nice priority 0 0
>>>>>>>> Max realtime priority 0 0
>>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>>
>>>>>>>>
>>>>>>>> Too may open files:
>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>> 6679
>>>>>>>>
>>>>>>>>
>>>>>>>> Most of open files are TCP sockets:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>>
>>>>>>>>
>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>> 6413
>>>>>>>
>>>>>>> Some potentially dumb questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>>
>>>>>>> -chris
>>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> --
>>> Noelette Stout
>>>
>>> ITS Enterprise Applications - Application Administrator - Senior
>>>
>>> Business Administration Building, Rm 109L
>>> 921 South 8th Ave 8037
>>> Idaho State University
>>> Pocatello ID 83209
>>> E-mail: stounoel "at" isu "dot" edu
>>> Desk: 208-282-2554
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi,
I restarted tomcat so PID has changed to 143152; I do not know how to trigger tomcat GC, I just restart it to reset the lsof to 0.
Please see outputs below:
# ps -ef | grep tomcat
root 143152 1 0 May22 ? 00:26:44 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
root 153962 153912 0 10:00 pts/1 00:00:00 grep --color=auto tomcat
# lsof -p 143152 | wc -l
41043
# lsof -p 143152 | grep "protocol: TCPv6"| wc -l
40487
# netstat -p -a -n --inet6 | grep 143152
tcp6 0 0 :::8443 :::* LISTEN 143152/java
tcp6 0 0 :::443 :::* LISTEN 143152/java
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 143152/java
tcp6 0 0 :::1099 :::* LISTEN 143152/java
tcp6 0 0 :::80 :::* LISTEN 143152/java
tcp6 0 0 :::36081 :::* LISTEN 143152/java
tcp6 0 0 10.4.3.55:60736 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60732 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60728 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:80 10.197.255.10:55446 ESTABLISHED 143152/java
tcp6 1 0 10.4.3.55:55958 10.4.3.55:11576 CLOSE_WAIT 143152/java
tcp6 0 0 10.4.3.55:53682 172.22.21.48:443 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:48622 127.0.0.1:5432 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60748 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 1 0 10.4.3.55:55956 10.4.3.55:11576 CLOSE_WAIT 143152/java
tcp6 0 0 10.4.3.55:40574 172.22.21.47:443 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:48620 127.0.0.1:5432 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:53782 172.22.21.48:443 ESTABLISHED 143152/java
tcp6 0 1 10.4.3.55:49808 10.12.3.78:443 SYN_SENT 143152/java
tcp6 0 0 10.4.3.55:60730 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60750 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60742 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60746 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:48624 127.0.0.1:5432 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60734 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60226 172.22.22.192:443 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:52312 127.0.0.1:5432 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:57302 127.0.0.1:11753 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60738 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60740 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:80 10.197.255.10:55444 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:46976 127.0.0.1:9300 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:40540 172.22.21.47:443 ESTABLISHED 143152/java
tcp6 0 0 127.0.0.1:52310 127.0.0.1:5432 ESTABLISHED 143152/java
tcp6 0 0 10.4.3.55:60726 10.4.3.55:9300 ESTABLISHED 143152/java
tcp6 0 669 10.4.3.55:60744 10.4.3.55:9300 ESTABLISHED 143152/java
# lsof -a -p 143152 -T s -i6
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 143152 root 58u IPv6 13816226 0t0 TCP *:http (LISTEN)
java 143152 root 63u IPv6 13816230 0t0 TCP *:https (LISTEN)
java 143152 root 82u IPv6 13816325 0t0 TCP localhost.localdomain:mxi (LISTEN)
java 143152 root 84u IPv6 13874431 0t0 TCP localhost.localdomain:52310->localhost.localdomain:postgres (ESTABLISHED)
java 143152 root 86u IPv6 13848367 0t0 TCP localhost.localdomain:52312->localhost.localdomain:postgres (ESTABLISHED)
java 143152 root 215u IPv6 13822102 0t0 TCP localhost.localdomain:48620->localhost.localdomain:postgres (ESTABLISHED)
java 143152 root 216u IPv6 13823064 0t0 TCP localhost.localdomain:48624->localhost.localdomain:postgres (ESTABLISHED)
java 143152 root 217u IPv6 13821107 0t0 TCP localhost.localdomain:48622->localhost.localdomain:postgres (ESTABLISHED)
java 143152 root 269u IPv6 13824001 0t0 TCP localhost.localdomain:46976->localhost.localdomain:vrace (ESTABLISHED)
java 143152 root 270u IPv6 13787907 0t0 TCP Yeggy-F8-FMSVA:60726->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 271u IPv6 13787908 0t0 TCP Yeggy-F8-FMSVA:60728->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 272u IPv6 13787909 0t0 TCP Yeggy-F8-FMSVA:60730->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 273u IPv6 13787910 0t0 TCP Yeggy-F8-FMSVA:60732->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 274u IPv6 13787911 0t0 TCP Yeggy-F8-FMSVA:60734->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 275u IPv6 13787912 0t0 TCP Yeggy-F8-FMSVA:60736->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 276u IPv6 13787913 0t0 TCP Yeggy-F8-FMSVA:60738->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 277u IPv6 13787914 0t0 TCP Yeggy-F8-FMSVA:60740->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 278u IPv6 13787915 0t0 TCP Yeggy-F8-FMSVA:60742->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 279u IPv6 13787916 0t0 TCP Yeggy-F8-FMSVA:60744->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 280u IPv6 13787917 0t0 TCP Yeggy-F8-FMSVA:60746->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 281u IPv6 13787918 0t0 TCP Yeggy-F8-FMSVA:60748->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 282u IPv6 13787919 0t0 TCP Yeggy-F8-FMSVA:60750->Yeggy-F8-FMSVA:vrace (ESTABLISHED)
java 143152 root 293u IPv6 13821122 0t0 TCP *:rmiregistry (LISTEN)
java 143152 root 294u IPv6 13816318 0t0 TCP *:36081 (LISTEN)
java 143152 root 304u IPv6 13787923 0t0 TCP *:pcsync-https (LISTEN)
java 143152 root *740u IPv6 15607907 0t0 TCP Yeggy-F8-FMSVA:53366->172.22.21.48:https (ESTABLISHED)
java 143152 root *876u IPv6 15608779 0t0 TCP Yeggy-F8-FMSVA:40328->172.22.21.47:https (ESTABLISHED)
java 143152 root *881u IPv6 15614275 0t0 TCP Yeggy-F8-FMSVA:59970->172.22.22.192:https (ESTABLISHED)
java 143152 root *882u IPv6 15558639 0t0 TCP Yeggy-F8-FMSVA:53526->172.22.21.48:https (ESTABLISHED)
java 143152 root *067u IPv6 15602583 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55037 (ESTABLISHED)
java 143152 root *070u IPv6 15602586 0t0 TCP Yeggy-F8-FMSVA:40318->Yeggy-F8-FMSVA:36081 (ESTABLISHED)
java 143152 root *073u IPv6 15602582 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55036 (ESTABLISHED)
java 143152 root *080u IPv6 15512947 0t0 TCP Yeggy-F8-FMSVA:55956->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
java 143152 root *081u IPv6 15512948 0t0 TCP Yeggy-F8-FMSVA:55958->Yeggy-F8-FMSVA:11576 (CLOSE_WAIT)
java 143152 root *817u IPv6 15611323 0t0 TCP Yeggy-F8-FMSVA:36081->Yeggy-F8-FMSVA:40318 (ESTABLISHED)
java 143152 root *818u IPv6 15602584 0t0 TCP Yeggy-F8-FMSVA:http->10.197.255.10:55039 (ESTABLISHED)
Thank
-----Original Message-----
From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
Sent: Sunday, May 23, 2021 6:30 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Hi.
I have no idea what lines like this are :
> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
There are obviously too many of them, for them to match with the sockets listed by netstat.
The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
ones marked as "LISTEN" in the netstat output.
But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
Could you redo a netstat as follows :
# netstat -p -a -n --inet6 | grep 130244
and can you also try this form of lsof :
# lsof -a -p 130244 -T s -i6
And finally (after copying the result of the above) : do you know how to trigger a GC
(Garbage Collection) in your tomcat JVM ?
(the point is to see if when a GC happens, these things disappear).
On 22.05.2021 18:03, Yeggy Javadi wrote:
> Here it is:
>
> # netstat -p -a --tcp | grep 130244
> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>
>
> # lsof -p 130244
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
> java 130244 root rtd DIR 8,3 4096 2 /
> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
> ...........................
> Keep seeing the same line
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Saturday, May 22, 2021 10:22 AM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Mmm. Nothing very special in that netstat output.
> The sockets seen there look quite normal for tomcat, and there are not a lot.
> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>
> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
> Where is the difference between the lsof count (19948) and the netstat count (25) ?
> (genuine question, I'm puzzled too)
>
> Can you give an example of the "lsof -p 130244" output lines ?
> (not all 19948 please, but enough to see some variety)
>
> On 21.05.2021 16:11, Yeggy Javadi wrote:
>> Hi,
>> Here its is:
>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>> 19948
>>
>> # netstat -p -a -6 --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>> Thanks
>> -----Original Message-----
>> From: Noelette Stout <no...@isu.edu>
>> Sent: Friday, May 21, 2021 8:28 AM
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> ss has all the same options as netstat
>>
>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>> <aw...@ice-sa.com>
>> wrote:
>>
>>> Soyrry to top-post, but it's getting cluttered down there..
>>>
>>> The next thing that you may want to do :
>>>
>>> > netstat -p -a -6 --tcp
>>>
>>> That is an alternative list of sockets, which also shows the "tcp state"
>>> of the sockets.
>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>> the last column.
>>> The type of thing you are looking for is the column which should show
>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>
>>> The options above :
>>> -p : show PID and program
>>> -a : show all sockets states
>>> -6 : only inet v6
>>> --tcp : only TCP sockets
>>>
>>> "netstat" may not be on your system by default, and you may need to
>>> install it.
>>> An alternative is "ss", but I don't know the options.
>>>
>>>
>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>> Hi,
>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>> 7474
>>> are for TCPv6 sockets:
>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>> protocol: TCPv6
>>>>
>>>> # ps -ef | grep tomcat
>>>> root 130244 1 1 11:01 ? 00:06:20
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>>
>>>> #lsof -p 130244 | wc -l
>>>> 8028
>>>>
>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>> 7474
>>>>
>>>> Thanks
>>>>
>>>> -----Original Message-----
>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Hi.
>>>> According to the list below, you have 2 java (JVM) processes running
>>>> on
>>> your system.
>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>> when
>>> you look at the whole command-line.
>>>>
>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>> relevant here.
>>>>
>>>> So you should run lsof with the "-p 130244" option, to show only the
>>> files opened by the tomcat JVM.
>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>> "apache-tomcat"
>>>> (or anything else unique in that line)
>>>>
>>>>
>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>> Hi Chris,
>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>
>>>>> Below is the list of running processes on my server:
>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>> root 1 0 0 May07 ? 00:00:14
>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>> root 6 2 0 May07 ? 00:00:00
>>> [kworker/0:0H-kblockd]
>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>> root 19 2 0 May07 ? 00:00:00
>>> [kworker/1:0H-kblockd]
>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>> root 25 2 0 May07 ? 00:00:00
>>> [kworker/2:0H-kblockd]
>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>> root 31 2 0 May07 ? 00:00:00
>>> [kworker/3:0H-kblockd]
>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>> root 37 2 0 May07 ? 00:00:00
>>> [kworker/4:0H-kblockd]
>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>> root 43 2 0 May07 ? 00:00:00
>>> [kworker/5:0H-kblockd]
>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>> root 49 2 0 May07 ? 00:00:00
>>> [kworker/6:0H-kblockd]
>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>> root 55 2 0 May07 ? 00:00:00
>>> [kworker/7:0H-kblockd]
>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>> root 540 2 0 May07 ? 00:00:01
>>> [kworker/7:1H-kblockd]
>>>>> root 541 2 0 May07 ? 00:00:01
>>> [kworker/5:1H-kblockd]
>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 613 2 0 May07 ? 00:00:01
>>> [kworker/6:1H-kblockd]
>>>>> root 650 2 0 May07 ? 00:00:01
>>> [kworker/4:1H-kblockd]
>>>>> root 659 2 0 May07 ? 00:00:02
>>> [kworker/3:1H-kblockd]
>>>>> root 660 2 0 May07 ? 00:00:02
>>> [kworker/2:1H-kblockd]
>>>>> root 662 2 0 May07 ? 00:00:01
>>> [kworker/0:1H-kblockd]
>>>>> root 679 1 0 May07 ? 00:00:03
>>> /usr/lib/systemd/systemd-journald
>>>>> root 682 2 0 May07 ? 00:00:02
>>> [kworker/1:1H-kblockd]
>>>>> root 716 1 0 May07 ? 00:00:01
>>> /usr/lib/systemd/systemd-udevd
>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>> dbus 883 1 0 May07 ? 00:00:03
>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>> --systemd-activation --syslog-only
>>>>> root 885 1 0 May07 ? 00:02:42
>>> /usr/sbin/NetworkManager --no-daemon
>>>>> root 887 1 0 May07 ? 00:00:44
>>> /usr/sbin/irqbalance --foreground
>>>>> root 888 1 0 May07 ? 00:00:00
>>> /usr/bin/VGAuthService -s
>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>> --logger=files
>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>> --fill-watermark=0
>>>>> root 946 890 0 May07 ? 00:00:06
>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>> --logger=files
>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>> -d2
>>>>> root 962 890 0 May07 ? 00:00:12
>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>> -D
>>>>> root 970 1 0 May07 ? 00:00:02
>>> /usr/lib/systemd/systemd-logind
>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>> -n
>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>> root 6223 1 0 May07 ? 00:07:27
>>> /usr/local/vtl/bin/sanidmgr
>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>> root 6654 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6655 6654 0 May07 ? 00:05:18
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6656 6655 0 May07 ? 00:00:36
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>> root 6721 1 0 May07 ? 00:00:35
>>> /usr/local/vtl/bin/iscliproxy
>>>>> root 6747 1 0 May07 ? 00:00:40
>>> /usr/local/vtl/bin/ipstorlog
>>>>> root 6780 1 0 May07 ? 00:04:50
>>> /usr/local/vtl/bin/snmpd
>>>>> root 7722 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7723 7722 0 May07 ? 01:08:12
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7901 1 0 May07 ? 00:00:00
>>> /usr/lib/systemd/systemd --user
>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>> -- \u --noclear tty1 linux
>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>> -
>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>> h
>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>> p
>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>> -
>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>> a
>>> 1,diffie-hellman-group14-sha1
>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m
>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>> root 128108 2 0 08:40 ? 00:00:00
>>> [kworker/4:1-events_power_efficient]
>>>>> root 128360 2 0 09:10 ? 00:00:00
>>> [kworker/4:2-cgroup_destroy]
>>>>> root 128430 2 0 09:30 ? 00:00:00
>>> [kworker/6:0-mm_percpu_wq]
>>>>> root 128738 2 0 11:00 ? 00:00:00
>>> [kworker/3:3-mm_percpu_wq]
>>>>> root 128761 2 0 11:00 ? 00:00:00
>>> [kworker/2:1-mm_percpu_wq]
>>>>> root 130244 1 1 11:01 ? 00:03:50
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=75
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>> -Des.path.home=/usr/share/elasticsearch -cp
>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>> t
>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>> -Des.default.path.home=/usr/share/elasticsearch
>>> -Des.default.path.logs=/var/log/elasticsearch
>>> -Des.default.path.data=/var/lib/elasticsearch
>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>> -Des.default.path.conf=/etc/elasticsearch
>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>> process
>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>> checkpointer process
>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>> process
>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>> writer process
>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>> autovacuum launcher process
>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>> collector process
>>>>> root 130570 1 0 11:01 ? 00:00:01
>>> /usr/local/freestor/bin/fmsd
>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42660) idle
>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>> freestor 127.0.0.1(42662) idle
>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42664) idle
>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>> root 131347 2 0 13:49 ? 00:00:00
>>> [kworker/5:1-mm_percpu_wq]
>>>>> root 131364 2 0 13:56 ? 00:00:00
>>> [kworker/u16:2-events_unbound]
>>>>> root 131401 2 0 14:05 ? 00:00:00
>>> [kworker/u16:1-events_unbound]
>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>> root 131515 2 0 14:49 ? 00:00:00
>>> [kworker/1:0-events_freezable]
>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>
>>>>> Thanks
>>>>> -----Original Message-----
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Yeggy,
>>>>>
>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>> 9.3
>>> to 9.6.
>>>>>>
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>> many
>>> open files) error when the limit of open files for tomcat has reached
>>> (262144 in my environment), for example, I can get a connectivity
>>> error when pulling info from a server as below :
>>>>>>
>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>> [YJ] It is as below in server.xml:
>>>>>>
>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>> maxThreads="150"
>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>> enableLookups="false" acceptCount="100"
>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>> compression="on"
>>>>>> compressionMinSize="2048"
>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>
>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>> redirectPort="443" /> ...........
>>>>>> <Connector address="0.0.0.0" port="443"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>
>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>> keystoreFile="conf/freestor"
>>> keystorePass="freestor"/>
>>>>>>
>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>> redirectPort="443" secretRequired="false" />
>>>>>
>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>> Is that intentional? Which one are you using? Or both?
>>>>>
>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>> Definitely
>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>> files for that process
>>>>>
>>>>> Ok.
>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>> [YJ] Here is the netstat output:
>>>>>> Active Internet connections (w/o servers)
>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>> State
>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>> ESTABLISHED
>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>> ESTABLISHED
>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>> ESTABLISHED
>>>>>
>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>> not
>>> what the whole OS is handling.
>>>>>
>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>> the
>>> process. But if you get "too many open files" being thrown inside the
>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>> process, then something else is going on. But remember to limit your
>>> lsof to *just the JVM* process.
>>>>>
>>>>> -chris
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>> Hi,
>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>>
>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>> changes in Linux
>>>>>>> 8 that can explain this
>>>>>>>
>>>>>>> Configuration:
>>>>>>> apache-tomcat-8.5.59
>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>> application using Postgres and elastic search databases
>>>>>>>
>>>>>>>
>>>>>>> # ps -ef | grep tomcat
>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>>>>>
>>>>>>>
>>>>>>> # cat /proc/51798/limits
>>>>>>> Limit Soft Limit Hard Limit
>>> Units
>>>>>>> Max cpu time unlimited unlimited
>>> seconds
>>>>>>> Max file size unlimited unlimited
>>> bytes
>>>>>>> Max data size unlimited unlimited
>>> bytes
>>>>>>> Max stack size 8388608 unlimited
>>> bytes
>>>>>>> Max core file size unlimited unlimited
>>> bytes
>>>>>>> Max resident set unlimited unlimited
>>> bytes
>>>>>>> Max processes 256518 256518
>>> processes
>>>>>>> Max open files 262144 262144
>>> files
>>>>>>> Max locked memory 65536 65536
>>> bytes
>>>>>>> Max address space unlimited unlimited
>>> bytes
>>>>>>> Max file locks unlimited unlimited
>>> locks
>>>>>>> Max pending signals 256518 256518
>>> signals
>>>>>>> Max msgqueue size 819200 819200
>>> bytes
>>>>>>> Max nice priority 0 0
>>>>>>> Max realtime priority 0 0
>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>
>>>>>>>
>>>>>>> Too may open files:
>>>>>>> # lsof -p 51798 | wc -l
>>>>>>> 6679
>>>>>>>
>>>>>>>
>>>>>>> Most of open files are TCP sockets:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>>
>>>>>>>
>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>> 6413
>>>>>>
>>>>>> Some potentially dumb questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>>
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>
>>>>>> -chris
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> --
>> Noelette Stout
>>
>> ITS Enterprise Applications - Application Administrator - Senior
>>
>> Business Administration Building, Rm 109L
>> 921 South 8th Ave 8037
>> Idaho State University
>> Pocatello ID 83209
>> E-mail: stounoel "at" isu "dot" edu
>> Desk: 208-282-2554
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Hi.
I have no idea what lines like this are :
> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
There are obviously too many of them, for them to match with the sockets listed by netstat.
The ones which in the lsof output, have "TYPE" set to "IPv6" seem to correspond to the
ones marked as "LISTEN" in the netstat output.
But the ones with TYPE="sock" and NAME="protocol: TCPv6" are a mystery to me.
Could you redo a netstat as follows :
# netstat -p -a -n --inet6 | grep 130244
and can you also try this form of lsof :
# lsof -a -p 130244 -T s -i6
And finally (after copying the result of the above) : do you know how to trigger a GC
(Garbage Collection) in your tomcat JVM ?
(the point is to see if when a GC happens, these things disappear).
On 22.05.2021 18:03, Yeggy Javadi wrote:
> Here it is:
>
> # netstat -p -a --tcp | grep 130244
> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>
>
> # lsof -p 130244
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
> java 130244 root rtd DIR 8,3 4096 2 /
> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
> ...........................
> Keep seeing the same line
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Saturday, May 22, 2021 10:22 AM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Mmm. Nothing very special in that netstat output.
> The sockets seen there look quite normal for tomcat, and there are not a lot.
> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>
> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
> Where is the difference between the lsof count (19948) and the netstat count (25) ?
> (genuine question, I'm puzzled too)
>
> Can you give an example of the "lsof -p 130244" output lines ?
> (not all 19948 please, but enough to see some variety)
>
> On 21.05.2021 16:11, Yeggy Javadi wrote:
>> Hi,
>> Here its is:
>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>> 19948
>>
>> # netstat -p -a -6 --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>> Thanks
>> -----Original Message-----
>> From: Noelette Stout <no...@isu.edu>
>> Sent: Friday, May 21, 2021 8:28 AM
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> ss has all the same options as netstat
>>
>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>> <aw...@ice-sa.com>
>> wrote:
>>
>>> Soyrry to top-post, but it's getting cluttered down there..
>>>
>>> The next thing that you may want to do :
>>>
>>> > netstat -p -a -6 --tcp
>>>
>>> That is an alternative list of sockets, which also shows the "tcp state"
>>> of the sockets.
>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>> the last column.
>>> The type of thing you are looking for is the column which should show
>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>
>>> The options above :
>>> -p : show PID and program
>>> -a : show all sockets states
>>> -6 : only inet v6
>>> --tcp : only TCP sockets
>>>
>>> "netstat" may not be on your system by default, and you may need to
>>> install it.
>>> An alternative is "ss", but I don't know the options.
>>>
>>>
>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>> Hi,
>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>> 7474
>>> are for TCPv6 sockets:
>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>> protocol: TCPv6
>>>>
>>>> # ps -ef | grep tomcat
>>>> root 130244 1 1 11:01 ? 00:06:20
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>>
>>>> #lsof -p 130244 | wc -l
>>>> 8028
>>>>
>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>> 7474
>>>>
>>>> Thanks
>>>>
>>>> -----Original Message-----
>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Hi.
>>>> According to the list below, you have 2 java (JVM) processes running
>>>> on
>>> your system.
>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>> when
>>> you look at the whole command-line.
>>>>
>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>> relevant here.
>>>>
>>>> So you should run lsof with the "-p 130244" option, to show only the
>>> files opened by the tomcat JVM.
>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>> "apache-tomcat"
>>>> (or anything else unique in that line)
>>>>
>>>>
>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>> Hi Chris,
>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>
>>>>> Below is the list of running processes on my server:
>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>> root 1 0 0 May07 ? 00:00:14
>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>> root 6 2 0 May07 ? 00:00:00
>>> [kworker/0:0H-kblockd]
>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>> root 19 2 0 May07 ? 00:00:00
>>> [kworker/1:0H-kblockd]
>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>> root 25 2 0 May07 ? 00:00:00
>>> [kworker/2:0H-kblockd]
>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>> root 31 2 0 May07 ? 00:00:00
>>> [kworker/3:0H-kblockd]
>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>> root 37 2 0 May07 ? 00:00:00
>>> [kworker/4:0H-kblockd]
>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>> root 43 2 0 May07 ? 00:00:00
>>> [kworker/5:0H-kblockd]
>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>> root 49 2 0 May07 ? 00:00:00
>>> [kworker/6:0H-kblockd]
>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>> root 55 2 0 May07 ? 00:00:00
>>> [kworker/7:0H-kblockd]
>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>> root 540 2 0 May07 ? 00:00:01
>>> [kworker/7:1H-kblockd]
>>>>> root 541 2 0 May07 ? 00:00:01
>>> [kworker/5:1H-kblockd]
>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 613 2 0 May07 ? 00:00:01
>>> [kworker/6:1H-kblockd]
>>>>> root 650 2 0 May07 ? 00:00:01
>>> [kworker/4:1H-kblockd]
>>>>> root 659 2 0 May07 ? 00:00:02
>>> [kworker/3:1H-kblockd]
>>>>> root 660 2 0 May07 ? 00:00:02
>>> [kworker/2:1H-kblockd]
>>>>> root 662 2 0 May07 ? 00:00:01
>>> [kworker/0:1H-kblockd]
>>>>> root 679 1 0 May07 ? 00:00:03
>>> /usr/lib/systemd/systemd-journald
>>>>> root 682 2 0 May07 ? 00:00:02
>>> [kworker/1:1H-kblockd]
>>>>> root 716 1 0 May07 ? 00:00:01
>>> /usr/lib/systemd/systemd-udevd
>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>> dbus 883 1 0 May07 ? 00:00:03
>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>> --systemd-activation --syslog-only
>>>>> root 885 1 0 May07 ? 00:02:42
>>> /usr/sbin/NetworkManager --no-daemon
>>>>> root 887 1 0 May07 ? 00:00:44
>>> /usr/sbin/irqbalance --foreground
>>>>> root 888 1 0 May07 ? 00:00:00
>>> /usr/bin/VGAuthService -s
>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>> --logger=files
>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>> --fill-watermark=0
>>>>> root 946 890 0 May07 ? 00:00:06
>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>> --logger=files
>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>> -d2
>>>>> root 962 890 0 May07 ? 00:00:12
>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>> -D
>>>>> root 970 1 0 May07 ? 00:00:02
>>> /usr/lib/systemd/systemd-logind
>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>> -n
>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>> root 6223 1 0 May07 ? 00:07:27
>>> /usr/local/vtl/bin/sanidmgr
>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>> root 6654 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6655 6654 0 May07 ? 00:05:18
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6656 6655 0 May07 ? 00:00:36
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>> root 6721 1 0 May07 ? 00:00:35
>>> /usr/local/vtl/bin/iscliproxy
>>>>> root 6747 1 0 May07 ? 00:00:40
>>> /usr/local/vtl/bin/ipstorlog
>>>>> root 6780 1 0 May07 ? 00:04:50
>>> /usr/local/vtl/bin/snmpd
>>>>> root 7722 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7723 7722 0 May07 ? 01:08:12
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7901 1 0 May07 ? 00:00:00
>>> /usr/lib/systemd/systemd --user
>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>> -- \u --noclear tty1 linux
>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>> -
>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>> h
>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>> p
>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>> -
>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>> a
>>> 1,diffie-hellman-group14-sha1
>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m
>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>> root 128108 2 0 08:40 ? 00:00:00
>>> [kworker/4:1-events_power_efficient]
>>>>> root 128360 2 0 09:10 ? 00:00:00
>>> [kworker/4:2-cgroup_destroy]
>>>>> root 128430 2 0 09:30 ? 00:00:00
>>> [kworker/6:0-mm_percpu_wq]
>>>>> root 128738 2 0 11:00 ? 00:00:00
>>> [kworker/3:3-mm_percpu_wq]
>>>>> root 128761 2 0 11:00 ? 00:00:00
>>> [kworker/2:1-mm_percpu_wq]
>>>>> root 130244 1 1 11:01 ? 00:03:50
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=75
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>> -Des.path.home=/usr/share/elasticsearch -cp
>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>> t
>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>> -Des.default.path.home=/usr/share/elasticsearch
>>> -Des.default.path.logs=/var/log/elasticsearch
>>> -Des.default.path.data=/var/lib/elasticsearch
>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>> -Des.default.path.conf=/etc/elasticsearch
>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>> process
>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>> checkpointer process
>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>> process
>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>> writer process
>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>> autovacuum launcher process
>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>> collector process
>>>>> root 130570 1 0 11:01 ? 00:00:01
>>> /usr/local/freestor/bin/fmsd
>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42660) idle
>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>> freestor 127.0.0.1(42662) idle
>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42664) idle
>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>> root 131347 2 0 13:49 ? 00:00:00
>>> [kworker/5:1-mm_percpu_wq]
>>>>> root 131364 2 0 13:56 ? 00:00:00
>>> [kworker/u16:2-events_unbound]
>>>>> root 131401 2 0 14:05 ? 00:00:00
>>> [kworker/u16:1-events_unbound]
>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>> root 131515 2 0 14:49 ? 00:00:00
>>> [kworker/1:0-events_freezable]
>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>
>>>>> Thanks
>>>>> -----Original Message-----
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Yeggy,
>>>>>
>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>> 9.3
>>> to 9.6.
>>>>>>
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>> many
>>> open files) error when the limit of open files for tomcat has reached
>>> (262144 in my environment), for example, I can get a connectivity
>>> error when pulling info from a server as below :
>>>>>>
>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>> [YJ] It is as below in server.xml:
>>>>>>
>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>> maxThreads="150"
>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>> enableLookups="false" acceptCount="100"
>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>> compression="on"
>>>>>> compressionMinSize="2048"
>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>
>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>> redirectPort="443" /> ...........
>>>>>> <Connector address="0.0.0.0" port="443"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>
>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>> keystoreFile="conf/freestor"
>>> keystorePass="freestor"/>
>>>>>>
>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>> redirectPort="443" secretRequired="false" />
>>>>>
>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>> Is that intentional? Which one are you using? Or both?
>>>>>
>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>> Definitely
>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>> files for that process
>>>>>
>>>>> Ok.
>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>> [YJ] Here is the netstat output:
>>>>>> Active Internet connections (w/o servers)
>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>> State
>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>> ESTABLISHED
>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>> ESTABLISHED
>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>> ESTABLISHED
>>>>>
>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>> not
>>> what the whole OS is handling.
>>>>>
>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>> the
>>> process. But if you get "too many open files" being thrown inside the
>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>> process, then something else is going on. But remember to limit your
>>> lsof to *just the JVM* process.
>>>>>
>>>>> -chris
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>> Hi,
>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>>
>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>> changes in Linux
>>>>>>> 8 that can explain this
>>>>>>>
>>>>>>> Configuration:
>>>>>>> apache-tomcat-8.5.59
>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>> application using Postgres and elastic search databases
>>>>>>>
>>>>>>>
>>>>>>> # ps -ef | grep tomcat
>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>>>>>
>>>>>>>
>>>>>>> # cat /proc/51798/limits
>>>>>>> Limit Soft Limit Hard Limit
>>> Units
>>>>>>> Max cpu time unlimited unlimited
>>> seconds
>>>>>>> Max file size unlimited unlimited
>>> bytes
>>>>>>> Max data size unlimited unlimited
>>> bytes
>>>>>>> Max stack size 8388608 unlimited
>>> bytes
>>>>>>> Max core file size unlimited unlimited
>>> bytes
>>>>>>> Max resident set unlimited unlimited
>>> bytes
>>>>>>> Max processes 256518 256518
>>> processes
>>>>>>> Max open files 262144 262144
>>> files
>>>>>>> Max locked memory 65536 65536
>>> bytes
>>>>>>> Max address space unlimited unlimited
>>> bytes
>>>>>>> Max file locks unlimited unlimited
>>> locks
>>>>>>> Max pending signals 256518 256518
>>> signals
>>>>>>> Max msgqueue size 819200 819200
>>> bytes
>>>>>>> Max nice priority 0 0
>>>>>>> Max realtime priority 0 0
>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>
>>>>>>>
>>>>>>> Too may open files:
>>>>>>> # lsof -p 51798 | wc -l
>>>>>>> 6679
>>>>>>>
>>>>>>>
>>>>>>> Most of open files are TCP sockets:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>>
>>>>>>>
>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>> 6413
>>>>>>
>>>>>> Some potentially dumb questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>>
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>
>>>>>> -chris
>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> --
>> Noelette Stout
>>
>> ITS Enterprise Applications - Application Administrator - Senior
>>
>> Business Administration Building, Rm 109L
>> 921 South 8th Ave 8037
>> Idaho State University
>> Pocatello ID 83209
>> E-mail: stounoel "at" isu "dot" edu
>> Desk: 208-282-2554
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Felix Schumacher <fe...@internetallee.de>.
Sorry for the noise, you gave the numbers of open files (8028) and the
limits (262144) in your first mail.
Felix
Am 23.05.21 um 12:18 schrieb Felix Schumacher:
> Am 22.05.21 um 18:03 schrieb Yeggy Javadi:
>> Here it is:
>>
>> # netstat -p -a --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>
>>
>> # lsof -p 130244
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
>> java 130244 root rtd DIR 8,3 4096 2 /
>> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
>> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
>> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
>> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
>> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
>> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
>> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
>> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
>> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
>> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
>> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
>> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
>> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
>> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
>> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
>> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
>> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
>> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
>> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
>> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
>> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
>> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
>> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
>> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
>> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
>> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
>> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
>> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
>> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
>> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
>> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
>> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
>> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
>> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
>> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
>> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
>> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
>> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
>> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
>> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
>> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
>> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
>> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
>> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
>> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
>> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
>> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
>> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
>> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
>> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
>> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
>> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
>> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
>> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
>> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
>> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
>> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
>> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
>> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
>> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
>> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
>> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
>> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
>> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
>> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
>> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
>> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
>> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
>> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
>> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
>> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
>> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
>> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
>> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
>> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
>> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
>> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
>> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
>> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
>> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
>> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
>> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
>> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
>> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
>> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
>> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
>> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
>> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
>> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
>> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
>> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
>> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
>> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
>> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
>> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
>> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
>> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
>> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
>> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
>> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
>> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
>> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
>> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
>> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
>> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
>> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
>> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
>> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
>> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
>> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
>> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
>> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
>> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
>> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
>> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
>> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
>> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
>> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
>> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
>> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
>> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
>> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
>> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
>> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
>> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
>> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
>> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
>> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
>> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
>> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
>> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
>> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
>> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
>> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
>> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
>> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
>> ...........................
>> Keep seeing the same line
>
> How many of those lines are there? (lsof -p ... | wc -l)
>
> Are you sure there are more than before? Maybe your ulimits where higher
> on your older system and are now
> bothering you?
>
> What is the setting of ulimit for Tomcats process? (cat /proc/<PID>/limits)
>
> Felix
>
>
>> -----Original Message-----
>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>> Sent: Saturday, May 22, 2021 10:22 AM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Mmm. Nothing very special in that netstat output.
>> The sockets seen there look quite normal for tomcat, and there are not a lot.
>> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>>
>> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
>> Where is the difference between the lsof count (19948) and the netstat count (25) ?
>> (genuine question, I'm puzzled too)
>>
>> Can you give an example of the "lsof -p 130244" output lines ?
>> (not all 19948 please, but enough to see some variety)
>>
>> On 21.05.2021 16:11, Yeggy Javadi wrote:
>>> Hi,
>>> Here its is:
>>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>> 19948
>>>
>>> # netstat -p -a -6 --tcp | grep 130244
>>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>>> Thanks
>>> -----Original Message-----
>>> From: Noelette Stout <no...@isu.edu>
>>> Sent: Friday, May 21, 2021 8:28 AM
>>> To: Tomcat Users List <us...@tomcat.apache.org>
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> ss has all the same options as netstat
>>>
>>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>>> <aw...@ice-sa.com>
>>> wrote:
>>>
>>>> Soyrry to top-post, but it's getting cluttered down there..
>>>>
>>>> The next thing that you may want to do :
>>>>
>>>> > netstat -p -a -6 --tcp
>>>>
>>>> That is an alternative list of sockets, which also shows the "tcp state"
>>>> of the sockets.
>>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>>> the last column.
>>>> The type of thing you are looking for is the column which should show
>>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>>
>>>> The options above :
>>>> -p : show PID and program
>>>> -a : show all sockets states
>>>> -6 : only inet v6
>>>> --tcp : only TCP sockets
>>>>
>>>> "netstat" may not be on your system by default, and you may need to
>>>> install it.
>>>> An alternative is "ss", but I don't know the options.
>>>>
>>>>
>>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>>> Hi,
>>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>>> 7474
>>>> are for TCPv6 sockets:
>>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>>> protocol: TCPv6
>>>>> # ps -ef | grep tomcat
>>>>> root 130244 1 1 11:01 ? 00:06:20
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>> #lsof -p 130244 | wc -l
>>>>> 8028
>>>>>
>>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>>> 7474
>>>>>
>>>>> Thanks
>>>>>
>>>>> -----Original Message-----
>>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Hi.
>>>>> According to the list below, you have 2 java (JVM) processes running
>>>>> on
>>>> your system.
>>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>>> when
>>>> you look at the whole command-line.
>>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>>> relevant here.
>>>>> So you should run lsof with the "-p 130244" option, to show only the
>>>> files opened by the tomcat JVM.
>>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>>> "apache-tomcat"
>>>>> (or anything else unique in that line)
>>>>>
>>>>>
>>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>>
>>>>>> Below is the list of running processes on my server:
>>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>>> root 1 0 0 May07 ? 00:00:14
>>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>>> root 6 2 0 May07 ? 00:00:00
>>>> [kworker/0:0H-kblockd]
>>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>>> root 19 2 0 May07 ? 00:00:00
>>>> [kworker/1:0H-kblockd]
>>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>>> root 25 2 0 May07 ? 00:00:00
>>>> [kworker/2:0H-kblockd]
>>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>>> root 31 2 0 May07 ? 00:00:00
>>>> [kworker/3:0H-kblockd]
>>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>>> root 37 2 0 May07 ? 00:00:00
>>>> [kworker/4:0H-kblockd]
>>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>>> root 43 2 0 May07 ? 00:00:00
>>>> [kworker/5:0H-kblockd]
>>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>>> root 49 2 0 May07 ? 00:00:00
>>>> [kworker/6:0H-kblockd]
>>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>>> root 55 2 0 May07 ? 00:00:00
>>>> [kworker/7:0H-kblockd]
>>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>>> root 540 2 0 May07 ? 00:00:01
>>>> [kworker/7:1H-kblockd]
>>>>>> root 541 2 0 May07 ? 00:00:01
>>>> [kworker/5:1H-kblockd]
>>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 613 2 0 May07 ? 00:00:01
>>>> [kworker/6:1H-kblockd]
>>>>>> root 650 2 0 May07 ? 00:00:01
>>>> [kworker/4:1H-kblockd]
>>>>>> root 659 2 0 May07 ? 00:00:02
>>>> [kworker/3:1H-kblockd]
>>>>>> root 660 2 0 May07 ? 00:00:02
>>>> [kworker/2:1H-kblockd]
>>>>>> root 662 2 0 May07 ? 00:00:01
>>>> [kworker/0:1H-kblockd]
>>>>>> root 679 1 0 May07 ? 00:00:03
>>>> /usr/lib/systemd/systemd-journald
>>>>>> root 682 2 0 May07 ? 00:00:02
>>>> [kworker/1:1H-kblockd]
>>>>>> root 716 1 0 May07 ? 00:00:01
>>>> /usr/lib/systemd/systemd-udevd
>>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>>> dbus 883 1 0 May07 ? 00:00:03
>>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>>> --systemd-activation --syslog-only
>>>>>> root 885 1 0 May07 ? 00:02:42
>>>> /usr/sbin/NetworkManager --no-daemon
>>>>>> root 887 1 0 May07 ? 00:00:44
>>>> /usr/sbin/irqbalance --foreground
>>>>>> root 888 1 0 May07 ? 00:00:00
>>>> /usr/bin/VGAuthService -s
>>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>>> --logger=files
>>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>>> --fill-watermark=0
>>>>>> root 946 890 0 May07 ? 00:00:06
>>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>>> --logger=files
>>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>>> -d2
>>>>>> root 962 890 0 May07 ? 00:00:12
>>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>>> -D
>>>>>> root 970 1 0 May07 ? 00:00:02
>>>> /usr/lib/systemd/systemd-logind
>>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>>> -n
>>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>>> root 6223 1 0 May07 ? 00:07:27
>>>> /usr/local/vtl/bin/sanidmgr
>>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>>> root 6654 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6655 6654 0 May07 ? 00:05:18
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6656 6655 0 May07 ? 00:00:36
>>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>>> root 6721 1 0 May07 ? 00:00:35
>>>> /usr/local/vtl/bin/iscliproxy
>>>>>> root 6747 1 0 May07 ? 00:00:40
>>>> /usr/local/vtl/bin/ipstorlog
>>>>>> root 6780 1 0 May07 ? 00:04:50
>>>> /usr/local/vtl/bin/snmpd
>>>>>> root 7722 1 0 May07 ? 00:00:00
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7723 7722 0 May07 ? 01:08:12
>>>> /usr/local/vtl/bin/ipstorsm
>>>>>> root 7901 1 0 May07 ? 00:00:00
>>>> /usr/lib/systemd/systemd --user
>>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>>> -- \u --noclear tty1 linux
>>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>>> -
>>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>>> h
>>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>>> p
>>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>>> -
>>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>>> a
>>>> 1,diffie-hellman-group14-sha1
>>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>>> m
>>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>>> root 128108 2 0 08:40 ? 00:00:00
>>>> [kworker/4:1-events_power_efficient]
>>>>>> root 128360 2 0 09:10 ? 00:00:00
>>>> [kworker/4:2-cgroup_destroy]
>>>>>> root 128430 2 0 09:30 ? 00:00:00
>>>> [kworker/6:0-mm_percpu_wq]
>>>>>> root 128738 2 0 11:00 ? 00:00:00
>>>> [kworker/3:3-mm_percpu_wq]
>>>>>> root 128761 2 0 11:00 ? 00:00:00
>>>> [kworker/2:1-mm_percpu_wq]
>>>>>> root 130244 1 1 11:01 ? 00:03:50
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>>> -XX:+UseConcMarkSweepGC
>>>> -XX:CMSInitiatingOccupancyFraction=75
>>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>>> -Des.path.home=/usr/share/elasticsearch -cp
>>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>>> t
>>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>>> -Des.default.path.home=/usr/share/elasticsearch
>>>> -Des.default.path.logs=/var/log/elasticsearch
>>>> -Des.default.path.data=/var/lib/elasticsearch
>>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>>> -Des.default.path.conf=/etc/elasticsearch
>>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>>> process
>>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>>> checkpointer process
>>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>>> process
>>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>>> writer process
>>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>>> autovacuum launcher process
>>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>>> collector process
>>>>>> root 130570 1 0 11:01 ? 00:00:01
>>>> /usr/local/freestor/bin/fmsd
>>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42660) idle
>>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>>> freestor 127.0.0.1(42662) idle
>>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>>> freestor 127.0.0.1(42664) idle
>>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>>> root 131347 2 0 13:49 ? 00:00:00
>>>> [kworker/5:1-mm_percpu_wq]
>>>>>> root 131364 2 0 13:56 ? 00:00:00
>>>> [kworker/u16:2-events_unbound]
>>>>>> root 131401 2 0 14:05 ? 00:00:00
>>>> [kworker/u16:1-events_unbound]
>>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>>> root 131515 2 0 14:49 ? 00:00:00
>>>> [kworker/1:0-events_freezable]
>>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>>
>>>>>> Thanks
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>>> Hi Chris,
>>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>>> 9.3
>>>> to 9.6.
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>>> many
>>>> open files) error when the limit of open files for tomcat has reached
>>>> (262144 in my environment), for example, I can get a connectivity
>>>> error when pulling info from a server as below :
>>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>> [YJ] It is as below in server.xml:
>>>>>>>
>>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>>> maxThreads="150"
>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>> enableLookups="false" acceptCount="100"
>>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>>> compression="on"
>>>>>>> compressionMinSize="2048"
>>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>>
>>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>>> redirectPort="443" /> ...........
>>>>>>> <Connector address="0.0.0.0" port="443"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>>
>>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>>> keystoreFile="conf/freestor"
>>>> keystorePass="freestor"/>
>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>>> redirectPort="443" secretRequired="false" />
>>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>>> Is that intentional? Which one are you using? Or both?
>>>>>>
>>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>>> Definitely
>>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>>> files for that process
>>>>>> Ok.
>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>> [YJ] Here is the netstat output:
>>>>>>> Active Internet connections (w/o servers)
>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>>> State
>>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>>> ESTABLISHED
>>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>>> ESTABLISHED
>>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>>> ESTABLISHED
>>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>>> not
>>>> what the whole OS is handling.
>>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>>> the
>>>> process. But if you get "too many open files" being thrown inside the
>>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>>> process, then something else is going on. But remember to limit your
>>>> lsof to *just the JVM* process.
>>>>>> -chris
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>>> To: users@tomcat.apache.org
>>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>>
>>>>>>> Yeggy,
>>>>>>>
>>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>>> Hi,
>>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>>> changes in Linux
>>>>>>>> 8 that can explain this
>>>>>>>>
>>>>>>>> Configuration:
>>>>>>>> apache-tomcat-8.5.59
>>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>>> application using Postgres and elastic search databases
>>>>>>>>
>>>>>>>>
>>>>>>>> # ps -ef | grep tomcat
>>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>>> /usr/local/jre/bin/java
>>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>>> properties
>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>>> -Djdk.tls.ephemeralDHKeySize=2048
>>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>>> -Dignore.endorsed.dirs= -classpath
>>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>>> -Dcatalina.home=/usr/local/apache-tomcat
>>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>>> org.apache.catalina.startup.Bootstrap start
>>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>>> tomcat
>>>>>>>> # cat /proc/51798/limits
>>>>>>>> Limit Soft Limit Hard Limit
>>>> Units
>>>>>>>> Max cpu time unlimited unlimited
>>>> seconds
>>>>>>>> Max file size unlimited unlimited
>>>> bytes
>>>>>>>> Max data size unlimited unlimited
>>>> bytes
>>>>>>>> Max stack size 8388608 unlimited
>>>> bytes
>>>>>>>> Max core file size unlimited unlimited
>>>> bytes
>>>>>>>> Max resident set unlimited unlimited
>>>> bytes
>>>>>>>> Max processes 256518 256518
>>>> processes
>>>>>>>> Max open files 262144 262144
>>>> files
>>>>>>>> Max locked memory 65536 65536
>>>> bytes
>>>>>>>> Max address space unlimited unlimited
>>>> bytes
>>>>>>>> Max file locks unlimited unlimited
>>>> locks
>>>>>>>> Max pending signals 256518 256518
>>>> signals
>>>>>>>> Max msgqueue size 819200 819200
>>>> bytes
>>>>>>>> Max nice priority 0 0
>>>>>>>> Max realtime priority 0 0
>>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>>
>>>>>>>>
>>>>>>>> Too may open files:
>>>>>>>> # lsof -p 51798 | wc -l
>>>>>>>> 6679
>>>>>>>>
>>>>>>>>
>>>>>>>> Most of open files are TCP sockets:
>>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>>> protocol: TCPv6
>>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>>> 6413
>>>>>>> Some potentially dumb questions:
>>>>>>>
>>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>>
>>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>>
>>>>>>> 3. What is your <Connector> configuration?
>>>>>>>
>>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>>> of
>>>> the sub-processes which represent the threads of the main process?
>>>>>>> Different kernel versions count things differently.
>>>>>>>
>>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>>> those sockets are bound to any specific port (listen or connect)?
>>>>>>> -chris
>>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>> --
>>> Noelette Stout
>>>
>>> ITS Enterprise Applications - Application Administrator - Senior
>>>
>>> Business Administration Building, Rm 109L
>>> 921 South 8th Ave 8037
>>> Idaho State University
>>> Pocatello ID 83209
>>> E-mail: stounoel "at" isu "dot" edu
>>> Desk: 208-282-2554
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 22.05.21 um 18:03 schrieb Yeggy Javadi:
> Here it is:
>
> # netstat -p -a --tcp | grep 130244
> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>
>
> # lsof -p 130244
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
> java 130244 root rtd DIR 8,3 4096 2 /
> java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
> java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
> java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
> java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
> java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
> java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
> java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
> java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
> java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
> java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
> java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
> java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
> java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
> java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
> java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
> java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
> java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
> java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
> java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
> java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
> java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
> java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
> java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
> java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
> java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
> java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
> java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
> java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
> java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
> java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
> java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
> java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
> java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
> java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
> java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
> java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
> java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
> java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
> java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
> java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
> java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
> java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
> java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
> java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
> java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
> java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
> java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
> java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
> java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
> java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
> java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
> java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
> java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
> java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
> java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
> java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
> java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
> java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
> java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
> java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
> java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
> java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
> java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
> java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
> java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
> java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
> java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
> java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
> java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
> java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
> java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
> java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
> java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
> java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
> java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
> java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
> java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
> java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
> java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
> java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
> java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
> java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
> java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
> java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
> java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
> java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
> java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
> java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
> java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
> java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
> java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
> java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
> java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
> java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
> java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
> java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
> java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
> java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
> java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
> java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
> java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
> java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
> java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
> java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
> java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
> java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
> java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
> java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
> java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
> java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
> java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
> java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
> java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
> java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
> java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
> java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
> java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
> java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
> java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
> java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
> java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
> java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
> java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
> java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
> java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
> java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
> java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
> java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
> java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
> java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
> java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
> java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
> java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
> java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
> java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
> java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
> java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
> java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
> java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
> java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
> java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
> java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
> java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
> java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
> java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
> ...........................
> Keep seeing the same line
How many of those lines are there? (lsof -p ... | wc -l)
Are you sure there are more than before? Maybe your ulimits where higher
on your older system and are now
bothering you?
What is the setting of ulimit for Tomcats process? (cat /proc/<PID>/limits)
Felix
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Saturday, May 22, 2021 10:22 AM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Mmm. Nothing very special in that netstat output.
> The sockets seen there look quite normal for tomcat, and there are not a lot.
> What about the IPv4 sockets ? (remove the -6 in your netstat command)
>
> Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
> Where is the difference between the lsof count (19948) and the netstat count (25) ?
> (genuine question, I'm puzzled too)
>
> Can you give an example of the "lsof -p 130244" output lines ?
> (not all 19948 please, but enough to see some variety)
>
> On 21.05.2021 16:11, Yeggy Javadi wrote:
>> Hi,
>> Here its is:
>> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>> 19948
>>
>> # netstat -p -a -6 --tcp | grep 130244
>> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
>> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
>> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
>> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
>> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
>> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
>> Thanks
>> -----Original Message-----
>> From: Noelette Stout <no...@isu.edu>
>> Sent: Friday, May 21, 2021 8:28 AM
>> To: Tomcat Users List <us...@tomcat.apache.org>
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> ss has all the same options as netstat
>>
>> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
>> <aw...@ice-sa.com>
>> wrote:
>>
>>> Soyrry to top-post, but it's getting cluttered down there..
>>>
>>> The next thing that you may want to do :
>>>
>>> > netstat -p -a -6 --tcp
>>>
>>> That is an alternative list of sockets, which also shows the "tcp state"
>>> of the sockets.
>>> To get only the ones of the tomcat JVM PID, filter with grep based of
>>> the last column.
>>> The type of thing you are looking for is the column which should show
>>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>>
>>> The options above :
>>> -p : show PID and program
>>> -a : show all sockets states
>>> -6 : only inet v6
>>> --tcp : only TCP sockets
>>>
>>> "netstat" may not be on your system by default, and you may need to
>>> install it.
>>> An alternative is "ss", but I don't know the options.
>>>
>>>
>>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>>> Hi,
>>>> Yes; that is what I get and as you can see among 8028 open files,
>>>> 7474
>>> are for TCPv6 sockets:
>>>> java 130244 root 7805u sock 0,9 0t0 12294251
>>> protocol: TCPv6
>>>> # ps -ef | grep tomcat
>>>> root 130244 1 1 11:01 ? 00:06:20
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>> #lsof -p 130244 | wc -l
>>>> 8028
>>>>
>>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>>> 7474
>>>>
>>>> Thanks
>>>>
>>>> -----Original Message-----
>>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>>> Sent: Thursday, May 20, 2021 4:19 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Hi.
>>>> According to the list below, you have 2 java (JVM) processes running
>>>> on
>>> your system.
>>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>>> when
>>> you look at the whole command-line.
>>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>>> relevant here.
>>>> So you should run lsof with the "-p 130244" option, to show only the
>>> files opened by the tomcat JVM.
>>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>>> "apache-tomcat"
>>>> (or anything else unique in that line)
>>>>
>>>>
>>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>>> Hi Chris,
>>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>>
>>>>> Below is the list of running processes on my server:
>>>>> UID PID PPID C STIME TTY TIME CMD
>>>>> root 1 0 0 May07 ? 00:00:14
>>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>>> root 6 2 0 May07 ? 00:00:00
>>> [kworker/0:0H-kblockd]
>>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>>> root 19 2 0 May07 ? 00:00:00
>>> [kworker/1:0H-kblockd]
>>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>>> root 25 2 0 May07 ? 00:00:00
>>> [kworker/2:0H-kblockd]
>>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>>> root 31 2 0 May07 ? 00:00:00
>>> [kworker/3:0H-kblockd]
>>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>>> root 37 2 0 May07 ? 00:00:00
>>> [kworker/4:0H-kblockd]
>>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>>> root 43 2 0 May07 ? 00:00:00
>>> [kworker/5:0H-kblockd]
>>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>>> root 49 2 0 May07 ? 00:00:00
>>> [kworker/6:0H-kblockd]
>>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>>> root 55 2 0 May07 ? 00:00:00
>>> [kworker/7:0H-kblockd]
>>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>>> root 540 2 0 May07 ? 00:00:01
>>> [kworker/7:1H-kblockd]
>>>>> root 541 2 0 May07 ? 00:00:01
>>> [kworker/5:1H-kblockd]
>>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 613 2 0 May07 ? 00:00:01
>>> [kworker/6:1H-kblockd]
>>>>> root 650 2 0 May07 ? 00:00:01
>>> [kworker/4:1H-kblockd]
>>>>> root 659 2 0 May07 ? 00:00:02
>>> [kworker/3:1H-kblockd]
>>>>> root 660 2 0 May07 ? 00:00:02
>>> [kworker/2:1H-kblockd]
>>>>> root 662 2 0 May07 ? 00:00:01
>>> [kworker/0:1H-kblockd]
>>>>> root 679 1 0 May07 ? 00:00:03
>>> /usr/lib/systemd/systemd-journald
>>>>> root 682 2 0 May07 ? 00:00:02
>>> [kworker/1:1H-kblockd]
>>>>> root 716 1 0 May07 ? 00:00:01
>>> /usr/lib/systemd/systemd-udevd
>>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>>> dbus 883 1 0 May07 ? 00:00:03
>>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>>> --systemd-activation --syslog-only
>>>>> root 885 1 0 May07 ? 00:02:42
>>> /usr/sbin/NetworkManager --no-daemon
>>>>> root 887 1 0 May07 ? 00:00:44
>>> /usr/sbin/irqbalance --foreground
>>>>> root 888 1 0 May07 ? 00:00:00
>>> /usr/bin/VGAuthService -s
>>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>>> --logger=files
>>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>>> --fill-watermark=0
>>>>> root 946 890 0 May07 ? 00:00:06
>>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>>> --logger=files
>>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>>> -d2
>>>>> root 962 890 0 May07 ? 00:00:12
>>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>>> -D
>>>>> root 970 1 0 May07 ? 00:00:02
>>> /usr/lib/systemd/systemd-logind
>>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>>> -n
>>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>>> root 6223 1 0 May07 ? 00:07:27
>>> /usr/local/vtl/bin/sanidmgr
>>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>>> root 6654 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6655 6654 0 May07 ? 00:05:18
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6656 6655 0 May07 ? 00:00:36
>>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>>> root 6721 1 0 May07 ? 00:00:35
>>> /usr/local/vtl/bin/iscliproxy
>>>>> root 6747 1 0 May07 ? 00:00:40
>>> /usr/local/vtl/bin/ipstorlog
>>>>> root 6780 1 0 May07 ? 00:04:50
>>> /usr/local/vtl/bin/snmpd
>>>>> root 7722 1 0 May07 ? 00:00:00
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7723 7722 0 May07 ? 01:08:12
>>> /usr/local/vtl/bin/ipstorsm
>>>>> root 7901 1 0 May07 ? 00:00:00
>>> /usr/lib/systemd/systemd --user
>>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>>> -- \u --noclear tty1 linux
>>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>>> -
>>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>>> h
>>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>>> p
>>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>>> -
>>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>>> a
>>> 1,diffie-hellman-group14-sha1
>>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>>> m
>>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>>> root 128108 2 0 08:40 ? 00:00:00
>>> [kworker/4:1-events_power_efficient]
>>>>> root 128360 2 0 09:10 ? 00:00:00
>>> [kworker/4:2-cgroup_destroy]
>>>>> root 128430 2 0 09:30 ? 00:00:00
>>> [kworker/6:0-mm_percpu_wq]
>>>>> root 128738 2 0 11:00 ? 00:00:00
>>> [kworker/3:3-mm_percpu_wq]
>>>>> root 128761 2 0 11:00 ? 00:00:00
>>> [kworker/2:1-mm_percpu_wq]
>>>>> root 130244 1 1 11:01 ? 00:03:50
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC
>>> -XX:CMSInitiatingOccupancyFraction=75
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>>> -Des.path.home=/usr/share/elasticsearch -cp
>>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>>> t
>>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>>> -Des.default.path.home=/usr/share/elasticsearch
>>> -Des.default.path.logs=/var/log/elasticsearch
>>> -Des.default.path.data=/var/lib/elasticsearch
>>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>>> -Des.default.path.conf=/etc/elasticsearch
>>> org.elasticsearch.bootstrap.Elasticsearch
>>>>> postgres 130548 1 0 11:01 ? 00:00:00
>>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>>> process
>>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>>> checkpointer process
>>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>>> process
>>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>>> writer process
>>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>>> autovacuum launcher process
>>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>>> collector process
>>>>> root 130570 1 0 11:01 ? 00:00:01
>>> /usr/local/freestor/bin/fmsd
>>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42660) idle
>>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>>> freestor 127.0.0.1(42662) idle
>>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>>> freestor 127.0.0.1(42664) idle
>>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>>> root 131347 2 0 13:49 ? 00:00:00
>>> [kworker/5:1-mm_percpu_wq]
>>>>> root 131364 2 0 13:56 ? 00:00:00
>>> [kworker/u16:2-events_unbound]
>>>>> root 131401 2 0 14:05 ? 00:00:00
>>> [kworker/u16:1-events_unbound]
>>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>>> root 131515 2 0 14:49 ? 00:00:00
>>> [kworker/1:0-events_freezable]
>>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>>
>>>>> Thanks
>>>>> -----Original Message-----
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Yeggy,
>>>>>
>>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>>> Hi Chris,
>>>>>> Thanks for your email. Below are replies to your questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>>> 9.3
>>> to 9.6.
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>> [YJ] The application error can occur in any place with the (Too
>>>>>> many
>>> open files) error when the limit of open files for tomcat has reached
>>> (262144 in my environment), for example, I can get a connectivity
>>> error when pulling info from a server as below :
>>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>>> connectivity.Error retrieving IPStorConfg for
>>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>> [YJ] It is as below in server.xml:
>>>>>>
>>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>>> maxThreads="150"
>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>> enableLookups="false" acceptCount="100"
>>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>>> compression="on"
>>>>>> compressionMinSize="2048"
>>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>>
>>> compressableMimeType="text/html,text/xml,application/json"
>>>>>> redirectPort="443" /> ...........
>>>>>> <Connector address="0.0.0.0" port="443"
>>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>>> maxThreads="150" scheme="https" secure="true"
>>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>>
>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>>> keystoreFile="conf/freestor"
>>> keystorePass="freestor"/>
>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>>> redirectPort="443" secretRequired="false" />
>>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>>> Is that intentional? Which one are you using? Or both?
>>>>>
>>>>> Are you using AJP? If not, disable the connetor entirely.
>>>>> Definitely
>>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>>> files for that process
>>>>> Ok.
>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>> [YJ] Here is the netstat output:
>>>>>> Active Internet connections (w/o servers)
>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>>> State
>>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>>> ESTABLISHED
>>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>>> ESTABLISHED
>>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>>> ESTABLISHED
>>>>> That's not helpful. I want to see what the JVM process is holding,
>>>>> not
>>> what the whole OS is handling.
>>>>> My guess is that lsof is going to be a better bet for inspecting
>>>>> the
>>> process. But if you get "too many open files" being thrown inside the
>>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>>> process, then something else is going on. But remember to limit your
>>> lsof to *just the JVM* process.
>>>>> -chris
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>>> To: users@tomcat.apache.org
>>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>>
>>>>>> Yeggy,
>>>>>>
>>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>>> Hi,
>>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>>> changes in Linux
>>>>>>> 8 that can explain this
>>>>>>>
>>>>>>> Configuration:
>>>>>>> apache-tomcat-8.5.59
>>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>>> application using Postgres and elastic search databases
>>>>>>>
>>>>>>>
>>>>>>> # ps -ef | grep tomcat
>>>>>>> root 51798 1 1 11:16 ? 00:06:01
>>> /usr/local/jre/bin/java
>>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>>> properties
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>>> -Djdk.tls.ephemeralDHKeySize=2048
>>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>>> -Dignore.endorsed.dirs= -classpath
>>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>>> -Dcatalina.home=/usr/local/apache-tomcat
>>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>>> org.apache.catalina.startup.Bootstrap start
>>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>>> tomcat
>>>>>>>
>>>>>>> # cat /proc/51798/limits
>>>>>>> Limit Soft Limit Hard Limit
>>> Units
>>>>>>> Max cpu time unlimited unlimited
>>> seconds
>>>>>>> Max file size unlimited unlimited
>>> bytes
>>>>>>> Max data size unlimited unlimited
>>> bytes
>>>>>>> Max stack size 8388608 unlimited
>>> bytes
>>>>>>> Max core file size unlimited unlimited
>>> bytes
>>>>>>> Max resident set unlimited unlimited
>>> bytes
>>>>>>> Max processes 256518 256518
>>> processes
>>>>>>> Max open files 262144 262144
>>> files
>>>>>>> Max locked memory 65536 65536
>>> bytes
>>>>>>> Max address space unlimited unlimited
>>> bytes
>>>>>>> Max file locks unlimited unlimited
>>> locks
>>>>>>> Max pending signals 256518 256518
>>> signals
>>>>>>> Max msgqueue size 819200 819200
>>> bytes
>>>>>>> Max nice priority 0 0
>>>>>>> Max realtime priority 0 0
>>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>>
>>>>>>>
>>>>>>> Too may open files:
>>>>>>> # lsof -p 51798 | wc -l
>>>>>>> 6679
>>>>>>>
>>>>>>>
>>>>>>> Most of open files are TCP sockets:
>>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>>> protocol: TCPv6
>>>>>>>
>>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>>> 6413
>>>>>> Some potentially dumb questions:
>>>>>>
>>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>>
>>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>>
>>>>>> 3. What is your <Connector> configuration?
>>>>>>
>>>>>> 4. Are you counting all the open files for a single process or all
>>>>>> of
>>> the sub-processes which represent the threads of the main process?
>>>>>> Different kernel versions count things differently.
>>>>>>
>>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>>> those sockets are bound to any specific port (listen or connect)?
>>>>>> -chris
>>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> --
>> Noelette Stout
>>
>> ITS Enterprise Applications - Application Administrator - Senior
>>
>> Business Administration Building, Rm 109L
>> 921 South 8th Ave 8037
>> Idaho State University
>> Pocatello ID 83209
>> E-mail: stounoel "at" isu "dot" edu
>> Desk: 208-282-2554
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Here it is:
# netstat -p -a --tcp | grep 130244
tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
tcp6 0 1 Yeggy-F8-FMSVA:53618 10.12.3.78:https SYN_SENT 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 1 Yeggy-F8-FMSVA:42922 10.12.3.77:https SYN_SENT 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:35794 172.22.22.192:https ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
# lsof -p 130244
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 130244 root cwd DIR 8,2 4096 157664 /usr/local/freestor/bin
java 130244 root rtd DIR 8,3 4096 2 /
java 130244 root txt REG 8,2 8712 8913 /usr/local/jdk/jre1.8.0_271/bin/java
java 130244 root mem REG 8,2 498864 9007 /usr/local/jdk/jre1.8.0_271/lib/amd64/libfontmanager.so
java 130244 root mem REG 8,2 39176 9006 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt_headless.so
java 130244 root mem REG 8,2 759184 8996 /usr/local/jdk/jre1.8.0_271/lib/amd64/libawt.so
java 130244 root mem REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
java 130244 root mem REG 8,2 99680 133076 /usr/lib64/libgcc_s-8-20191121.so.1
java 130244 root mem REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
java 130244 root mem REG 8,2 283368 8980 /usr/local/jdk/jre1.8.0_271/lib/amd64/libsunec.so
java 130244 root mem REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
java 130244 root mem REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
java 130244 root mem REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
java 130244 root mem REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
java 130244 root mem REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
java 130244 root mem REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
java 130244 root mem REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
java 130244 root mem REG 8,2 51984 8995 /usr/local/jdk/jre1.8.0_271/lib/amd64/libmanagement.so
java 130244 root mem REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
java 130244 root mem REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
java 130244 root mem REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
java 130244 root mem REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
java 130244 root mem REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
java 130244 root mem REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
java 130244 root mem REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
java 130244 root mem REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
java 130244 root mem REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
java 130244 root mem REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
java 130244 root mem REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
java 130244 root mem REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
java 130244 root mem REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
java 130244 root mem REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
java 130244 root mem REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
java 130244 root mem REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
java 130244 root mem REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
java 130244 root mem REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
java 130244 root mem REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
java 130244 root mem REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
java 130244 root mem REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
java 130244 root mem REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
java 130244 root mem REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
java 130244 root mem REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
java 130244 root mem REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
java 130244 root mem REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
java 130244 root mem REG 8,2 113008 8989 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnet.so
java 130244 root mem REG 8,2 93872 9003 /usr/local/jdk/jre1.8.0_271/lib/amd64/libnio.so
java 130244 root mem REG 8,2 337024 395971 /usr/lib/locale/en_US.utf8/LC_CTYPE
java 130244 root mem REG 8,2 2586930 390297 /usr/lib/locale/en_US.utf8/LC_COLLATE
java 130244 root mem REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
java 130244 root mem REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
java 130244 root mem REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
java 130244 root mem REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
java 130244 root mem REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
java 130244 root mem REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
java 130244 root mem REG 8,2 54 395975 /usr/lib/locale/en_US.utf8/LC_NUMERIC
java 130244 root mem REG 8,2 3316 521695 /usr/lib/locale/en_US.utf8/LC_TIME
java 130244 root mem REG 8,2 286 521693 /usr/lib/locale/en_US.utf8/LC_MONETARY
java 130244 root mem REG 8,2 26998 130155 /usr/lib64/gconv/gconv-modules.cache
java 130244 root mem REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
java 130244 root mem REG 8,2 127016 8988 /usr/local/jdk/jre1.8.0_271/lib/amd64/libzip.so
java 130244 root mem REG 8,2 76880 133068 /usr/lib64/libnss_files-2.28.so
java 130244 root mem REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
java 130244 root mem REG 8,2 46432 134002 /usr/lib64/libnss_sss.so.2
java 130244 root mem REG 8,2 231960 8986 /usr/local/jdk/jre1.8.0_271/lib/amd64/libjava.so
java 130244 root mem REG 8,2 66112 8975 /usr/local/jdk/jre1.8.0_271/lib/amd64/libverify.so
java 130244 root mem REG 8,2 68944 133091 /usr/lib64/librt-2.28.so
java 130244 root mem REG 8,2 2191808 133023 /usr/lib64/libm-2.28.so
java 130244 root mem REG 8,2 17108824 8969 /usr/local/jdk/jre1.8.0_271/lib/amd64/server/libjvm.so
java 130244 root mem REG 8,2 3154704 129871 /usr/lib64/libc-2.28.so
java 130244 root mem REG 8,2 28968 133013 /usr/lib64/libdl-2.28.so
java 130244 root mem REG 8,2 108616 8973 /usr/local/jdk/jre1.8.0_271/lib/amd64/jli/libjli.so
java 130244 root mem REG 8,2 321552 133079 /usr/lib64/libpthread-2.28.so
java 130244 root mem REG 8,2 252280 129864 /usr/lib64/ld-2.28.so
java 130244 root mem REG 8,3 32768 18 /tmp/hsperfdata_root/130244
java 130244 root mem REG 8,2 57 395973 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
java 130244 root mem REG 8,2 34 390317 /usr/lib/locale/en_US.utf8/LC_PAPER
java 130244 root mem REG 8,2 77 395974 /usr/lib/locale/en_US.utf8/LC_NAME
java 130244 root mem REG 8,2 167 519390 /usr/lib/locale/en_US.utf8/LC_ADDRESS
java 130244 root mem REG 8,2 59 521694 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
java 130244 root mem REG 8,2 23 521692 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
java 130244 root mem REG 8,2 368 521691 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
java 130244 root 0r CHR 1,3 0t0 1037 /dev/null
java 130244 root 1w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
java 130244 root 2w REG 8,6 394654 1835011 /apps/apache-tomcat/logs/catalina.out
java 130244 root 3r REG 8,3 9253600 129922 /var/lib/sss/mc/passwd
java 130244 root 4u unix 0xffff985d1bd7b180 0t0 11980864 type=STREAM
java 130244 root 5r REG 8,2 67053209 9140 /usr/local/jdk/jre1.8.0_271/lib/rt.jar
java 130244 root 6r REG 8,2 36132 8837 /usr/local/apache-tomcat-8.5.59/bin/bootstrap.jar
java 130244 root 7r REG 8,2 25287 8835 /usr/local/apache-tomcat-8.5.59/bin/commons-daemon.jar
java 130244 root 8r REG 8,2 51205 8857 /usr/local/apache-tomcat-8.5.59/bin/tomcat-juli.jar
java 130244 root 9w REG 8,6 21883 1835065 /apps/apache-tomcat/logs/catalina.2021-05-20.log
java 130244 root 10w REG 8,6 761 1835066 /apps/apache-tomcat/logs/localhost.2021-05-20.log
java 130244 root 11w REG 8,6 0 1835067 /apps/apache-tomcat/logs/manager.2021-05-20.log
java 130244 root 12w REG 8,6 0 1835068 /apps/apache-tomcat/logs/host-manager.2021-05-20.log
java 130244 root 13r REG 8,2 158590 8883 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-zh-CN.jar
java 130244 root 14r REG 8,2 38561 8875 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ru.jar
java 130244 root 15r REG 8,2 26872 8885 /usr/local/apache-tomcat-8.5.59/lib/jaspic-api.jar
java 130244 root 16r REG 8,2 155094 8866 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-fr.jar
java 130244 root 17r REG 8,2 215400 8874 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util-scan.jar
java 130244 root 18r REG 8,2 868336 8888 /usr/local/apache-tomcat-8.5.59/lib/tomcat-coyote.jar
java 130244 root 19r REG 8,2 228391 8879 /usr/local/apache-tomcat-8.5.59/lib/tomcat-websocket.jar
java 130244 root 20r REG 8,2 120073 8876 /usr/local/apache-tomcat-8.5.59/lib/catalina-ha.jar
java 130244 root 21r REG 8,2 106051 8869 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-es.jar
java 130244 root 22r REG 8,2 38241 8882 /usr/local/apache-tomcat-8.5.59/lib/websocket-api.jar
java 130244 root 23r REG 8,2 1705863 8860 /usr/local/apache-tomcat-8.5.59/lib/catalina.jar
java 130244 root 24r REG 8,2 54667 8878 /usr/local/apache-tomcat-8.5.59/lib/catalina-ant.jar
java 130244 root 25r REG 8,2 72656 8865 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-de.jar
java 130244 root 26r REG 8,2 284036 8872 /usr/local/apache-tomcat-8.5.59/lib/tomcat-dbcp.jar
java 130244 root 27r REG 8,2 10673 8887 /usr/local/apache-tomcat-8.5.59/lib/tomcat-api.jar
java 130244 root 28r REG 8,2 173282 8880 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ko.jar
java 130244 root 29r REG 8,2 294856 8861 /usr/local/apache-tomcat-8.5.59/lib/catalina-tribes.jar
java 130244 root 30r REG 8,2 174393 8870 /usr/local/apache-tomcat-8.5.59/lib/tomcat-i18n-ja.jar
java 130244 root 31r REG 8,2 599431 8884 /usr/local/apache-tomcat-8.5.59/lib/jasper.jar
java 130244 root 32r REG 8,2 89695 8877 /usr/local/apache-tomcat-8.5.59/lib/el-api.jar
java 130244 root 33r REG 8,2 78087 8886 /usr/local/apache-tomcat-8.5.59/lib/catalina-storeconfig.jar
java 130244 root 34r REG 8,2 34798 8873 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jni.jar
java 130244 root 35r REG 8,2 2450404 8864 /usr/local/apache-tomcat-8.5.59/lib/ecj-4.6.3.jar
java 130244 root 36r REG 8,2 62004 8863 /usr/local/apache-tomcat-8.5.59/lib/jsp-api.jar
java 130244 root 37r REG 8,2 148761 8881 /usr/local/apache-tomcat-8.5.59/lib/tomcat-jdbc.jar
java 130244 root 38r REG 8,2 244451 8862 /usr/local/apache-tomcat-8.5.59/lib/servlet-api.jar
java 130244 root 39r REG 8,2 168382 8867 /usr/local/apache-tomcat-8.5.59/lib/jasper-el.jar
java 130244 root 40r REG 8,2 169952 8868 /usr/local/apache-tomcat-8.5.59/lib/tomcat-util.jar
java 130244 root 41r REG 8,2 12373 8871 /usr/local/apache-tomcat-8.5.59/lib/annotations-api.jar
java 130244 root 42r REG 8,2 1907341 9138 /usr/local/jdk/jre1.8.0_271/lib/jsse.jar
java 130244 root 43r REG 8,2 3860563 9129 /usr/local/jdk/jre1.8.0_271/lib/ext/cldrdata.jar
java 130244 root 44u unix 0xffff985d1bd7a400 0t0 11980871 type=STREAM
java 130244 root 45r REG 8,2 1179462 9128 /usr/local/jdk/jre1.8.0_271/lib/ext/localedata.jar
java 130244 root 46r CHR 1,8 0t0 1041 /dev/random
java 130244 root 47r CHR 1,9 0t0 1042 /dev/urandom
java 130244 root 48r CHR 1,8 0t0 1041 /dev/random
java 130244 root 49r CHR 1,8 0t0 1041 /dev/random
java 130244 root 50r CHR 1,9 0t0 1042 /dev/urandom
java 130244 root 51r CHR 1,9 0t0 1042 /dev/urandom
java 130244 root 52r REG 8,2 118047 9136 /usr/local/jdk/jre1.8.0_271/lib/jce.jar
java 130244 root 53r REG 8,2 283206 9127 /usr/local/jdk/jre1.8.0_271/lib/ext/sunpkcs11.jar
java 130244 root 54r REG 8,2 43191 9125 /usr/local/jdk/jre1.8.0_271/lib/ext/sunec.jar
java 130244 root 55r REG 8,2 285868 9126 /usr/local/jdk/jre1.8.0_271/lib/ext/sunjce_provider.jar
java 130244 root 56r REG 8,2 3559360 9139 /usr/local/jdk/jre1.8.0_271/lib/resources.jar
java 130244 root 57r REG 8,2 3135658 9133 /usr/local/jdk/jre1.8.0_271/lib/charsets.jar
java 130244 root 58u IPv6 11980876 0t0 TCP *:http (LISTEN)
java 130244 root 59u unix 0xffff985d1bd7e780 0t0 11980877 type=STREAM
java 130244 root 60r FIFO 0,13 0t0 11980879 pipe
java 130244 root 61w FIFO 0,13 0t0 11980879 pipe
java 130244 root 62u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 63u IPv6 11980880 0t0 TCP *:https (LISTEN)
java 130244 root 64r FIFO 0,13 0t0 11980884 pipe
java 130244 root 65w FIFO 0,13 0t0 11980884 pipe
java 130244 root 66u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 67u IPv6 11980885 0t0 TCP *:8009 (LISTEN)
java 130244 root 68r FIFO 0,13 0t0 11980886 pipe
java 130244 root 69w FIFO 0,13 0t0 11980886 pipe
java 130244 root 70u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 71w REG 8,6 342714 1835069 /apps/apache-tomcat/logs/localhost_access_log.2021-05-21.txt
java 130244 root 72r FIFO 0,13 0t0 11968845 pipe
java 130244 root 73w FIFO 0,13 0t0 11968845 pipe
java 130244 root 74u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 75r FIFO 0,13 0t0 11968846 pipe
java 130244 root 76w FIFO 0,13 0t0 11968846 pipe
java 130244 root 77u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 78u sock 0,9 0t0 12154374 protocol: TCPv6
java 130244 root 79r FIFO 0,13 0t0 11968847 pipe
java 130244 root 80w FIFO 0,13 0t0 11968847 pipe
java 130244 root 81u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 82r FIFO 0,13 0t0 11968848 pipe
java 130244 root 83w FIFO 0,13 0t0 11968848 pipe
java 130244 root 84u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 86r FIFO 0,13 0t0 11968849 pipe
java 130244 root 87w FIFO 0,13 0t0 11968849 pipe
java 130244 root 88u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 89r FIFO 0,13 0t0 11968850 pipe
java 130244 root 90w FIFO 0,13 0t0 11968850 pipe
java 130244 root 91u a_inode 0,14 0 10970 [eventpoll]
java 130244 root 93u IPv6 11968853 0t0 TCP localhost.localdomain:mxi (LISTEN)
java 130244 root 94u IPv6 12298916 0t0 TCP localhost.localdomain:45734->localhost.localdomain:postgres (ESTABLISHED)
java 130244 root 95u sock 0,9 0t0 12017983 protocol: TCPv6
java 130244 root 96u sock 0,9 0t0 11983086 protocol: TCPv6
java 130244 root 97u sock 0,9 0t0 11983087 protocol: TCPv6
java 130244 root 98u sock 0,9 0t0 11983088 protocol: TCPv6
java 130244 root 99u sock 0,9 0t0 11983089 protocol: TCPv6
java 130244 root 100u sock 0,9 0t0 11983090 protocol: TCPv6
java 130244 root 101u sock 0,9 0t0 11983091 protocol: TCPv6
java 130244 root 102u sock 0,9 0t0 11983092 protocol: TCPv6
java 130244 root 103u sock 0,9 0t0 11969375 protocol: TCPv6
java 130244 root 104u sock 0,9 0t0 11987339 protocol: TCPv6
java 130244 root 105u sock 0,9 0t0 11987347 protocol: TCPv6
java 130244 root 106u sock 0,9 0t0 11969384 protocol: TCPv6
java 130244 root 107u sock 0,9 0t0 11987348 protocol: TCPv6
java 130244 root 108r CHR 1,8 0t0 1041 /dev/random
java 130244 root 109w REG 8,6 19536380 1835053 /apps/apache-tomcat/logs/fms.log
java 130244 root 110u sock 0,9 0t0 11979291 protocol: TCPv6
java 130244 root 111u sock 0,9 0t0 11979292 protocol: TCPv6
java 130244 root 112w REG 8,6 835408 1835019 /apps/apache-tomcat/logs/obd.log
java 130244 root 113u sock 0,9 0t0 11987349 protocol: TCPv6
java 130244 root 114u sock 0,9 0t0 11987350 protocol: TCPv6
java 130244 root 115u sock 0,9 0t0 11987351 protocol: TCPv6
java 130244 root 116u sock 0,9 0t0 11987352 protocol: TCPv6
java 130244 root 117u sock 0,9 0t0 11987353 protocol: TCPv6
java 130244 root 118u sock 0,9 0t0 11987354 protocol: TCPv6
java 130244 root 119u sock 0,9 0t0 11987355 protocol: TCPv6
java 130244 root 120u sock 0,9 0t0 11987356 protocol: TCPv6
java 130244 root 121u sock 0,9 0t0 11987357 protocol: TCPv6
java 130244 root 122u sock 0,9 0t0 11987358 protocol: TCPv6
java 130244 root 123u sock 0,9 0t0 11987359 protocol: TCPv6
java 130244 root 124u sock 0,9 0t0 11987360 protocol: TCPv6
...........................
Keep seeing the same line
-----Original Message-----
From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
Sent: Saturday, May 22, 2021 10:22 AM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Mmm. Nothing very special in that netstat output.
The sockets seen there look quite normal for tomcat, and there are not a lot.
What about the IPv4 sockets ? (remove the -6 in your netstat command)
Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat JVM process.
Where is the difference between the lsof count (19948) and the netstat count (25) ?
(genuine question, I'm puzzled too)
Can you give an example of the "lsof -p 130244" output lines ?
(not all 19948 please, but enough to see some variety)
On 21.05.2021 16:11, Yeggy Javadi wrote:
> Hi,
> Here its is:
> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> 19948
>
> # netstat -p -a -6 --tcp | grep 130244
> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
> Thanks
> -----Original Message-----
> From: Noelette Stout <no...@isu.edu>
> Sent: Friday, May 21, 2021 8:28 AM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> ss has all the same options as netstat
>
> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl)
> <aw...@ice-sa.com>
> wrote:
>
>> Soyrry to top-post, but it's getting cluttered down there..
>>
>> The next thing that you may want to do :
>>
>> > netstat -p -a -6 --tcp
>>
>> That is an alternative list of sockets, which also shows the "tcp state"
>> of the sockets.
>> To get only the ones of the tomcat JVM PID, filter with grep based of
>> the last column.
>> The type of thing you are looking for is the column which should show
>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>
>> The options above :
>> -p : show PID and program
>> -a : show all sockets states
>> -6 : only inet v6
>> --tcp : only TCP sockets
>>
>> "netstat" may not be on your system by default, and you may need to
>> install it.
>> An alternative is "ss", but I don't know the options.
>>
>>
>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>> Hi,
>>> Yes; that is what I get and as you can see among 8028 open files,
>>> 7474
>> are for TCPv6 sockets:
>>> java 130244 root 7805u sock 0,9 0t0 12294251
>> protocol: TCPv6
>>>
>>> # ps -ef | grep tomcat
>>> root 130244 1 1 11:01 ? 00:06:20
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>> tomcat
>>>
>>> #lsof -p 130244 | wc -l
>>> 8028
>>>
>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>> 7474
>>>
>>> Thanks
>>>
>>> -----Original Message-----
>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>> Sent: Thursday, May 20, 2021 4:19 PM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Hi.
>>> According to the list below, you have 2 java (JVM) processes running
>>> on
>> your system.
>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>> when
>> you look at the whole command-line.
>>>
>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>> relevant here.
>>>
>>> So you should run lsof with the "-p 130244" option, to show only the
>> files opened by the tomcat JVM.
>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>> "apache-tomcat"
>>> (or anything else unique in that line)
>>>
>>>
>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>> Hi Chris,
>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>
>>>> Below is the list of running processes on my server:
>>>> UID PID PPID C STIME TTY TIME CMD
>>>> root 1 0 0 May07 ? 00:00:14
>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>> root 6 2 0 May07 ? 00:00:00
>> [kworker/0:0H-kblockd]
>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>> root 19 2 0 May07 ? 00:00:00
>> [kworker/1:0H-kblockd]
>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>> root 25 2 0 May07 ? 00:00:00
>> [kworker/2:0H-kblockd]
>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>> root 31 2 0 May07 ? 00:00:00
>> [kworker/3:0H-kblockd]
>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>> root 37 2 0 May07 ? 00:00:00
>> [kworker/4:0H-kblockd]
>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>> root 43 2 0 May07 ? 00:00:00
>> [kworker/5:0H-kblockd]
>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>> root 49 2 0 May07 ? 00:00:00
>> [kworker/6:0H-kblockd]
>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>> root 55 2 0 May07 ? 00:00:00
>> [kworker/7:0H-kblockd]
>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>> root 540 2 0 May07 ? 00:00:01
>> [kworker/7:1H-kblockd]
>>>> root 541 2 0 May07 ? 00:00:01
>> [kworker/5:1H-kblockd]
>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 613 2 0 May07 ? 00:00:01
>> [kworker/6:1H-kblockd]
>>>> root 650 2 0 May07 ? 00:00:01
>> [kworker/4:1H-kblockd]
>>>> root 659 2 0 May07 ? 00:00:02
>> [kworker/3:1H-kblockd]
>>>> root 660 2 0 May07 ? 00:00:02
>> [kworker/2:1H-kblockd]
>>>> root 662 2 0 May07 ? 00:00:01
>> [kworker/0:1H-kblockd]
>>>> root 679 1 0 May07 ? 00:00:03
>> /usr/lib/systemd/systemd-journald
>>>> root 682 2 0 May07 ? 00:00:02
>> [kworker/1:1H-kblockd]
>>>> root 716 1 0 May07 ? 00:00:01
>> /usr/lib/systemd/systemd-udevd
>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>> dbus 883 1 0 May07 ? 00:00:03
>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>> --systemd-activation --syslog-only
>>>> root 885 1 0 May07 ? 00:02:42
>> /usr/sbin/NetworkManager --no-daemon
>>>> root 887 1 0 May07 ? 00:00:44
>> /usr/sbin/irqbalance --foreground
>>>> root 888 1 0 May07 ? 00:00:00
>> /usr/bin/VGAuthService -s
>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>> --logger=files
>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>> --fill-watermark=0
>>>> root 946 890 0 May07 ? 00:00:06
>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>> --logger=files
>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>> -d2
>>>> root 962 890 0 May07 ? 00:00:12
>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>> -D
>>>> root 970 1 0 May07 ? 00:00:02
>> /usr/lib/systemd/systemd-logind
>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>> -n
>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>> root 6223 1 0 May07 ? 00:07:27
>> /usr/local/vtl/bin/sanidmgr
>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>> root 6654 1 0 May07 ? 00:00:00
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6655 6654 0 May07 ? 00:05:18
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6656 6655 0 May07 ? 00:00:36
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>> root 6721 1 0 May07 ? 00:00:35
>> /usr/local/vtl/bin/iscliproxy
>>>> root 6747 1 0 May07 ? 00:00:40
>> /usr/local/vtl/bin/ipstorlog
>>>> root 6780 1 0 May07 ? 00:04:50
>> /usr/local/vtl/bin/snmpd
>>>> root 7722 1 0 May07 ? 00:00:00
>> /usr/local/vtl/bin/ipstorsm
>>>> root 7723 7722 0 May07 ? 01:08:12
>> /usr/local/vtl/bin/ipstorsm
>>>> root 7901 1 0 May07 ? 00:00:00
>> /usr/lib/systemd/systemd --user
>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>> -- \u --noclear tty1 linux
>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss
>> -
>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-s
>> h
>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-grou
>> p
>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16
>> -
>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sh
>> a
>> 1,diffie-hellman-group14-sha1
>> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>> m -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.co
>> m
>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa
>> - sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>> root 128108 2 0 08:40 ? 00:00:00
>> [kworker/4:1-events_power_efficient]
>>>> root 128360 2 0 09:10 ? 00:00:00
>> [kworker/4:2-cgroup_destroy]
>>>> root 128430 2 0 09:30 ? 00:00:00
>> [kworker/6:0-mm_percpu_wq]
>>>> root 128738 2 0 11:00 ? 00:00:00
>> [kworker/3:3-mm_percpu_wq]
>>>> root 128761 2 0 11:00 ? 00:00:00
>> [kworker/2:1-mm_percpu_wq]
>>>> root 130244 1 1 11:01 ? 00:03:50
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC
>> -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>> -Des.path.home=/usr/share/elasticsearch -cp
>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elas
>> t
>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>> -Des.default.path.home=/usr/share/elasticsearch
>> -Des.default.path.logs=/var/log/elasticsearch
>> -Des.default.path.data=/var/lib/elasticsearch
>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>> -Des.default.path.conf=/etc/elasticsearch
>> org.elasticsearch.bootstrap.Elasticsearch
>>>> postgres 130548 1 0 11:01 ? 00:00:00
>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>> process
>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>> checkpointer process
>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>> process
>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>> writer process
>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>> autovacuum launcher process
>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>> collector process
>>>> root 130570 1 0 11:01 ? 00:00:01
>> /usr/local/freestor/bin/fmsd
>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>> freestor 127.0.0.1(42660) idle
>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>> freestor 127.0.0.1(42662) idle
>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>> freestor 127.0.0.1(42664) idle
>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>> root 131347 2 0 13:49 ? 00:00:00
>> [kworker/5:1-mm_percpu_wq]
>>>> root 131364 2 0 13:56 ? 00:00:00
>> [kworker/u16:2-events_unbound]
>>>> root 131401 2 0 14:05 ? 00:00:00
>> [kworker/u16:1-events_unbound]
>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>> root 131515 2 0 14:49 ? 00:00:00
>> [kworker/1:0-events_freezable]
>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>
>>>> Thanks
>>>> -----Original Message-----
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Yeggy,
>>>>
>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>> Hi Chris,
>>>>> Thanks for your email. Below are replies to your questions:
>>>>>
>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>> 9.3
>> to 9.6.
>>>>>
>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>> [YJ] The application error can occur in any place with the (Too
>>>>> many
>> open files) error when the limit of open files for tomcat has reached
>> (262144 in my environment), for example, I can get a connectivity
>> error when pulling info from a server as below :
>>>>>
>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>> connectivity.Error retrieving IPStorConfg for
>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>
>>>>> 3. What is your <Connector> configuration?
>>>>> [YJ] It is as below in server.xml:
>>>>>
>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>> maxThreads="150"
>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>> enableLookups="false" acceptCount="100"
>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>> compression="on"
>>>>> compressionMinSize="2048"
>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>
>> compressableMimeType="text/html,text/xml,application/json"
>>>>> redirectPort="443" /> ...........
>>>>> <Connector address="0.0.0.0" port="443"
>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>> maxThreads="150" scheme="https" secure="true"
>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>
>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>> keystoreFile="conf/freestor"
>> keystorePass="freestor"/>
>>>>>
>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>> redirectPort="443" secretRequired="false" />
>>>>
>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>> Is that intentional? Which one are you using? Or both?
>>>>
>>>> Are you using AJP? If not, disable the connetor entirely.
>>>> Definitely
>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>
>>>>> 4. Are you counting all the open files for a single process or all
>>>>> of
>> the sub-processes which represent the threads of the main process?
>>>>> Different kernel versions count things differently.
>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>> files for that process
>>>>
>>>> Ok.
>>>>
>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>> those sockets are bound to any specific port (listen or connect)?
>>>>> [YJ] Here is the netstat output:
>>>>> Active Internet connections (w/o servers)
>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State
>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>> ESTABLISHED
>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>> ESTABLISHED
>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>> ESTABLISHED
>>>>
>>>> That's not helpful. I want to see what the JVM process is holding,
>>>> not
>> what the whole OS is handling.
>>>>
>>>> My guess is that lsof is going to be a better bet for inspecting
>>>> the
>> process. But if you get "too many open files" being thrown inside the
>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>> process, then something else is going on. But remember to limit your
>> lsof to *just the JVM* process.
>>>>
>>>> -chris
>>>>
>>>>> -----Original Message-----
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Yeggy,
>>>>>
>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>> Hi,
>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>> protocol: TCPv6
>>>>>>
>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>> changes in Linux
>>>>>> 8 that can explain this
>>>>>>
>>>>>> Configuration:
>>>>>> apache-tomcat-8.5.59
>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>> application using Postgres and elastic search databases
>>>>>>
>>>>>>
>>>>>> # ps -ef | grep tomcat
>>>>>> root 51798 1 1 11:16 ? 00:06:01
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/b
>> i n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>> tomcat
>>>>>>
>>>>>>
>>>>>> # cat /proc/51798/limits
>>>>>> Limit Soft Limit Hard Limit
>> Units
>>>>>> Max cpu time unlimited unlimited
>> seconds
>>>>>> Max file size unlimited unlimited
>> bytes
>>>>>> Max data size unlimited unlimited
>> bytes
>>>>>> Max stack size 8388608 unlimited
>> bytes
>>>>>> Max core file size unlimited unlimited
>> bytes
>>>>>> Max resident set unlimited unlimited
>> bytes
>>>>>> Max processes 256518 256518
>> processes
>>>>>> Max open files 262144 262144
>> files
>>>>>> Max locked memory 65536 65536
>> bytes
>>>>>> Max address space unlimited unlimited
>> bytes
>>>>>> Max file locks unlimited unlimited
>> locks
>>>>>> Max pending signals 256518 256518
>> signals
>>>>>> Max msgqueue size 819200 819200
>> bytes
>>>>>> Max nice priority 0 0
>>>>>> Max realtime priority 0 0
>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>
>>>>>>
>>>>>> Too may open files:
>>>>>> # lsof -p 51798 | wc -l
>>>>>> 6679
>>>>>>
>>>>>>
>>>>>> Most of open files are TCP sockets:
>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>> protocol: TCPv6
>>>>>>
>>>>>>
>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>> 6413
>>>>>
>>>>> Some potentially dumb questions:
>>>>>
>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>
>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>
>>>>> 3. What is your <Connector> configuration?
>>>>>
>>>>> 4. Are you counting all the open files for a single process or all
>>>>> of
>> the sub-processes which represent the threads of the main process?
>>>>> Different kernel versions count things differently.
>>>>>
>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>> those sockets are bound to any specific port (listen or connect)?
>>>>>
>>>>> -chris
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> --
> Noelette Stout
>
> ITS Enterprise Applications - Application Administrator - Senior
>
> Business Administration Building, Rm 109L
> 921 South 8th Ave 8037
> Idaho State University
> Pocatello ID 83209
> E-mail: stounoel "at" isu "dot" edu
> Desk: 208-282-2554
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Mmm. Nothing very special in that netstat output.
The sockets seen there look quite normal for tomcat, and there are not a lot.
What about the IPv4 sockets ? (remove the -6 in your netstat command)
Looks like lsof is counting things which are not IPv6 TCP sockets belonging to the tomcat
JVM process.
Where is the difference between the lsof count (19948) and the netstat count (25) ?
(genuine question, I'm puzzled too)
Can you give an example of the "lsof -p 130244" output lines ?
(not all 19948 please, but enough to see some variety)
On 21.05.2021 16:11, Yeggy Javadi wrote:
> Hi,
> Here its is:
> # lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> 19948
>
> # netstat -p -a -6 --tcp | grep 130244
> tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
> tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
> tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
> tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
> tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
> tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
> Thanks
> -----Original Message-----
> From: Noelette Stout <no...@isu.edu>
> Sent: Friday, May 21, 2021 8:28 AM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> ss has all the same options as netstat
>
> On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) <aw...@ice-sa.com>
> wrote:
>
>> Soyrry to top-post, but it's getting cluttered down there..
>>
>> The next thing that you may want to do :
>>
>> > netstat -p -a -6 --tcp
>>
>> That is an alternative list of sockets, which also shows the "tcp state"
>> of the sockets.
>> To get only the ones of the tomcat JVM PID, filter with grep based of
>> the last column.
>> The type of thing you are looking for is the column which should show
>> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>>
>> The options above :
>> -p : show PID and program
>> -a : show all sockets states
>> -6 : only inet v6
>> --tcp : only TCP sockets
>>
>> "netstat" may not be on your system by default, and you may need to
>> install it.
>> An alternative is "ss", but I don't know the options.
>>
>>
>> On 21.05.2021 02:14, Yeggy Javadi wrote:
>>> Hi,
>>> Yes; that is what I get and as you can see among 8028 open files,
>>> 7474
>> are for TCPv6 sockets:
>>> java 130244 root 7805u sock 0,9 0t0 12294251
>> protocol: TCPv6
>>>
>>> # ps -ef | grep tomcat
>>> root 130244 1 1 11:01 ? 00:06:20
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
>> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
>> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
>> tomcat
>>>
>>> #lsof -p 130244 | wc -l
>>> 8028
>>>
>>> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
>>> 7474
>>>
>>> Thanks
>>>
>>> -----Original Message-----
>>> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
>>> Sent: Thursday, May 20, 2021 4:19 PM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Hi.
>>> According to the list below, you have 2 java (JVM) processes running
>>> on
>> your system.
>>> One (PID = 130244) is the JVM which runs tomcat. This is visible
>>> when
>> you look at the whole command-line.
>>>
>>> The other (PID = 130516) runs ElasticSearch, which I believe is not
>> relevant here.
>>>
>>> So you should run lsof with the "-p 130244" option, to show only the
>> files opened by the tomcat JVM.
>>> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
>> "apache-tomcat"
>>> (or anything else unique in that line)
>>>
>>>
>>> On 20.05.2021 21:00, Yeggy Javadi wrote:
>>>> Hi Chris,
>>>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>>>
>>>> Below is the list of running processes on my server:
>>>> UID PID PPID C STIME TTY TIME CMD
>>>> root 1 0 0 May07 ? 00:00:14
>> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>>>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>>>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>>>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>>>> root 6 2 0 May07 ? 00:00:00
>> [kworker/0:0H-kblockd]
>>>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>>>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>>>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>>>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>>>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>>>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>>>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>>>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>>>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>>>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>>>> root 19 2 0 May07 ? 00:00:00
>> [kworker/1:0H-kblockd]
>>>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>>>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>>>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>>>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>>>> root 25 2 0 May07 ? 00:00:00
>> [kworker/2:0H-kblockd]
>>>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>>>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>>>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>>>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>>>> root 31 2 0 May07 ? 00:00:00
>> [kworker/3:0H-kblockd]
>>>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>>>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>>>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>>>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>>>> root 37 2 0 May07 ? 00:00:00
>> [kworker/4:0H-kblockd]
>>>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>>>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>>>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>>>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>>>> root 43 2 0 May07 ? 00:00:00
>> [kworker/5:0H-kblockd]
>>>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>>>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>>>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>>>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>>>> root 49 2 0 May07 ? 00:00:00
>> [kworker/6:0H-kblockd]
>>>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>>>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>>>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>>>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>>>> root 55 2 0 May07 ? 00:00:00
>> [kworker/7:0H-kblockd]
>>>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>>>> root 58 2 0 May07 ? 00:00:00 [netns]
>>>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>>>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>>>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>>>> root 64 2 0 May07 ? 00:00:00 [writeback]
>>>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>>>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>>>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>>>> root 68 2 0 May07 ? 00:00:00 [crypto]
>>>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>>>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>>>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>>>> root 72 2 0 May07 ? 00:00:00 [md]
>>>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>>>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>>>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>>>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>>>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>>>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>>>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>>>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>>>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>>>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>>>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>>>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>>>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>>>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>>>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>>>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>>>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>>>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>>>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>>>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>>>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>>>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>>>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>>>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>>>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>>>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>>>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>>>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>>>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>>>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>>>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>>>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>>>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>>>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>>>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>>>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>>>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>>>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>>>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>>>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>>>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>>>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>>>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>>>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>>>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>>>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>>>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>>>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>>>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>>>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>>>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>>>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>>>> root 540 2 0 May07 ? 00:00:01
>> [kworker/7:1H-kblockd]
>>>> root 541 2 0 May07 ? 00:00:01
>> [kworker/5:1H-kblockd]
>>>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>>>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>>>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 613 2 0 May07 ? 00:00:01
>> [kworker/6:1H-kblockd]
>>>> root 650 2 0 May07 ? 00:00:01
>> [kworker/4:1H-kblockd]
>>>> root 659 2 0 May07 ? 00:00:02
>> [kworker/3:1H-kblockd]
>>>> root 660 2 0 May07 ? 00:00:02
>> [kworker/2:1H-kblockd]
>>>> root 662 2 0 May07 ? 00:00:01
>> [kworker/0:1H-kblockd]
>>>> root 679 1 0 May07 ? 00:00:03
>> /usr/lib/systemd/systemd-journald
>>>> root 682 2 0 May07 ? 00:00:02
>> [kworker/1:1H-kblockd]
>>>> root 716 1 0 May07 ? 00:00:01
>> /usr/lib/systemd/systemd-udevd
>>>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>>>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>>>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>>>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>>>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>>>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>>>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>>>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>>>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>>>> dbus 883 1 0 May07 ? 00:00:03
>> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
>> --systemd-activation --syslog-only
>>>> root 885 1 0 May07 ? 00:02:42
>> /usr/sbin/NetworkManager --no-daemon
>>>> root 887 1 0 May07 ? 00:00:44
>> /usr/sbin/irqbalance --foreground
>>>> root 888 1 0 May07 ? 00:00:00
>> /usr/bin/VGAuthService -s
>>>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>>>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
>> --logger=files
>>>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>>>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
>> --fill-watermark=0
>>>> root 946 890 0 May07 ? 00:00:06
>> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
>> --logger=files
>>>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
>> -d2
>>>> root 962 890 0 May07 ? 00:00:12
>> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>>>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
>> -D
>>>> root 970 1 0 May07 ? 00:00:02
>> /usr/lib/systemd/systemd-logind
>>>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>>>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
>> -n
>>>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>>>> root 6223 1 0 May07 ? 00:07:27
>> /usr/local/vtl/bin/sanidmgr
>>>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>>>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>>>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>>>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>>>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>>>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>>>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>>>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>>>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>>>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>>>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>>>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>>>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>>>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>>>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>>>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>>>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>>>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>>>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>>>> root 6654 1 0 May07 ? 00:00:00
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6655 6654 0 May07 ? 00:05:18
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6656 6655 0 May07 ? 00:00:36
>> /usr/local/vtl/bin/ipstorcomm -d 0
>>>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>>>> root 6699 1 0 May07 ? 00:00:12 ikmd
>>>> root 6721 1 0 May07 ? 00:00:35
>> /usr/local/vtl/bin/iscliproxy
>>>> root 6747 1 0 May07 ? 00:00:40
>> /usr/local/vtl/bin/ipstorlog
>>>> root 6780 1 0 May07 ? 00:04:50
>> /usr/local/vtl/bin/snmpd
>>>> root 7722 1 0 May07 ? 00:00:00
>> /usr/local/vtl/bin/ipstorsm
>>>> root 7723 7722 0 May07 ? 01:08:12
>> /usr/local/vtl/bin/ipstorsm
>>>> root 7901 1 0 May07 ? 00:00:00
>> /usr/lib/systemd/systemd --user
>>>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>>>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
>> -- \u --noclear tty1 linux
>>>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>>>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>>>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>>>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
>> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
>> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
>> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
>> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
>> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
>> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-
>> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
>> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sh
>> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group
>> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-
>> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha
>> 1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
>> -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
>> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
>> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
>> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
>> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
>> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
>> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
>> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-
>> sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>>>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>>>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>>>> root 128108 2 0 08:40 ? 00:00:00
>> [kworker/4:1-events_power_efficient]
>>>> root 128360 2 0 09:10 ? 00:00:00
>> [kworker/4:2-cgroup_destroy]
>>>> root 128430 2 0 09:30 ? 00:00:00
>> [kworker/6:0-mm_percpu_wq]
>>>> root 128738 2 0 11:00 ? 00:00:00
>> [kworker/3:3-mm_percpu_wq]
>>>> root 128761 2 0 11:00 ? 00:00:00
>> [kworker/2:1-mm_percpu_wq]
>>>> root 130244 1 1 11:01 ? 00:03:50
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
>> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
>> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
>> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC
>> -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
>> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
>> -Des.path.home=/usr/share/elasticsearch -cp
>> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elast
>> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
>> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
>> -Des.default.path.home=/usr/share/elasticsearch
>> -Des.default.path.logs=/var/log/elasticsearch
>> -Des.default.path.data=/var/lib/elasticsearch
>> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
>> -Des.default.path.conf=/etc/elasticsearch
>> org.elasticsearch.bootstrap.Elasticsearch
>>>> postgres 130548 1 0 11:01 ? 00:00:00
>> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>>>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
>> process
>>>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
>> checkpointer process
>>>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
>> process
>>>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
>> writer process
>>>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
>> autovacuum launcher process
>>>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
>> collector process
>>>> root 130570 1 0 11:01 ? 00:00:01
>> /usr/local/freestor/bin/fmsd
>>>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
>> freestor 127.0.0.1(42660) idle
>>>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
>> freestor 127.0.0.1(42662) idle
>>>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
>> freestor 127.0.0.1(42664) idle
>>>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>>>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>>>> root 131347 2 0 13:49 ? 00:00:00
>> [kworker/5:1-mm_percpu_wq]
>>>> root 131364 2 0 13:56 ? 00:00:00
>> [kworker/u16:2-events_unbound]
>>>> root 131401 2 0 14:05 ? 00:00:00
>> [kworker/u16:1-events_unbound]
>>>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>>>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>>>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>>>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>>>> root 131515 2 0 14:49 ? 00:00:00
>> [kworker/1:0-events_freezable]
>>>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>>>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>>>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>>>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>>>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>>>
>>>> Thanks
>>>> -----Original Message-----
>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>> Sent: Thursday, May 20, 2021 12:49 PM
>>>> To: users@tomcat.apache.org
>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>
>>>> Yeggy,
>>>>
>>>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>>>> Hi Chris,
>>>>> Thanks for your email. Below are replies to your questions:
>>>>>
>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>> [YJ] To support Linux 8, only Postgres was upgraded from version
>>>>> 9.3
>> to 9.6.
>>>>>
>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>> [YJ] The application error can occur in any place with the (Too
>>>>> many
>> open files) error when the limit of open files for tomcat has reached
>> (262144 in my environment), for example, I can get a connectivity
>> error when pulling info from a server as below :
>>>>>
>>>>> [InventoryPullerTask - 10.1.6.25] ERROR
>>>>> FSSDataCollectorService - Error : The XML configuration file
>>>>> failed to be retrieved for server 10.1.6.25. Check server
>>>>> connectivity.Error retrieving IPStorConfg for
>>>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>>>
>>>>> 3. What is your <Connector> configuration?
>>>>> [YJ] It is as below in server.xml:
>>>>>
>>>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>>>> maxThreads="150"
>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>> enableLookups="false" acceptCount="100"
>>>>> connectionTimeout="20000" disableUploadTimeout="true"
>>>>> compression="on"
>>>>> compressionMinSize="2048"
>>>>> noCompressionUserAgents="gozilla, traviata"
>>>>>
>> compressableMimeType="text/html,text/xml,application/json"
>>>>> redirectPort="443" /> ...........
>>>>> <Connector address="0.0.0.0" port="443"
>> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>>>> maxThreads="150" scheme="https" secure="true"
>>>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>>>>
>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>>>> keystoreFile="conf/freestor"
>> keystorePass="freestor"/>
>>>>>
>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>>>> redirectPort="443" secretRequired="false" />
>>>>
>>>> You have lots of settings on the HTTP connector but not the HTTPS one.
>>>> Is that intentional? Which one are you using? Or both?
>>>>
>>>> Are you using AJP? If not, disable the connetor entirely.
>>>> Definitely
>> never enable it on 0.0.0.0 unless you really know what you are doing.
>>>>
>>>>> 4. Are you counting all the open files for a single process or all
>>>>> of
>> the sub-processes which represent the threads of the main process?
>>>>> Different kernel versions count things differently.
>>>>> [YJ] I am just getting the process ID of tomcat and count open
>>>>> files for that process
>>>>
>>>> Ok.
>>>>
>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>> those sockets are bound to any specific port (listen or connect)?
>>>>> [YJ] Here is the netstat output:
>>>>> Active Internet connections (w/o servers)
>>>>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State
>>>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
>> ESTABLISHED
>>>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
>> ESTABLISHED
>>>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
>> ESTABLISHED
>>>>
>>>> That's not helpful. I want to see what the JVM process is holding,
>>>> not
>> what the whole OS is handling.
>>>>
>>>> My guess is that lsof is going to be a better bet for inspecting
>>>> the
>> process. But if you get "too many open files" being thrown inside the
>> JVM and you don't see a 1/4 million lines of "lsof" output on your
>> process, then something else is going on. But remember to limit your
>> lsof to *just the JVM* process.
>>>>
>>>> -chris
>>>>
>>>>> -----Original Message-----
>>>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>>>> Sent: Monday, May 17, 2021 5:18 PM
>>>>> To: users@tomcat.apache.org
>>>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>>>
>>>>> Yeggy,
>>>>>
>>>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>>>> Hi,
>>>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>> protocol: TCPv6
>>>>>>
>>>>>> I was running the same application on Linux 7.6 with the same
>>>>>> apache-tomcat version without seeing this issue. I saw that
>>>>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
>>>>>> Please help with throubleshooting and indicating if there is any
>>>>>> changes in Linux
>>>>>> 8 that can explain this
>>>>>>
>>>>>> Configuration:
>>>>>> apache-tomcat-8.5.59
>>>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
>>>>>> application using Postgres and elastic search databases
>>>>>>
>>>>>>
>>>>>> # ps -ef | grep tomcat
>>>>>> root 51798 1 1 11:16 ? 00:06:01
>> /usr/local/jre/bin/java
>> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
>> properties
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
>> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
>> -Djdk.tls.ephemeralDHKeySize=2048
>> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
>> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
>> -Dignore.endorsed.dirs= -classpath
>> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
>> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
>> -Dcatalina.home=/usr/local/apache-tomcat
>> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
>> org.apache.catalina.startup.Bootstrap start
>>>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
>> tomcat
>>>>>>
>>>>>>
>>>>>> # cat /proc/51798/limits
>>>>>> Limit Soft Limit Hard Limit
>> Units
>>>>>> Max cpu time unlimited unlimited
>> seconds
>>>>>> Max file size unlimited unlimited
>> bytes
>>>>>> Max data size unlimited unlimited
>> bytes
>>>>>> Max stack size 8388608 unlimited
>> bytes
>>>>>> Max core file size unlimited unlimited
>> bytes
>>>>>> Max resident set unlimited unlimited
>> bytes
>>>>>> Max processes 256518 256518
>> processes
>>>>>> Max open files 262144 262144
>> files
>>>>>> Max locked memory 65536 65536
>> bytes
>>>>>> Max address space unlimited unlimited
>> bytes
>>>>>> Max file locks unlimited unlimited
>> locks
>>>>>> Max pending signals 256518 256518
>> signals
>>>>>> Max msgqueue size 819200 819200
>> bytes
>>>>>> Max nice priority 0 0
>>>>>> Max realtime priority 0 0
>>>>>> Max realtime timeout unlimited unlimited us
>>>>>>
>>>>>>
>>>>>> Too may open files:
>>>>>> # lsof -p 51798 | wc -l
>>>>>> 6679
>>>>>>
>>>>>>
>>>>>> Most of open files are TCP sockets:
>>>>>> java 51798 root 6601u sock 0,9 0t0 3959342
>> protocol: TCPv6
>>>>>>
>>>>>>
>>>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>>>> 6413
>>>>>
>>>>> Some potentially dumb questions:
>>>>>
>>>>> 1. Did you upgrade anything recently (like Java VM)?
>>>>>
>>>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>>>
>>>>> 3. What is your <Connector> configuration?
>>>>>
>>>>> 4. Are you counting all the open files for a single process or all
>>>>> of
>> the sub-processes which represent the threads of the main process?
>>>>> Different kernel versions count things differently.
>>>>>
>>>>> 5. Running lsof, netstat, etc. can you see if some large number of
>> those sockets are bound to any specific port (listen or connect)?
>>>>>
>>>>> -chris
>>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> --
> Noelette Stout
>
> ITS Enterprise Applications - Application Administrator - Senior
>
> Business Administration Building, Rm 109L
> 921 South 8th Ave 8037
> Idaho State University
> Pocatello ID 83209
> E-mail: stounoel "at" isu "dot" edu
> Desk: 208-282-2554
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi,
Here its is:
# lsof -p 130244 | grep "protocol: TCPv6"| wc -l
19948
# netstat -p -a -6 --tcp | grep 130244
tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java
tcp6 0 0 [::]:https [::]:* LISTEN 130244/java
tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java
tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java
tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java
tcp6 0 0 [::]:rmiregistry [::]:* LISTEN 130244/java
tcp6 0 0 [::]:http [::]:* LISTEN 130244/java
tcp6 86 0 Yeggy-F8-FMSVA:39680 172.22.22.192:https CLOSE_WAIT 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54772 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54782 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54766 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54778 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54788 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54770 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54790 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54776 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54786 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54780 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54768 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54784 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:54774 Yeggy-F8-FMSVA:vrace ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java
tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java
Thanks
-----Original Message-----
From: Noelette Stout <no...@isu.edu>
Sent: Friday, May 21, 2021 8:28 AM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
ss has all the same options as netstat
On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) <aw...@ice-sa.com>
wrote:
> Soyrry to top-post, but it's getting cluttered down there..
>
> The next thing that you may want to do :
>
> > netstat -p -a -6 --tcp
>
> That is an alternative list of sockets, which also shows the "tcp state"
> of the sockets.
> To get only the ones of the tomcat JVM PID, filter with grep based of
> the last column.
> The type of thing you are looking for is the column which should show
> "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc..
>
> The options above :
> -p : show PID and program
> -a : show all sockets states
> -6 : only inet v6
> --tcp : only TCP sockets
>
> "netstat" may not be on your system by default, and you may need to
> install it.
> An alternative is "ss", but I don't know the options.
>
>
> On 21.05.2021 02:14, Yeggy Javadi wrote:
> > Hi,
> > Yes; that is what I get and as you can see among 8028 open files,
> > 7474
> are for TCPv6 sockets:
> > java 130244 root 7805u sock 0,9 0t0 12294251
> protocol: TCPv6
> >
> > # ps -ef | grep tomcat
> > root 130244 1 1 11:01 ? 00:06:20
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
> properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> > root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
> tomcat
> >
> > #lsof -p 130244 | wc -l
> > 8028
> >
> > #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> > 7474
> >
> > Thanks
> >
> > -----Original Message-----
> > From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> > Sent: Thursday, May 20, 2021 4:19 PM
> > To: users@tomcat.apache.org
> > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >
> > Hi.
> > According to the list below, you have 2 java (JVM) processes running
> > on
> your system.
> > One (PID = 130244) is the JVM which runs tomcat. This is visible
> > when
> you look at the whole command-line.
> >
> > The other (PID = 130516) runs ElasticSearch, which I believe is not
> relevant here.
> >
> > So you should run lsof with the "-p 130244" option, to show only the
> files opened by the tomcat JVM.
> > To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
> "apache-tomcat"
> > (or anything else unique in that line)
> >
> >
> > On 20.05.2021 21:00, Yeggy Javadi wrote:
> >> Hi Chris,
> >> Please indicate how to show lsof or netstat to *just the JVM* process.
> >>
> >> Below is the list of running processes on my server:
> >> UID PID PPID C STIME TTY TIME CMD
> >> root 1 0 0 May07 ? 00:00:14
> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
> >> root 2 0 0 May07 ? 00:00:00 [kthreadd]
> >> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
> >> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
> >> root 6 2 0 May07 ? 00:00:00
> [kworker/0:0H-kblockd]
> >> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
> >> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
> >> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
> >> root 11 2 0 May07 ? 00:00:00 [migration/0]
> >> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
> >> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
> >> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
> >> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
> >> root 16 2 0 May07 ? 00:00:00 [migration/1]
> >> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
> >> root 19 2 0 May07 ? 00:00:00
> [kworker/1:0H-kblockd]
> >> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
> >> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
> >> root 22 2 0 May07 ? 00:00:00 [migration/2]
> >> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
> >> root 25 2 0 May07 ? 00:00:00
> [kworker/2:0H-kblockd]
> >> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
> >> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
> >> root 28 2 0 May07 ? 00:00:00 [migration/3]
> >> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
> >> root 31 2 0 May07 ? 00:00:00
> [kworker/3:0H-kblockd]
> >> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
> >> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
> >> root 34 2 0 May07 ? 00:00:00 [migration/4]
> >> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
> >> root 37 2 0 May07 ? 00:00:00
> [kworker/4:0H-kblockd]
> >> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
> >> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
> >> root 40 2 0 May07 ? 00:00:00 [migration/5]
> >> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
> >> root 43 2 0 May07 ? 00:00:00
> [kworker/5:0H-kblockd]
> >> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
> >> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
> >> root 46 2 0 May07 ? 00:00:00 [migration/6]
> >> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
> >> root 49 2 0 May07 ? 00:00:00
> [kworker/6:0H-kblockd]
> >> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
> >> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
> >> root 52 2 0 May07 ? 00:00:00 [migration/7]
> >> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
> >> root 55 2 0 May07 ? 00:00:00
> [kworker/7:0H-kblockd]
> >> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
> >> root 58 2 0 May07 ? 00:00:00 [netns]
> >> root 59 2 0 May07 ? 00:00:00 [kauditd]
> >> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
> >> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
> >> root 64 2 0 May07 ? 00:00:00 [writeback]
> >> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
> >> root 66 2 0 May07 ? 00:00:00 [ksmd]
> >> root 67 2 0 May07 ? 00:00:02 [khugepaged]
> >> root 68 2 0 May07 ? 00:00:00 [crypto]
> >> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
> >> root 70 2 0 May07 ? 00:00:00 [kblockd]
> >> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
> >> root 72 2 0 May07 ? 00:00:00 [md]
> >> root 73 2 0 May07 ? 00:00:00 [edac-poller]
> >> root 74 2 0 May07 ? 00:00:00 [watchdogd]
> >> root 92 2 0 May07 ? 00:00:00 [kswapd0]
> >> root 188 2 0 May07 ? 00:00:00 [kthrotld]
> >> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
> >> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
> >> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
> >> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
> >> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
> >> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
> >> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
> >> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
> >> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
> >> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
> >> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
> >> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
> >> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
> >> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
> >> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
> >> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
> >> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
> >> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
> >> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
> >> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
> >> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
> >> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
> >> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
> >> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
> >> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
> >> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
> >> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
> >> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
> >> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
> >> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
> >> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
> >> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
> >> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
> >> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
> >> root 224 2 0 May07 ? 00:00:00 [kaluad]
> >> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
> >> root 227 2 0 May07 ? 00:00:00 [kstrp]
> >> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
> >> root 509 2 0 May07 ? 00:00:00 [ata_sff]
> >> root 510 2 0 May07 ? 00:00:00 [mpt/0]
> >> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
> >> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
> >> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
> >> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
> >> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
> >> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
> >> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
> >> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
> >> root 540 2 0 May07 ? 00:00:01
> [kworker/7:1H-kblockd]
> >> root 541 2 0 May07 ? 00:00:01
> [kworker/5:1H-kblockd]
> >> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
> >> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
> >> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 613 2 0 May07 ? 00:00:01
> [kworker/6:1H-kblockd]
> >> root 650 2 0 May07 ? 00:00:01
> [kworker/4:1H-kblockd]
> >> root 659 2 0 May07 ? 00:00:02
> [kworker/3:1H-kblockd]
> >> root 660 2 0 May07 ? 00:00:02
> [kworker/2:1H-kblockd]
> >> root 662 2 0 May07 ? 00:00:01
> [kworker/0:1H-kblockd]
> >> root 679 1 0 May07 ? 00:00:03
> /usr/lib/systemd/systemd-journald
> >> root 682 2 0 May07 ? 00:00:02
> [kworker/1:1H-kblockd]
> >> root 716 1 0 May07 ? 00:00:01
> /usr/lib/systemd/systemd-udevd
> >> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
> >> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
> >> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
> >> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
> >> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
> >> root 790 2 0 May07 ? 00:00:00 [rpciod]
> >> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
> >> root 792 2 0 May07 ? 00:00:00 [xprtiod]
> >> dbus 883 1 0 May07 ? 00:00:03
> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
> --systemd-activation --syslog-only
> >> root 885 1 0 May07 ? 00:02:42
> /usr/sbin/NetworkManager --no-daemon
> >> root 887 1 0 May07 ? 00:00:44
> /usr/sbin/irqbalance --foreground
> >> root 888 1 0 May07 ? 00:00:00
> /usr/bin/VGAuthService -s
> >> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
> >> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
> --logger=files
> >> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
> >> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
> --fill-watermark=0
> >> root 946 890 0 May07 ? 00:00:06
> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
> --logger=files
> >> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
> -d2
> >> root 962 890 0 May07 ? 00:00:12
> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
> >> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
> -D
> >> root 970 1 0 May07 ? 00:00:02
> /usr/lib/systemd/systemd-logind
> >> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
> >> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
> -n
> >> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
> >> root 6223 1 0 May07 ? 00:07:27
> /usr/local/vtl/bin/sanidmgr
> >> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
> >> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
> >> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
> >> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
> >> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
> >> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
> >> root 6463 2 0 May07 ? 00:00:12 [iocore]
> >> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
> >> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
> >> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
> >> root 6523 2 0 May07 ? 00:01:50 [trudpd]
> >> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
> >> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
> >> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
> >> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
> >> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
> >> root 6557 2 0 May07 ? 00:00:00 [ipstord]
> >> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
> >> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
> >> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
> >> root 6654 1 0 May07 ? 00:00:00
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6655 6654 0 May07 ? 00:05:18
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6656 6655 0 May07 ? 00:00:36
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6693 1 0 May07 ? 00:00:00 rdesirkey
> >> root 6699 1 0 May07 ? 00:00:12 ikmd
> >> root 6721 1 0 May07 ? 00:00:35
> /usr/local/vtl/bin/iscliproxy
> >> root 6747 1 0 May07 ? 00:00:40
> /usr/local/vtl/bin/ipstorlog
> >> root 6780 1 0 May07 ? 00:04:50
> /usr/local/vtl/bin/snmpd
> >> root 7722 1 0 May07 ? 00:00:00
> /usr/local/vtl/bin/ipstorsm
> >> root 7723 7722 0 May07 ? 01:08:12
> /usr/local/vtl/bin/ipstorsm
> >> root 7901 1 0 May07 ? 00:00:00
> /usr/lib/systemd/systemd --user
> >> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
> >> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
> -- \u --noclear tty1 linux
> >> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
> >> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
> >> root 14202 14201 0 May08 pts/0 00:00:00 -bash
> >> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-
> group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sh
> a2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group
> -exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-
> sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha
> 1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
> -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-
> sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
> >> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
> >> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
> >> root 128108 2 0 08:40 ? 00:00:00
> [kworker/4:1-events_power_efficient]
> >> root 128360 2 0 09:10 ? 00:00:00
> [kworker/4:2-cgroup_destroy]
> >> root 128430 2 0 09:30 ? 00:00:00
> [kworker/6:0-mm_percpu_wq]
> >> root 128738 2 0 11:00 ? 00:00:00
> [kworker/3:3-mm_percpu_wq]
> >> root 128761 2 0 11:00 ? 00:00:00
> [kworker/2:1-mm_percpu_wq]
> >> root 130244 1 1 11:01 ? 00:03:50
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
> properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> >> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
> -Des.path.home=/usr/share/elasticsearch -cp
> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elast
> icsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
> -Des.default.path.home=/usr/share/elasticsearch
> -Des.default.path.logs=/var/log/elasticsearch
> -Des.default.path.data=/var/lib/elasticsearch
> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
> -Des.default.path.conf=/etc/elasticsearch
> org.elasticsearch.bootstrap.Elasticsearch
> >> postgres 130548 1 0 11:01 ? 00:00:00
> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
> >> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
> process
> >> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
> checkpointer process
> >> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
> process
> >> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
> writer process
> >> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
> autovacuum launcher process
> >> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
> collector process
> >> root 130570 1 0 11:01 ? 00:00:01
> /usr/local/freestor/bin/fmsd
> >> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
> freestor 127.0.0.1(42660) idle
> >> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
> freestor 127.0.0.1(42662) idle
> >> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
> freestor 127.0.0.1(42664) idle
> >> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
> >> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
> >> root 131347 2 0 13:49 ? 00:00:00
> [kworker/5:1-mm_percpu_wq]
> >> root 131364 2 0 13:56 ? 00:00:00
> [kworker/u16:2-events_unbound]
> >> root 131401 2 0 14:05 ? 00:00:00
> [kworker/u16:1-events_unbound]
> >> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
> >> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
> >> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
> >> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
> >> root 131515 2 0 14:49 ? 00:00:00
> [kworker/1:0-events_freezable]
> >> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
> >> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
> >> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
> >> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
> >> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
> >>
> >> Thanks
> >> -----Original Message-----
> >> From: Christopher Schultz <ch...@christopherschultz.net>
> >> Sent: Thursday, May 20, 2021 12:49 PM
> >> To: users@tomcat.apache.org
> >> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >>
> >> Yeggy,
> >>
> >> On 5/19/21 19:37, Yeggy Javadi wrote:
> >>> Hi Chris,
> >>> Thanks for your email. Below are replies to your questions:
> >>>
> >>> 1. Did you upgrade anything recently (like Java VM)?
> >>> [YJ] To support Linux 8, only Postgres was upgraded from version
> >>> 9.3
> to 9.6.
> >>>
> >>> 2. What is error you are seeing? A full stack trace would be helpful.
> >>> [YJ] The application error can occur in any place with the (Too
> >>> many
> open files) error when the limit of open files for tomcat has reached
> (262144 in my environment), for example, I can get a connectivity
> error when pulling info from a server as below :
> >>>
> >>> [InventoryPullerTask - 10.1.6.25] ERROR
> >>> FSSDataCollectorService - Error : The XML configuration file
> >>> failed to be retrieved for server 10.1.6.25. Check server
> >>> connectivity.Error retrieving IPStorConfg for
> >>> server=10.1.6.25 error=java.io.FileNotFoundException:
> >>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
> >>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
> >>>
> >>> 3. What is your <Connector> configuration?
> >>> [YJ] It is as below in server.xml:
> >>>
> >>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
> >>> maxThreads="150"
> >>> minSpareThreads="25" maxSpareThreads="75"
> >>> enableLookups="false" acceptCount="100"
> >>> connectionTimeout="20000" disableUploadTimeout="true"
> >>> compression="on"
> >>> compressionMinSize="2048"
> >>> noCompressionUserAgents="gozilla, traviata"
> >>>
> compressableMimeType="text/html,text/xml,application/json"
> >>> redirectPort="443" /> ...........
> >>> <Connector address="0.0.0.0" port="443"
> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> >>> maxThreads="150" scheme="https" secure="true"
> >>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
> >>>
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
> >>> keystoreFile="conf/freestor"
> keystorePass="freestor"/>
> >>>
> >>> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
> >>> redirectPort="443" secretRequired="false" />
> >>
> >> You have lots of settings on the HTTP connector but not the HTTPS one.
> >> Is that intentional? Which one are you using? Or both?
> >>
> >> Are you using AJP? If not, disable the connetor entirely.
> >> Definitely
> never enable it on 0.0.0.0 unless you really know what you are doing.
> >>
> >>> 4. Are you counting all the open files for a single process or all
> >>> of
> the sub-processes which represent the threads of the main process?
> >>> Different kernel versions count things differently.
> >>> [YJ] I am just getting the process ID of tomcat and count open
> >>> files for that process
> >>
> >> Ok.
> >>
> >>> 5. Running lsof, netstat, etc. can you see if some large number of
> those sockets are bound to any specific port (listen or connect)?
> >>> [YJ] Here is the netstat output:
> >>> Active Internet connections (w/o servers)
> >>> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> >>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
> ESTABLISHED
> >>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
> ESTABLISHED
> >>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
> ESTABLISHED
> >>
> >> That's not helpful. I want to see what the JVM process is holding,
> >> not
> what the whole OS is handling.
> >>
> >> My guess is that lsof is going to be a better bet for inspecting
> >> the
> process. But if you get "too many open files" being thrown inside the
> JVM and you don't see a 1/4 million lines of "lsof" output on your
> process, then something else is going on. But remember to limit your
> lsof to *just the JVM* process.
> >>
> >> -chris
> >>
> >>> -----Original Message-----
> >>> From: Christopher Schultz <ch...@christopherschultz.net>
> >>> Sent: Monday, May 17, 2021 5:18 PM
> >>> To: users@tomcat.apache.org
> >>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >>>
> >>> Yeggy,
> >>>
> >>> On 5/17/21 15:17, Yeggy Javadi wrote:
> >>>> Hi,
> >>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> >>>> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
> >>>>
> >>>> I was running the same application on Linux 7.6 with the same
> >>>> apache-tomcat version without seeing this issue. I saw that
> >>>> RedHat has removed apache-tomcat fom their Linux 8 distribution.
> >>>> Please help with throubleshooting and indicating if there is any
> >>>> changes in Linux
> >>>> 8 that can explain this
> >>>>
> >>>> Configuration:
> >>>> apache-tomcat-8.5.59
> >>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based
> >>>> application using Postgres and elastic search databases
> >>>>
> >>>>
> >>>> # ps -ef | grep tomcat
> >>>> root 51798 1 1 11:16 ? 00:06:01
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.
> properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi
> n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> >>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
> tomcat
> >>>>
> >>>>
> >>>> # cat /proc/51798/limits
> >>>> Limit Soft Limit Hard Limit
> Units
> >>>> Max cpu time unlimited unlimited
> seconds
> >>>> Max file size unlimited unlimited
> bytes
> >>>> Max data size unlimited unlimited
> bytes
> >>>> Max stack size 8388608 unlimited
> bytes
> >>>> Max core file size unlimited unlimited
> bytes
> >>>> Max resident set unlimited unlimited
> bytes
> >>>> Max processes 256518 256518
> processes
> >>>> Max open files 262144 262144
> files
> >>>> Max locked memory 65536 65536
> bytes
> >>>> Max address space unlimited unlimited
> bytes
> >>>> Max file locks unlimited unlimited
> locks
> >>>> Max pending signals 256518 256518
> signals
> >>>> Max msgqueue size 819200 819200
> bytes
> >>>> Max nice priority 0 0
> >>>> Max realtime priority 0 0
> >>>> Max realtime timeout unlimited unlimited us
> >>>>
> >>>>
> >>>> Too may open files:
> >>>> # lsof -p 51798 | wc -l
> >>>> 6679
> >>>>
> >>>>
> >>>> Most of open files are TCP sockets:
> >>>> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
> >>>>
> >>>>
> >>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> >>>> 6413
> >>>
> >>> Some potentially dumb questions:
> >>>
> >>> 1. Did you upgrade anything recently (like Java VM)?
> >>>
> >>> 2. What is error you are seeing? A full stack trace would be helpful.
> >>>
> >>> 3. What is your <Connector> configuration?
> >>>
> >>> 4. Are you counting all the open files for a single process or all
> >>> of
> the sub-processes which represent the threads of the main process?
> >>> Different kernel versions count things differently.
> >>>
> >>> 5. Running lsof, netstat, etc. can you see if some large number of
> those sockets are bound to any specific port (listen or connect)?
> >>>
> >>> -chris
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Noelette Stout
ITS Enterprise Applications - Application Administrator - Senior
Business Administration Building, Rm 109L
921 South 8th Ave 8037
Idaho State University
Pocatello ID 83209
E-mail: stounoel "at" isu "dot" edu
Desk: 208-282-2554
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Noelette Stout <no...@isu.edu>.
ss has all the same options as netstat
On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) <aw...@ice-sa.com>
wrote:
> Soyrry to top-post, but it's getting cluttered down there..
>
> The next thing that you may want to do :
>
> > netstat -p -a -6 --tcp
>
> That is an alternative list of sockets, which also shows the "tcp state"
> of the sockets.
> To get only the ones of the tomcat JVM PID, filter with grep based of the
> last column.
> The type of thing you are looking for is the column which should show
> "LISTEN", or
> "ESTABLISHED" or "CLOSE_WAIT" etc..
>
> The options above :
> -p : show PID and program
> -a : show all sockets states
> -6 : only inet v6
> --tcp : only TCP sockets
>
> "netstat" may not be on your system by default, and you may need to
> install it.
> An alternative is "ss", but I don't know the options.
>
>
> On 21.05.2021 02:14, Yeggy Javadi wrote:
> > Hi,
> > Yes; that is what I get and as you can see among 8028 open files, 7474
> are for TCPv6 sockets:
> > java 130244 root 7805u sock 0,9 0t0 12294251
> protocol: TCPv6
> >
> > # ps -ef | grep tomcat
> > root 130244 1 1 11:01 ? 00:06:20
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> > root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto
> tomcat
> >
> > #lsof -p 130244 | wc -l
> > 8028
> >
> > #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> > 7474
> >
> > Thanks
> >
> > -----Original Message-----
> > From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> > Sent: Thursday, May 20, 2021 4:19 PM
> > To: users@tomcat.apache.org
> > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >
> > Hi.
> > According to the list below, you have 2 java (JVM) processes running on
> your system.
> > One (PID = 130244) is the JVM which runs tomcat. This is visible when
> you look at the whole command-line.
> >
> > The other (PID = 130516) runs ElasticSearch, which I believe is not
> relevant here.
> >
> > So you should run lsof with the "-p 130244" option, to show only the
> files opened by the tomcat JVM.
> > To show the current Tomcat JVM PID, do e.g. : ps -ef | grep
> "apache-tomcat"
> > (or anything else unique in that line)
> >
> >
> > On 20.05.2021 21:00, Yeggy Javadi wrote:
> >> Hi Chris,
> >> Please indicate how to show lsof or netstat to *just the JVM* process.
> >>
> >> Below is the list of running processes on my server:
> >> UID PID PPID C STIME TTY TIME CMD
> >> root 1 0 0 May07 ? 00:00:14
> /usr/lib/systemd/systemd --switched-root --system --deserialize 17
> >> root 2 0 0 May07 ? 00:00:00 [kthreadd]
> >> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
> >> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
> >> root 6 2 0 May07 ? 00:00:00
> [kworker/0:0H-kblockd]
> >> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
> >> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
> >> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
> >> root 11 2 0 May07 ? 00:00:00 [migration/0]
> >> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
> >> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
> >> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
> >> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
> >> root 16 2 0 May07 ? 00:00:00 [migration/1]
> >> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
> >> root 19 2 0 May07 ? 00:00:00
> [kworker/1:0H-kblockd]
> >> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
> >> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
> >> root 22 2 0 May07 ? 00:00:00 [migration/2]
> >> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
> >> root 25 2 0 May07 ? 00:00:00
> [kworker/2:0H-kblockd]
> >> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
> >> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
> >> root 28 2 0 May07 ? 00:00:00 [migration/3]
> >> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
> >> root 31 2 0 May07 ? 00:00:00
> [kworker/3:0H-kblockd]
> >> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
> >> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
> >> root 34 2 0 May07 ? 00:00:00 [migration/4]
> >> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
> >> root 37 2 0 May07 ? 00:00:00
> [kworker/4:0H-kblockd]
> >> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
> >> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
> >> root 40 2 0 May07 ? 00:00:00 [migration/5]
> >> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
> >> root 43 2 0 May07 ? 00:00:00
> [kworker/5:0H-kblockd]
> >> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
> >> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
> >> root 46 2 0 May07 ? 00:00:00 [migration/6]
> >> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
> >> root 49 2 0 May07 ? 00:00:00
> [kworker/6:0H-kblockd]
> >> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
> >> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
> >> root 52 2 0 May07 ? 00:00:00 [migration/7]
> >> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
> >> root 55 2 0 May07 ? 00:00:00
> [kworker/7:0H-kblockd]
> >> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
> >> root 58 2 0 May07 ? 00:00:00 [netns]
> >> root 59 2 0 May07 ? 00:00:00 [kauditd]
> >> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
> >> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
> >> root 64 2 0 May07 ? 00:00:00 [writeback]
> >> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
> >> root 66 2 0 May07 ? 00:00:00 [ksmd]
> >> root 67 2 0 May07 ? 00:00:02 [khugepaged]
> >> root 68 2 0 May07 ? 00:00:00 [crypto]
> >> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
> >> root 70 2 0 May07 ? 00:00:00 [kblockd]
> >> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
> >> root 72 2 0 May07 ? 00:00:00 [md]
> >> root 73 2 0 May07 ? 00:00:00 [edac-poller]
> >> root 74 2 0 May07 ? 00:00:00 [watchdogd]
> >> root 92 2 0 May07 ? 00:00:00 [kswapd0]
> >> root 188 2 0 May07 ? 00:00:00 [kthrotld]
> >> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
> >> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
> >> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
> >> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
> >> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
> >> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
> >> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
> >> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
> >> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
> >> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
> >> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
> >> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
> >> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
> >> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
> >> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
> >> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
> >> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
> >> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
> >> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
> >> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
> >> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
> >> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
> >> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
> >> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
> >> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
> >> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
> >> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
> >> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
> >> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
> >> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
> >> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
> >> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
> >> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
> >> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
> >> root 224 2 0 May07 ? 00:00:00 [kaluad]
> >> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
> >> root 227 2 0 May07 ? 00:00:00 [kstrp]
> >> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
> >> root 509 2 0 May07 ? 00:00:00 [ata_sff]
> >> root 510 2 0 May07 ? 00:00:00 [mpt/0]
> >> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
> >> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
> >> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
> >> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
> >> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
> >> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
> >> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
> >> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
> >> root 540 2 0 May07 ? 00:00:01
> [kworker/7:1H-kblockd]
> >> root 541 2 0 May07 ? 00:00:01
> [kworker/5:1H-kblockd]
> >> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
> >> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
> >> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 613 2 0 May07 ? 00:00:01
> [kworker/6:1H-kblockd]
> >> root 650 2 0 May07 ? 00:00:01
> [kworker/4:1H-kblockd]
> >> root 659 2 0 May07 ? 00:00:02
> [kworker/3:1H-kblockd]
> >> root 660 2 0 May07 ? 00:00:02
> [kworker/2:1H-kblockd]
> >> root 662 2 0 May07 ? 00:00:01
> [kworker/0:1H-kblockd]
> >> root 679 1 0 May07 ? 00:00:03
> /usr/lib/systemd/systemd-journald
> >> root 682 2 0 May07 ? 00:00:02
> [kworker/1:1H-kblockd]
> >> root 716 1 0 May07 ? 00:00:01
> /usr/lib/systemd/systemd-udevd
> >> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
> >> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
> >> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
> >> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
> >> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> >> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
> >> root 790 2 0 May07 ? 00:00:00 [rpciod]
> >> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
> >> root 792 2 0 May07 ? 00:00:00 [xprtiod]
> >> dbus 883 1 0 May07 ? 00:00:03
> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
> --systemd-activation --syslog-only
> >> root 885 1 0 May07 ? 00:02:42
> /usr/sbin/NetworkManager --no-daemon
> >> root 887 1 0 May07 ? 00:00:44
> /usr/sbin/irqbalance --foreground
> >> root 888 1 0 May07 ? 00:00:00
> /usr/bin/VGAuthService -s
> >> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
> >> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i
> --logger=files
> >> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
> >> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f
> --fill-watermark=0
> >> root 946 890 0 May07 ? 00:00:06
> /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0
> --logger=files
> >> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f
> -d2
> >> root 962 890 0 May07 ? 00:00:12
> /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
> >> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy
> -D
> >> root 970 1 0 May07 ? 00:00:02
> /usr/lib/systemd/systemd-logind
> >> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
> >> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd
> -n
> >> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
> >> root 6223 1 0 May07 ? 00:07:27
> /usr/local/vtl/bin/sanidmgr
> >> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
> >> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
> >> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
> >> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
> >> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
> >> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
> >> root 6463 2 0 May07 ? 00:00:12 [iocore]
> >> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
> >> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
> >> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
> >> root 6523 2 0 May07 ? 00:01:50 [trudpd]
> >> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
> >> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
> >> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
> >> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
> >> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
> >> root 6557 2 0 May07 ? 00:00:00 [ipstord]
> >> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
> >> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
> >> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
> >> root 6654 1 0 May07 ? 00:00:00
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6655 6654 0 May07 ? 00:05:18
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6656 6655 0 May07 ? 00:00:36
> /usr/local/vtl/bin/ipstorcomm -d 0
> >> root 6693 1 0 May07 ? 00:00:00 rdesirkey
> >> root 6699 1 0 May07 ? 00:00:12 ikmd
> >> root 6721 1 0 May07 ? 00:00:35
> /usr/local/vtl/bin/iscliproxy
> >> root 6747 1 0 May07 ? 00:00:40
> /usr/local/vtl/bin/ipstorlog
> >> root 6780 1 0 May07 ? 00:04:50
> /usr/local/vtl/bin/snmpd
> >> root 7722 1 0 May07 ? 00:00:00
> /usr/local/vtl/bin/ipstorsm
> >> root 7723 7722 0 May07 ? 01:08:12
> /usr/local/vtl/bin/ipstorsm
> >> root 7901 1 0 May07 ? 00:00:00
> /usr/lib/systemd/systemd --user
> >> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
> >> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p
> -- \u --noclear tty1 linux
> >> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
> >> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
> >> root 14202 14201 0 May08 pts/0 00:00:00 -bash
> >> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D
> -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
> ,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
> -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,
> umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com
> ,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
> -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-
> -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
> -oHostKeyAlgorithms=ecdsa-sha2-nistp256,
> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
> -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,
> ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,
> ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,
> ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,
> ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,
> rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,
> rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com
> -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
> >> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
> >> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
> >> root 128108 2 0 08:40 ? 00:00:00
> [kworker/4:1-events_power_efficient]
> >> root 128360 2 0 09:10 ? 00:00:00
> [kworker/4:2-cgroup_destroy]
> >> root 128430 2 0 09:30 ? 00:00:00
> [kworker/6:0-mm_percpu_wq]
> >> root 128738 2 0 11:00 ? 00:00:00
> [kworker/3:3-mm_percpu_wq]
> >> root 128761 2 0 11:00 ? 00:00:00
> [kworker/2:1-mm_percpu_wq]
> >> root 130244 1 1 11:01 ? 00:03:50
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> >> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g
> -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
> -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes
> -Des.path.home=/usr/share/elasticsearch -cp
> :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
> -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid
> -Des.default.path.home=/usr/share/elasticsearch
> -Des.default.path.logs=/var/log/elasticsearch
> -Des.default.path.data=/var/lib/elasticsearch
> -Des.default.config=/etc/elasticsearch/elasticsearch.yml
> -Des.default.path.conf=/etc/elasticsearch
> org.elasticsearch.bootstrap.Elasticsearch
> >> postgres 130548 1 0 11:01 ? 00:00:00
> /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
> >> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger
> process
> >> postgres 130563 130548 0 11:01 ? 00:00:00 postgres:
> checkpointer process
> >> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer
> process
> >> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal
> writer process
> >> postgres 130566 130548 0 11:01 ? 00:00:00 postgres:
> autovacuum launcher process
> >> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats
> collector process
> >> root 130570 1 0 11:01 ? 00:00:01
> /usr/local/freestor/bin/fmsd
> >> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres
> freestor 127.0.0.1(42660) idle
> >> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres
> freestor 127.0.0.1(42662) idle
> >> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres
> freestor 127.0.0.1(42664) idle
> >> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
> >> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
> >> root 131347 2 0 13:49 ? 00:00:00
> [kworker/5:1-mm_percpu_wq]
> >> root 131364 2 0 13:56 ? 00:00:00
> [kworker/u16:2-events_unbound]
> >> root 131401 2 0 14:05 ? 00:00:00
> [kworker/u16:1-events_unbound]
> >> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
> >> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
> >> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
> >> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
> >> root 131515 2 0 14:49 ? 00:00:00
> [kworker/1:0-events_freezable]
> >> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
> >> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
> >> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
> >> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
> >> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
> >>
> >> Thanks
> >> -----Original Message-----
> >> From: Christopher Schultz <ch...@christopherschultz.net>
> >> Sent: Thursday, May 20, 2021 12:49 PM
> >> To: users@tomcat.apache.org
> >> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >>
> >> Yeggy,
> >>
> >> On 5/19/21 19:37, Yeggy Javadi wrote:
> >>> Hi Chris,
> >>> Thanks for your email. Below are replies to your questions:
> >>>
> >>> 1. Did you upgrade anything recently (like Java VM)?
> >>> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3
> to 9.6.
> >>>
> >>> 2. What is error you are seeing? A full stack trace would be helpful.
> >>> [YJ] The application error can occur in any place with the (Too many
> open files) error when the limit of open files for tomcat has reached
> (262144 in my environment), for example, I can get a connectivity error
> when pulling info from a server as below :
> >>>
> >>> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
> >>> Error : The XML configuration file failed to be retrieved for server
> >>> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
> >>> server=10.1.6.25 error=java.io.FileNotFoundException:
> >>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
> >>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
> >>>
> >>> 3. What is your <Connector> configuration?
> >>> [YJ] It is as below in server.xml:
> >>>
> >>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
> >>> maxThreads="150"
> >>> minSpareThreads="25" maxSpareThreads="75"
> >>> enableLookups="false" acceptCount="100"
> >>> connectionTimeout="20000" disableUploadTimeout="true"
> >>> compression="on"
> >>> compressionMinSize="2048"
> >>> noCompressionUserAgents="gozilla, traviata"
> >>>
> compressableMimeType="text/html,text/xml,application/json"
> >>> redirectPort="443" />
> >>> ...........
> >>> <Connector address="0.0.0.0" port="443"
> protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> >>> maxThreads="150" scheme="https" secure="true"
> >>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
> >>>
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
> >>> keystoreFile="conf/freestor"
> keystorePass="freestor"/>
> >>>
> >>> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
> >>> redirectPort="443" secretRequired="false" />
> >>
> >> You have lots of settings on the HTTP connector but not the HTTPS one.
> >> Is that intentional? Which one are you using? Or both?
> >>
> >> Are you using AJP? If not, disable the connetor entirely. Definitely
> never enable it on 0.0.0.0 unless you really know what you are doing.
> >>
> >>> 4. Are you counting all the open files for a single process or all of
> the sub-processes which represent the threads of the main process?
> >>> Different kernel versions count things differently.
> >>> [YJ] I am just getting the process ID of tomcat and count open files
> >>> for that process
> >>
> >> Ok.
> >>
> >>> 5. Running lsof, netstat, etc. can you see if some large number of
> those sockets are bound to any specific port (listen or connect)?
> >>> [YJ] Here is the netstat output:
> >>> Active Internet connections (w/o servers)
> >>> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> >>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236
> ESTABLISHED
> >>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378
> ESTABLISHED
> >>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846
> ESTABLISHED
> >>
> >> That's not helpful. I want to see what the JVM process is holding, not
> what the whole OS is handling.
> >>
> >> My guess is that lsof is going to be a better bet for inspecting the
> process. But if you get "too many open files" being thrown inside the JVM
> and you don't see a 1/4 million lines of "lsof" output on your process,
> then something else is going on. But remember to limit your lsof to *just
> the JVM* process.
> >>
> >> -chris
> >>
> >>> -----Original Message-----
> >>> From: Christopher Schultz <ch...@christopherschultz.net>
> >>> Sent: Monday, May 17, 2021 5:18 PM
> >>> To: users@tomcat.apache.org
> >>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
> >>>
> >>> Yeggy,
> >>>
> >>> On 5/17/21 15:17, Yeggy Javadi wrote:
> >>>> Hi,
> >>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> >>>> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
> >>>>
> >>>> I was running the same application on Linux 7.6 with the same
> >>>> apache-tomcat version without seeing this issue. I saw that RedHat
> >>>> has removed apache-tomcat fom their Linux 8 distribution. Please help
> >>>> with throubleshooting and indicating if there is any changes in Linux
> >>>> 8 that can explain this
> >>>>
> >>>> Configuration:
> >>>> apache-tomcat-8.5.59
> >>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
> >>>> using Postgres and elastic search databases
> >>>>
> >>>>
> >>>> # ps -ef | grep tomcat
> >>>> root 51798 1 1 11:16 ? 00:06:01
> /usr/local/jre/bin/java
> -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64
> -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
> -Dignore.endorsed.dirs= -classpath
> /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/local/apache-tomcat
> -Dcatalina.home=/usr/local/apache-tomcat
> -Djava.io.tmpdir=/usr/local/apache-tomcat/temp
> org.apache.catalina.startup.Bootstrap start
> >>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto
> tomcat
> >>>>
> >>>>
> >>>> # cat /proc/51798/limits
> >>>> Limit Soft Limit Hard Limit
> Units
> >>>> Max cpu time unlimited unlimited
> seconds
> >>>> Max file size unlimited unlimited
> bytes
> >>>> Max data size unlimited unlimited
> bytes
> >>>> Max stack size 8388608 unlimited
> bytes
> >>>> Max core file size unlimited unlimited
> bytes
> >>>> Max resident set unlimited unlimited
> bytes
> >>>> Max processes 256518 256518
> processes
> >>>> Max open files 262144 262144
> files
> >>>> Max locked memory 65536 65536
> bytes
> >>>> Max address space unlimited unlimited
> bytes
> >>>> Max file locks unlimited unlimited
> locks
> >>>> Max pending signals 256518 256518
> signals
> >>>> Max msgqueue size 819200 819200
> bytes
> >>>> Max nice priority 0 0
> >>>> Max realtime priority 0 0
> >>>> Max realtime timeout unlimited unlimited us
> >>>>
> >>>>
> >>>> Too may open files:
> >>>> # lsof -p 51798 | wc -l
> >>>> 6679
> >>>>
> >>>>
> >>>> Most of open files are TCP sockets:
> >>>> java 51798 root 6601u sock 0,9 0t0 3959342
> protocol: TCPv6
> >>>>
> >>>>
> >>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> >>>> 6413
> >>>
> >>> Some potentially dumb questions:
> >>>
> >>> 1. Did you upgrade anything recently (like Java VM)?
> >>>
> >>> 2. What is error you are seeing? A full stack trace would be helpful.
> >>>
> >>> 3. What is your <Connector> configuration?
> >>>
> >>> 4. Are you counting all the open files for a single process or all of
> the sub-processes which represent the threads of the main process?
> >>> Different kernel versions count things differently.
> >>>
> >>> 5. Running lsof, netstat, etc. can you see if some large number of
> those sockets are bound to any specific port (listen or connect)?
> >>>
> >>> -chris
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Noelette Stout
ITS Enterprise Applications - Application Administrator - Senior
Business Administration Building, Rm 109L
921 South 8th Ave 8037
Idaho State University
Pocatello ID 83209
E-mail: stounoel "at" isu "dot" edu
Desk: 208-282-2554
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Soyrry to top-post, but it's getting cluttered down there..
The next thing that you may want to do :
> netstat -p -a -6 --tcp
That is an alternative list of sockets, which also shows the "tcp state" of the sockets.
To get only the ones of the tomcat JVM PID, filter with grep based of the last column.
The type of thing you are looking for is the column which should show "LISTEN", or
"ESTABLISHED" or "CLOSE_WAIT" etc..
The options above :
-p : show PID and program
-a : show all sockets states
-6 : only inet v6
--tcp : only TCP sockets
"netstat" may not be on your system by default, and you may need to install it.
An alternative is "ss", but I don't know the options.
On 21.05.2021 02:14, Yeggy Javadi wrote:
> Hi,
> Yes; that is what I get and as you can see among 8028 open files, 7474 are for TCPv6 sockets:
> java 130244 root 7805u sock 0,9 0t0 12294251 protocol: TCPv6
>
> # ps -ef | grep tomcat
> root 130244 1 1 11:01 ? 00:06:20 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto tomcat
>
> #lsof -p 130244 | wc -l
> 8028
>
> #lsof -p 130244 | grep "protocol: TCPv6"| wc -l
> 7474
>
> Thanks
>
> -----Original Message-----
> From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
> Sent: Thursday, May 20, 2021 4:19 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Hi.
> According to the list below, you have 2 java (JVM) processes running on your system.
> One (PID = 130244) is the JVM which runs tomcat. This is visible when you look at the whole command-line.
>
> The other (PID = 130516) runs ElasticSearch, which I believe is not relevant here.
>
> So you should run lsof with the "-p 130244" option, to show only the files opened by the tomcat JVM.
> To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat"
> (or anything else unique in that line)
>
>
> On 20.05.2021 21:00, Yeggy Javadi wrote:
>> Hi Chris,
>> Please indicate how to show lsof or netstat to *just the JVM* process.
>>
>> Below is the list of running processes on my server:
>> UID PID PPID C STIME TTY TIME CMD
>> root 1 0 0 May07 ? 00:00:14 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
>> root 2 0 0 May07 ? 00:00:00 [kthreadd]
>> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
>> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
>> root 6 2 0 May07 ? 00:00:00 [kworker/0:0H-kblockd]
>> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
>> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
>> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
>> root 11 2 0 May07 ? 00:00:00 [migration/0]
>> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
>> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
>> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
>> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
>> root 16 2 0 May07 ? 00:00:00 [migration/1]
>> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
>> root 19 2 0 May07 ? 00:00:00 [kworker/1:0H-kblockd]
>> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
>> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
>> root 22 2 0 May07 ? 00:00:00 [migration/2]
>> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
>> root 25 2 0 May07 ? 00:00:00 [kworker/2:0H-kblockd]
>> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
>> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
>> root 28 2 0 May07 ? 00:00:00 [migration/3]
>> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
>> root 31 2 0 May07 ? 00:00:00 [kworker/3:0H-kblockd]
>> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
>> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
>> root 34 2 0 May07 ? 00:00:00 [migration/4]
>> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
>> root 37 2 0 May07 ? 00:00:00 [kworker/4:0H-kblockd]
>> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
>> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
>> root 40 2 0 May07 ? 00:00:00 [migration/5]
>> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
>> root 43 2 0 May07 ? 00:00:00 [kworker/5:0H-kblockd]
>> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
>> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
>> root 46 2 0 May07 ? 00:00:00 [migration/6]
>> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
>> root 49 2 0 May07 ? 00:00:00 [kworker/6:0H-kblockd]
>> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
>> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
>> root 52 2 0 May07 ? 00:00:00 [migration/7]
>> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
>> root 55 2 0 May07 ? 00:00:00 [kworker/7:0H-kblockd]
>> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
>> root 58 2 0 May07 ? 00:00:00 [netns]
>> root 59 2 0 May07 ? 00:00:00 [kauditd]
>> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
>> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
>> root 64 2 0 May07 ? 00:00:00 [writeback]
>> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
>> root 66 2 0 May07 ? 00:00:00 [ksmd]
>> root 67 2 0 May07 ? 00:00:02 [khugepaged]
>> root 68 2 0 May07 ? 00:00:00 [crypto]
>> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
>> root 70 2 0 May07 ? 00:00:00 [kblockd]
>> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
>> root 72 2 0 May07 ? 00:00:00 [md]
>> root 73 2 0 May07 ? 00:00:00 [edac-poller]
>> root 74 2 0 May07 ? 00:00:00 [watchdogd]
>> root 92 2 0 May07 ? 00:00:00 [kswapd0]
>> root 188 2 0 May07 ? 00:00:00 [kthrotld]
>> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
>> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
>> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
>> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
>> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
>> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
>> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
>> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
>> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
>> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
>> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
>> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
>> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
>> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
>> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
>> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
>> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
>> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
>> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
>> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
>> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
>> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
>> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
>> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
>> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
>> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
>> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
>> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
>> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
>> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
>> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
>> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
>> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
>> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
>> root 224 2 0 May07 ? 00:00:00 [kaluad]
>> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
>> root 227 2 0 May07 ? 00:00:00 [kstrp]
>> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
>> root 509 2 0 May07 ? 00:00:00 [ata_sff]
>> root 510 2 0 May07 ? 00:00:00 [mpt/0]
>> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
>> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
>> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
>> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
>> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
>> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
>> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
>> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
>> root 540 2 0 May07 ? 00:00:01 [kworker/7:1H-kblockd]
>> root 541 2 0 May07 ? 00:00:01 [kworker/5:1H-kblockd]
>> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
>> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
>> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 613 2 0 May07 ? 00:00:01 [kworker/6:1H-kblockd]
>> root 650 2 0 May07 ? 00:00:01 [kworker/4:1H-kblockd]
>> root 659 2 0 May07 ? 00:00:02 [kworker/3:1H-kblockd]
>> root 660 2 0 May07 ? 00:00:02 [kworker/2:1H-kblockd]
>> root 662 2 0 May07 ? 00:00:01 [kworker/0:1H-kblockd]
>> root 679 1 0 May07 ? 00:00:03 /usr/lib/systemd/systemd-journald
>> root 682 2 0 May07 ? 00:00:02 [kworker/1:1H-kblockd]
>> root 716 1 0 May07 ? 00:00:01 /usr/lib/systemd/systemd-udevd
>> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
>> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
>> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
>> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
>> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
>> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
>> root 790 2 0 May07 ? 00:00:00 [rpciod]
>> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
>> root 792 2 0 May07 ? 00:00:00 [xprtiod]
>> dbus 883 1 0 May07 ? 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
>> root 885 1 0 May07 ? 00:02:42 /usr/sbin/NetworkManager --no-daemon
>> root 887 1 0 May07 ? 00:00:44 /usr/sbin/irqbalance --foreground
>> root 888 1 0 May07 ? 00:00:00 /usr/bin/VGAuthService -s
>> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
>> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i --logger=files
>> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
>> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f --fill-watermark=0
>> root 946 890 0 May07 ? 00:00:06 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
>> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f -d2
>> root 962 890 0 May07 ? 00:00:12 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy -D
>> root 970 1 0 May07 ? 00:00:02 /usr/lib/systemd/systemd-logind
>> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
>> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd -n
>> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
>> root 6223 1 0 May07 ? 00:07:27 /usr/local/vtl/bin/sanidmgr
>> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
>> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
>> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
>> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
>> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
>> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
>> root 6463 2 0 May07 ? 00:00:12 [iocore]
>> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
>> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
>> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
>> root 6523 2 0 May07 ? 00:01:50 [trudpd]
>> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
>> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
>> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
>> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
>> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
>> root 6557 2 0 May07 ? 00:00:00 [ipstord]
>> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
>> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
>> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
>> root 6654 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorcomm -d 0
>> root 6655 6654 0 May07 ? 00:05:18 /usr/local/vtl/bin/ipstorcomm -d 0
>> root 6656 6655 0 May07 ? 00:00:36 /usr/local/vtl/bin/ipstorcomm -d 0
>> root 6693 1 0 May07 ? 00:00:00 rdesirkey
>> root 6699 1 0 May07 ? 00:00:12 ikmd
>> root 6721 1 0 May07 ? 00:00:35 /usr/local/vtl/bin/iscliproxy
>> root 6747 1 0 May07 ? 00:00:40 /usr/local/vtl/bin/ipstorlog
>> root 6780 1 0 May07 ? 00:04:50 /usr/local/vtl/bin/snmpd
>> root 7722 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorsm
>> root 7723 7722 0 May07 ? 01:08:12 /usr/local/vtl/bin/ipstorsm
>> root 7901 1 0 May07 ? 00:00:00 /usr/lib/systemd/systemd --user
>> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
>> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
>> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
>> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
>> root 14202 14201 0 May08 pts/0 00:00:00 -bash
>> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
>> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
>> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
>> root 128108 2 0 08:40 ? 00:00:00 [kworker/4:1-events_power_efficient]
>> root 128360 2 0 09:10 ? 00:00:00 [kworker/4:2-cgroup_destroy]
>> root 128430 2 0 09:30 ? 00:00:00 [kworker/6:0-mm_percpu_wq]
>> root 128738 2 0 11:00 ? 00:00:00 [kworker/3:3-mm_percpu_wq]
>> root 128761 2 0 11:00 ? 00:00:00 [kworker/2:1-mm_percpu_wq]
>> root 130244 1 1 11:01 ? 00:03:50 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
>> postgres 130548 1 0 11:01 ? 00:00:00 /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
>> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger process
>> postgres 130563 130548 0 11:01 ? 00:00:00 postgres: checkpointer process
>> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer process
>> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal writer process
>> postgres 130566 130548 0 11:01 ? 00:00:00 postgres: autovacuum launcher process
>> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats collector process
>> root 130570 1 0 11:01 ? 00:00:01 /usr/local/freestor/bin/fmsd
>> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42660) idle
>> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres freestor 127.0.0.1(42662) idle
>> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42664) idle
>> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
>> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
>> root 131347 2 0 13:49 ? 00:00:00 [kworker/5:1-mm_percpu_wq]
>> root 131364 2 0 13:56 ? 00:00:00 [kworker/u16:2-events_unbound]
>> root 131401 2 0 14:05 ? 00:00:00 [kworker/u16:1-events_unbound]
>> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
>> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
>> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
>> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
>> root 131515 2 0 14:49 ? 00:00:00 [kworker/1:0-events_freezable]
>> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
>> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
>> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
>> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
>> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>>
>> Thanks
>> -----Original Message-----
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Thursday, May 20, 2021 12:49 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Yeggy,
>>
>> On 5/19/21 19:37, Yeggy Javadi wrote:
>>> Hi Chris,
>>> Thanks for your email. Below are replies to your questions:
>>>
>>> 1. Did you upgrade anything recently (like Java VM)?
>>> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
>>>
>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>> [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
>>>
>>> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
>>> Error : The XML configuration file failed to be retrieved for server
>>> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
>>> server=10.1.6.25 error=java.io.FileNotFoundException:
>>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>>
>>> 3. What is your <Connector> configuration?
>>> [YJ] It is as below in server.xml:
>>>
>>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>>> maxThreads="150"
>>> minSpareThreads="25" maxSpareThreads="75"
>>> enableLookups="false" acceptCount="100"
>>> connectionTimeout="20000" disableUploadTimeout="true"
>>> compression="on"
>>> compressionMinSize="2048"
>>> noCompressionUserAgents="gozilla, traviata"
>>> compressableMimeType="text/html,text/xml,application/json"
>>> redirectPort="443" />
>>> ...........
>>> <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>>> maxThreads="150" scheme="https" secure="true"
>>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>>> keystoreFile="conf/freestor" keystorePass="freestor"/>
>>>
>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>>> redirectPort="443" secretRequired="false" />
>>
>> You have lots of settings on the HTTP connector but not the HTTPS one.
>> Is that intentional? Which one are you using? Or both?
>>
>> Are you using AJP? If not, disable the connetor entirely. Definitely never enable it on 0.0.0.0 unless you really know what you are doing.
>>
>>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>>> Different kernel versions count things differently.
>>> [YJ] I am just getting the process ID of tomcat and count open files
>>> for that process
>>
>> Ok.
>>
>>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>>> [YJ] Here is the netstat output:
>>> Active Internet connections (w/o servers)
>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
>>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
>>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
>>
>> That's not helpful. I want to see what the JVM process is holding, not what the whole OS is handling.
>>
>> My guess is that lsof is going to be a better bet for inspecting the process. But if you get "too many open files" being thrown inside the JVM and you don't see a 1/4 million lines of "lsof" output on your process, then something else is going on. But remember to limit your lsof to *just the JVM* process.
>>
>> -chris
>>
>>> -----Original Message-----
>>> From: Christopher Schultz <ch...@christopherschultz.net>
>>> Sent: Monday, May 17, 2021 5:18 PM
>>> To: users@tomcat.apache.org
>>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>>
>>> Yeggy,
>>>
>>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>>> Hi,
>>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>>
>>>> I was running the same application on Linux 7.6 with the same
>>>> apache-tomcat version without seeing this issue. I saw that RedHat
>>>> has removed apache-tomcat fom their Linux 8 distribution. Please help
>>>> with throubleshooting and indicating if there is any changes in Linux
>>>> 8 that can explain this
>>>>
>>>> Configuration:
>>>> apache-tomcat-8.5.59
>>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
>>>> using Postgres and elastic search databases
>>>>
>>>>
>>>> # ps -ef | grep tomcat
>>>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>>>>
>>>>
>>>> # cat /proc/51798/limits
>>>> Limit Soft Limit Hard Limit Units
>>>> Max cpu time unlimited unlimited seconds
>>>> Max file size unlimited unlimited bytes
>>>> Max data size unlimited unlimited bytes
>>>> Max stack size 8388608 unlimited bytes
>>>> Max core file size unlimited unlimited bytes
>>>> Max resident set unlimited unlimited bytes
>>>> Max processes 256518 256518 processes
>>>> Max open files 262144 262144 files
>>>> Max locked memory 65536 65536 bytes
>>>> Max address space unlimited unlimited bytes
>>>> Max file locks unlimited unlimited locks
>>>> Max pending signals 256518 256518 signals
>>>> Max msgqueue size 819200 819200 bytes
>>>> Max nice priority 0 0
>>>> Max realtime priority 0 0
>>>> Max realtime timeout unlimited unlimited us
>>>>
>>>>
>>>> Too may open files:
>>>> # lsof -p 51798 | wc -l
>>>> 6679
>>>>
>>>>
>>>> Most of open files are TCP sockets:
>>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>>
>>>>
>>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>>> 6413
>>>
>>> Some potentially dumb questions:
>>>
>>> 1. Did you upgrade anything recently (like Java VM)?
>>>
>>> 2. What is error you are seeing? A full stack trace would be helpful.
>>>
>>> 3. What is your <Connector> configuration?
>>>
>>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>>> Different kernel versions count things differently.
>>>
>>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>>>
>>> -chris
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi,
Yes; that is what I get and as you can see among 8028 open files, 7474 are for TCPv6 sockets:
java 130244 root 7805u sock 0,9 0t0 12294251 protocol: TCPv6
# ps -ef | grep tomcat
root 130244 1 1 11:01 ? 00:06:20 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
root 132566 132492 0 20:10 pts/1 00:00:00 grep --color=auto tomcat
#lsof -p 130244 | wc -l
8028
#lsof -p 130244 | grep "protocol: TCPv6"| wc -l
7474
Thanks
-----Original Message-----
From: André Warnier (tomcat/perl) <aw...@ice-sa.com>
Sent: Thursday, May 20, 2021 4:19 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Hi.
According to the list below, you have 2 java (JVM) processes running on your system.
One (PID = 130244) is the JVM which runs tomcat. This is visible when you look at the whole command-line.
The other (PID = 130516) runs ElasticSearch, which I believe is not relevant here.
So you should run lsof with the "-p 130244" option, to show only the files opened by the tomcat JVM.
To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat"
(or anything else unique in that line)
On 20.05.2021 21:00, Yeggy Javadi wrote:
> Hi Chris,
> Please indicate how to show lsof or netstat to *just the JVM* process.
>
> Below is the list of running processes on my server:
> UID PID PPID C STIME TTY TIME CMD
> root 1 0 0 May07 ? 00:00:14 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
> root 2 0 0 May07 ? 00:00:00 [kthreadd]
> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
> root 6 2 0 May07 ? 00:00:00 [kworker/0:0H-kblockd]
> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
> root 11 2 0 May07 ? 00:00:00 [migration/0]
> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
> root 16 2 0 May07 ? 00:00:00 [migration/1]
> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
> root 19 2 0 May07 ? 00:00:00 [kworker/1:0H-kblockd]
> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
> root 22 2 0 May07 ? 00:00:00 [migration/2]
> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
> root 25 2 0 May07 ? 00:00:00 [kworker/2:0H-kblockd]
> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
> root 28 2 0 May07 ? 00:00:00 [migration/3]
> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
> root 31 2 0 May07 ? 00:00:00 [kworker/3:0H-kblockd]
> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
> root 34 2 0 May07 ? 00:00:00 [migration/4]
> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
> root 37 2 0 May07 ? 00:00:00 [kworker/4:0H-kblockd]
> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
> root 40 2 0 May07 ? 00:00:00 [migration/5]
> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
> root 43 2 0 May07 ? 00:00:00 [kworker/5:0H-kblockd]
> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
> root 46 2 0 May07 ? 00:00:00 [migration/6]
> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
> root 49 2 0 May07 ? 00:00:00 [kworker/6:0H-kblockd]
> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
> root 52 2 0 May07 ? 00:00:00 [migration/7]
> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
> root 55 2 0 May07 ? 00:00:00 [kworker/7:0H-kblockd]
> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
> root 58 2 0 May07 ? 00:00:00 [netns]
> root 59 2 0 May07 ? 00:00:00 [kauditd]
> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
> root 64 2 0 May07 ? 00:00:00 [writeback]
> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
> root 66 2 0 May07 ? 00:00:00 [ksmd]
> root 67 2 0 May07 ? 00:00:02 [khugepaged]
> root 68 2 0 May07 ? 00:00:00 [crypto]
> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
> root 70 2 0 May07 ? 00:00:00 [kblockd]
> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
> root 72 2 0 May07 ? 00:00:00 [md]
> root 73 2 0 May07 ? 00:00:00 [edac-poller]
> root 74 2 0 May07 ? 00:00:00 [watchdogd]
> root 92 2 0 May07 ? 00:00:00 [kswapd0]
> root 188 2 0 May07 ? 00:00:00 [kthrotld]
> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
> root 224 2 0 May07 ? 00:00:00 [kaluad]
> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
> root 227 2 0 May07 ? 00:00:00 [kstrp]
> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
> root 509 2 0 May07 ? 00:00:00 [ata_sff]
> root 510 2 0 May07 ? 00:00:00 [mpt/0]
> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
> root 540 2 0 May07 ? 00:00:01 [kworker/7:1H-kblockd]
> root 541 2 0 May07 ? 00:00:01 [kworker/5:1H-kblockd]
> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 613 2 0 May07 ? 00:00:01 [kworker/6:1H-kblockd]
> root 650 2 0 May07 ? 00:00:01 [kworker/4:1H-kblockd]
> root 659 2 0 May07 ? 00:00:02 [kworker/3:1H-kblockd]
> root 660 2 0 May07 ? 00:00:02 [kworker/2:1H-kblockd]
> root 662 2 0 May07 ? 00:00:01 [kworker/0:1H-kblockd]
> root 679 1 0 May07 ? 00:00:03 /usr/lib/systemd/systemd-journald
> root 682 2 0 May07 ? 00:00:02 [kworker/1:1H-kblockd]
> root 716 1 0 May07 ? 00:00:01 /usr/lib/systemd/systemd-udevd
> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
> root 790 2 0 May07 ? 00:00:00 [rpciod]
> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
> root 792 2 0 May07 ? 00:00:00 [xprtiod]
> dbus 883 1 0 May07 ? 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
> root 885 1 0 May07 ? 00:02:42 /usr/sbin/NetworkManager --no-daemon
> root 887 1 0 May07 ? 00:00:44 /usr/sbin/irqbalance --foreground
> root 888 1 0 May07 ? 00:00:00 /usr/bin/VGAuthService -s
> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i --logger=files
> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f --fill-watermark=0
> root 946 890 0 May07 ? 00:00:06 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f -d2
> root 962 890 0 May07 ? 00:00:12 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy -D
> root 970 1 0 May07 ? 00:00:02 /usr/lib/systemd/systemd-logind
> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd -n
> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
> root 6223 1 0 May07 ? 00:07:27 /usr/local/vtl/bin/sanidmgr
> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
> root 6463 2 0 May07 ? 00:00:12 [iocore]
> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
> root 6523 2 0 May07 ? 00:01:50 [trudpd]
> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
> root 6557 2 0 May07 ? 00:00:00 [ipstord]
> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
> root 6654 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6655 6654 0 May07 ? 00:05:18 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6656 6655 0 May07 ? 00:00:36 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6693 1 0 May07 ? 00:00:00 rdesirkey
> root 6699 1 0 May07 ? 00:00:12 ikmd
> root 6721 1 0 May07 ? 00:00:35 /usr/local/vtl/bin/iscliproxy
> root 6747 1 0 May07 ? 00:00:40 /usr/local/vtl/bin/ipstorlog
> root 6780 1 0 May07 ? 00:04:50 /usr/local/vtl/bin/snmpd
> root 7722 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorsm
> root 7723 7722 0 May07 ? 01:08:12 /usr/local/vtl/bin/ipstorsm
> root 7901 1 0 May07 ? 00:00:00 /usr/lib/systemd/systemd --user
> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
> root 14202 14201 0 May08 pts/0 00:00:00 -bash
> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
> root 128108 2 0 08:40 ? 00:00:00 [kworker/4:1-events_power_efficient]
> root 128360 2 0 09:10 ? 00:00:00 [kworker/4:2-cgroup_destroy]
> root 128430 2 0 09:30 ? 00:00:00 [kworker/6:0-mm_percpu_wq]
> root 128738 2 0 11:00 ? 00:00:00 [kworker/3:3-mm_percpu_wq]
> root 128761 2 0 11:00 ? 00:00:00 [kworker/2:1-mm_percpu_wq]
> root 130244 1 1 11:01 ? 00:03:50 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
> postgres 130548 1 0 11:01 ? 00:00:00 /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger process
> postgres 130563 130548 0 11:01 ? 00:00:00 postgres: checkpointer process
> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer process
> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal writer process
> postgres 130566 130548 0 11:01 ? 00:00:00 postgres: autovacuum launcher process
> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats collector process
> root 130570 1 0 11:01 ? 00:00:01 /usr/local/freestor/bin/fmsd
> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42660) idle
> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres freestor 127.0.0.1(42662) idle
> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42664) idle
> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
> root 131347 2 0 13:49 ? 00:00:00 [kworker/5:1-mm_percpu_wq]
> root 131364 2 0 13:56 ? 00:00:00 [kworker/u16:2-events_unbound]
> root 131401 2 0 14:05 ? 00:00:00 [kworker/u16:1-events_unbound]
> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
> root 131515 2 0 14:49 ? 00:00:00 [kworker/1:0-events_freezable]
> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>
> Thanks
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Thursday, May 20, 2021 12:49 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Yeggy,
>
> On 5/19/21 19:37, Yeggy Javadi wrote:
>> Hi Chris,
>> Thanks for your email. Below are replies to your questions:
>>
>> 1. Did you upgrade anything recently (like Java VM)?
>> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
>>
>> 2. What is error you are seeing? A full stack trace would be helpful.
>> [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
>>
>> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
>> Error : The XML configuration file failed to be retrieved for server
>> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
>> server=10.1.6.25 error=java.io.FileNotFoundException:
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>
>> 3. What is your <Connector> configuration?
>> [YJ] It is as below in server.xml:
>>
>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>> maxThreads="150"
>> minSpareThreads="25" maxSpareThreads="75"
>> enableLookups="false" acceptCount="100"
>> connectionTimeout="20000" disableUploadTimeout="true"
>> compression="on"
>> compressionMinSize="2048"
>> noCompressionUserAgents="gozilla, traviata"
>> compressableMimeType="text/html,text/xml,application/json"
>> redirectPort="443" />
>> ...........
>> <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>> maxThreads="150" scheme="https" secure="true"
>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>> keystoreFile="conf/freestor" keystorePass="freestor"/>
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>> redirectPort="443" secretRequired="false" />
>
> You have lots of settings on the HTTP connector but not the HTTPS one.
> Is that intentional? Which one are you using? Or both?
>
> Are you using AJP? If not, disable the connetor entirely. Definitely never enable it on 0.0.0.0 unless you really know what you are doing.
>
>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>> Different kernel versions count things differently.
>> [YJ] I am just getting the process ID of tomcat and count open files
>> for that process
>
> Ok.
>
>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>> [YJ] Here is the netstat output:
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
>
> That's not helpful. I want to see what the JVM process is holding, not what the whole OS is handling.
>
> My guess is that lsof is going to be a better bet for inspecting the process. But if you get "too many open files" being thrown inside the JVM and you don't see a 1/4 million lines of "lsof" output on your process, then something else is going on. But remember to limit your lsof to *just the JVM* process.
>
> -chris
>
>> -----Original Message-----
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Monday, May 17, 2021 5:18 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Yeggy,
>>
>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>> Hi,
>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>
>>> I was running the same application on Linux 7.6 with the same
>>> apache-tomcat version without seeing this issue. I saw that RedHat
>>> has removed apache-tomcat fom their Linux 8 distribution. Please help
>>> with throubleshooting and indicating if there is any changes in Linux
>>> 8 that can explain this
>>>
>>> Configuration:
>>> apache-tomcat-8.5.59
>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
>>> using Postgres and elastic search databases
>>>
>>>
>>> # ps -ef | grep tomcat
>>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>>>
>>>
>>> # cat /proc/51798/limits
>>> Limit Soft Limit Hard Limit Units
>>> Max cpu time unlimited unlimited seconds
>>> Max file size unlimited unlimited bytes
>>> Max data size unlimited unlimited bytes
>>> Max stack size 8388608 unlimited bytes
>>> Max core file size unlimited unlimited bytes
>>> Max resident set unlimited unlimited bytes
>>> Max processes 256518 256518 processes
>>> Max open files 262144 262144 files
>>> Max locked memory 65536 65536 bytes
>>> Max address space unlimited unlimited bytes
>>> Max file locks unlimited unlimited locks
>>> Max pending signals 256518 256518 signals
>>> Max msgqueue size 819200 819200 bytes
>>> Max nice priority 0 0
>>> Max realtime priority 0 0
>>> Max realtime timeout unlimited unlimited us
>>>
>>>
>>> Too may open files:
>>> # lsof -p 51798 | wc -l
>>> 6679
>>>
>>>
>>> Most of open files are TCP sockets:
>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>
>>>
>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>> 6413
>>
>> Some potentially dumb questions:
>>
>> 1. Did you upgrade anything recently (like Java VM)?
>>
>> 2. What is error you are seeing? A full stack trace would be helpful.
>>
>> 3. What is your <Connector> configuration?
>>
>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>> Different kernel versions count things differently.
>>
>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Hi.
According to the list below, you have 2 java (JVM) processes running on your system.
One (PID = 130244) is the JVM which runs tomcat. This is visible when you look at the
whole command-line.
The other (PID = 130516) runs ElasticSearch, which I believe is not relevant here.
So you should run lsof with the "-p 130244" option, to show only the files opened by the
tomcat JVM.
To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat"
(or anything else unique in that line)
On 20.05.2021 21:00, Yeggy Javadi wrote:
> Hi Chris,
> Please indicate how to show lsof or netstat to *just the JVM* process.
>
> Below is the list of running processes on my server:
> UID PID PPID C STIME TTY TIME CMD
> root 1 0 0 May07 ? 00:00:14 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
> root 2 0 0 May07 ? 00:00:00 [kthreadd]
> root 3 2 0 May07 ? 00:00:00 [rcu_gp]
> root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
> root 6 2 0 May07 ? 00:00:00 [kworker/0:0H-kblockd]
> root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
> root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
> root 10 2 0 May07 ? 00:02:18 [rcu_sched]
> root 11 2 0 May07 ? 00:00:00 [migration/0]
> root 12 2 0 May07 ? 00:00:00 [watchdog/0]
> root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
> root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
> root 15 2 0 May07 ? 00:00:00 [watchdog/1]
> root 16 2 0 May07 ? 00:00:00 [migration/1]
> root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
> root 19 2 0 May07 ? 00:00:00 [kworker/1:0H-kblockd]
> root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
> root 21 2 0 May07 ? 00:00:00 [watchdog/2]
> root 22 2 0 May07 ? 00:00:00 [migration/2]
> root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
> root 25 2 0 May07 ? 00:00:00 [kworker/2:0H-kblockd]
> root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
> root 27 2 0 May07 ? 00:00:00 [watchdog/3]
> root 28 2 0 May07 ? 00:00:00 [migration/3]
> root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
> root 31 2 0 May07 ? 00:00:00 [kworker/3:0H-kblockd]
> root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
> root 33 2 0 May07 ? 00:00:00 [watchdog/4]
> root 34 2 0 May07 ? 00:00:00 [migration/4]
> root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
> root 37 2 0 May07 ? 00:00:00 [kworker/4:0H-kblockd]
> root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
> root 39 2 0 May07 ? 00:00:00 [watchdog/5]
> root 40 2 0 May07 ? 00:00:00 [migration/5]
> root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
> root 43 2 0 May07 ? 00:00:00 [kworker/5:0H-kblockd]
> root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
> root 45 2 0 May07 ? 00:00:00 [watchdog/6]
> root 46 2 0 May07 ? 00:00:00 [migration/6]
> root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
> root 49 2 0 May07 ? 00:00:00 [kworker/6:0H-kblockd]
> root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
> root 51 2 0 May07 ? 00:00:00 [watchdog/7]
> root 52 2 0 May07 ? 00:00:00 [migration/7]
> root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
> root 55 2 0 May07 ? 00:00:00 [kworker/7:0H-kblockd]
> root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
> root 58 2 0 May07 ? 00:00:00 [netns]
> root 59 2 0 May07 ? 00:00:00 [kauditd]
> root 62 2 0 May07 ? 00:00:00 [khungtaskd]
> root 63 2 0 May07 ? 00:00:00 [oom_reaper]
> root 64 2 0 May07 ? 00:00:00 [writeback]
> root 65 2 0 May07 ? 00:00:00 [kcompactd0]
> root 66 2 0 May07 ? 00:00:00 [ksmd]
> root 67 2 0 May07 ? 00:00:02 [khugepaged]
> root 68 2 0 May07 ? 00:00:00 [crypto]
> root 69 2 0 May07 ? 00:00:00 [kintegrityd]
> root 70 2 0 May07 ? 00:00:00 [kblockd]
> root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
> root 72 2 0 May07 ? 00:00:00 [md]
> root 73 2 0 May07 ? 00:00:00 [edac-poller]
> root 74 2 0 May07 ? 00:00:00 [watchdogd]
> root 92 2 0 May07 ? 00:00:00 [kswapd0]
> root 188 2 0 May07 ? 00:00:00 [kthrotld]
> root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
> root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
> root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
> root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
> root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
> root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
> root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
> root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
> root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
> root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
> root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
> root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
> root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
> root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
> root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
> root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
> root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
> root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
> root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
> root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
> root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
> root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
> root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
> root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
> root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
> root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
> root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
> root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
> root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
> root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
> root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
> root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
> root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
> root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
> root 224 2 0 May07 ? 00:00:00 [kaluad]
> root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
> root 227 2 0 May07 ? 00:00:00 [kstrp]
> root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
> root 509 2 0 May07 ? 00:00:00 [ata_sff]
> root 510 2 0 May07 ? 00:00:00 [mpt/0]
> root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
> root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
> root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
> root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
> root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
> root 525 2 0 May07 ? 00:00:00 [ttm_swap]
> root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
> root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
> root 540 2 0 May07 ? 00:00:01 [kworker/7:1H-kblockd]
> root 541 2 0 May07 ? 00:00:01 [kworker/5:1H-kblockd]
> root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
> root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
> root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 613 2 0 May07 ? 00:00:01 [kworker/6:1H-kblockd]
> root 650 2 0 May07 ? 00:00:01 [kworker/4:1H-kblockd]
> root 659 2 0 May07 ? 00:00:02 [kworker/3:1H-kblockd]
> root 660 2 0 May07 ? 00:00:02 [kworker/2:1H-kblockd]
> root 662 2 0 May07 ? 00:00:01 [kworker/0:1H-kblockd]
> root 679 1 0 May07 ? 00:00:03 /usr/lib/systemd/systemd-journald
> root 682 2 0 May07 ? 00:00:02 [kworker/1:1H-kblockd]
> root 716 1 0 May07 ? 00:00:01 /usr/lib/systemd/systemd-udevd
> root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
> root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
> root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
> root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
> root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
> root 776 1 0 May07 ? 00:00:01 /sbin/auditd
> root 790 2 0 May07 ? 00:00:00 [rpciod]
> root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
> root 792 2 0 May07 ? 00:00:00 [xprtiod]
> dbus 883 1 0 May07 ? 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
> root 885 1 0 May07 ? 00:02:42 /usr/sbin/NetworkManager --no-daemon
> root 887 1 0 May07 ? 00:00:44 /usr/sbin/irqbalance --foreground
> root 888 1 0 May07 ? 00:00:00 /usr/bin/VGAuthService -s
> root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
> root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i --logger=files
> chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
> rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f --fill-watermark=0
> root 946 890 0 May07 ? 00:00:06 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
> root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f -d2
> root 962 890 0 May07 ? 00:00:12 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
> root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy -D
> root 970 1 0 May07 ? 00:00:02 /usr/lib/systemd/systemd-logind
> root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
> root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd -n
> root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
> root 6223 1 0 May07 ? 00:07:27 /usr/local/vtl/bin/sanidmgr
> root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
> root 6272 2 0 May07 ? 00:00:03 [bksc_done]
> root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
> root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
> root 6460 2 0 May07 ? 00:00:00 [kdownstream]
> root 6461 2 0 May07 ? 00:00:07 [msgfilter]
> root 6463 2 0 May07 ? 00:00:12 [iocore]
> root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
> root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
> root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
> root 6523 2 0 May07 ? 00:01:50 [trudpd]
> root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
> root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
> root 6551 2 0 May07 ? 00:00:00 [misc_thread]
> root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
> root 6554 2 0 May07 ? 00:00:00 [listen_handle]
> root 6557 2 0 May07 ? 00:00:00 [ipstord]
> root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
> root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
> root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
> root 6654 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6655 6654 0 May07 ? 00:05:18 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6656 6655 0 May07 ? 00:00:36 /usr/local/vtl/bin/ipstorcomm -d 0
> root 6693 1 0 May07 ? 00:00:00 rdesirkey
> root 6699 1 0 May07 ? 00:00:12 ikmd
> root 6721 1 0 May07 ? 00:00:35 /usr/local/vtl/bin/iscliproxy
> root 6747 1 0 May07 ? 00:00:40 /usr/local/vtl/bin/ipstorlog
> root 6780 1 0 May07 ? 00:04:50 /usr/local/vtl/bin/snmpd
> root 7722 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorsm
> root 7723 7722 0 May07 ? 01:08:12 /usr/local/vtl/bin/ipstorsm
> root 7901 1 0 May07 ? 00:00:00 /usr/lib/systemd/systemd --user
> root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
> root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
> root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
> root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
> root 14202 14201 0 May08 pts/0 00:00:00 -bash
> root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
> root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
> root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
> root 128108 2 0 08:40 ? 00:00:00 [kworker/4:1-events_power_efficient]
> root 128360 2 0 09:10 ? 00:00:00 [kworker/4:2-cgroup_destroy]
> root 128430 2 0 09:30 ? 00:00:00 [kworker/6:0-mm_percpu_wq]
> root 128738 2 0 11:00 ? 00:00:00 [kworker/3:3-mm_percpu_wq]
> root 128761 2 0 11:00 ? 00:00:00 [kworker/2:1-mm_percpu_wq]
> root 130244 1 1 11:01 ? 00:03:50 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
> postgres 130548 1 0 11:01 ? 00:00:00 /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
> postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger process
> postgres 130563 130548 0 11:01 ? 00:00:00 postgres: checkpointer process
> postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer process
> postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal writer process
> postgres 130566 130548 0 11:01 ? 00:00:00 postgres: autovacuum launcher process
> postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats collector process
> root 130570 1 0 11:01 ? 00:00:01 /usr/local/freestor/bin/fmsd
> postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42660) idle
> postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres freestor 127.0.0.1(42662) idle
> postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42664) idle
> root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
> root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
> root 131347 2 0 13:49 ? 00:00:00 [kworker/5:1-mm_percpu_wq]
> root 131364 2 0 13:56 ? 00:00:00 [kworker/u16:2-events_unbound]
> root 131401 2 0 14:05 ? 00:00:00 [kworker/u16:1-events_unbound]
> root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
> root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
> root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
> root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
> root 131515 2 0 14:49 ? 00:00:00 [kworker/1:0-events_freezable]
> root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
> root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
> root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
> root 131530 131529 0 14:52 pts/1 00:00:00 -bash
> root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
>
> Thanks
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Thursday, May 20, 2021 12:49 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Yeggy,
>
> On 5/19/21 19:37, Yeggy Javadi wrote:
>> Hi Chris,
>> Thanks for your email. Below are replies to your questions:
>>
>> 1. Did you upgrade anything recently (like Java VM)?
>> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
>>
>> 2. What is error you are seeing? A full stack trace would be helpful.
>> [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
>>
>> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
>> Error : The XML configuration file failed to be retrieved for server
>> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
>> server=10.1.6.25 error=java.io.FileNotFoundException:
>> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
>> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>>
>> 3. What is your <Connector> configuration?
>> [YJ] It is as below in server.xml:
>>
>> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
>> maxThreads="150"
>> minSpareThreads="25" maxSpareThreads="75"
>> enableLookups="false" acceptCount="100"
>> connectionTimeout="20000" disableUploadTimeout="true"
>> compression="on"
>> compressionMinSize="2048"
>> noCompressionUserAgents="gozilla, traviata"
>> compressableMimeType="text/html,text/xml,application/json"
>> redirectPort="443" />
>> ...........
>> <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
>> maxThreads="150" scheme="https" secure="true"
>> clientAuth="false" sslEnabledProtocols="TLSv1.2"
>> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
>> keystoreFile="conf/freestor" keystorePass="freestor"/>
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
>> redirectPort="443" secretRequired="false" />
>
> You have lots of settings on the HTTP connector but not the HTTPS one.
> Is that intentional? Which one are you using? Or both?
>
> Are you using AJP? If not, disable the connetor entirely. Definitely never enable it on 0.0.0.0 unless you really know what you are doing.
>
>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>> Different kernel versions count things differently.
>> [YJ] I am just getting the process ID of tomcat and count open files
>> for that process
>
> Ok.
>
>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>> [YJ] Here is the netstat output:
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
>> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
>> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
>
> That's not helpful. I want to see what the JVM process is holding, not what the whole OS is handling.
>
> My guess is that lsof is going to be a better bet for inspecting the process. But if you get "too many open files" being thrown inside the JVM and you don't see a 1/4 million lines of "lsof" output on your process, then something else is going on. But remember to limit your lsof to *just the JVM* process.
>
> -chris
>
>> -----Original Message-----
>> From: Christopher Schultz <ch...@christopherschultz.net>
>> Sent: Monday, May 17, 2021 5:18 PM
>> To: users@tomcat.apache.org
>> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>>
>> Yeggy,
>>
>> On 5/17/21 15:17, Yeggy Javadi wrote:
>>> Hi,
>>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>
>>> I was running the same application on Linux 7.6 with the same
>>> apache-tomcat version without seeing this issue. I saw that RedHat
>>> has removed apache-tomcat fom their Linux 8 distribution. Please help
>>> with throubleshooting and indicating if there is any changes in Linux
>>> 8 that can explain this
>>>
>>> Configuration:
>>> apache-tomcat-8.5.59
>>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
>>> using Postgres and elastic search databases
>>>
>>>
>>> # ps -ef | grep tomcat
>>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>>>
>>>
>>> # cat /proc/51798/limits
>>> Limit Soft Limit Hard Limit Units
>>> Max cpu time unlimited unlimited seconds
>>> Max file size unlimited unlimited bytes
>>> Max data size unlimited unlimited bytes
>>> Max stack size 8388608 unlimited bytes
>>> Max core file size unlimited unlimited bytes
>>> Max resident set unlimited unlimited bytes
>>> Max processes 256518 256518 processes
>>> Max open files 262144 262144 files
>>> Max locked memory 65536 65536 bytes
>>> Max address space unlimited unlimited bytes
>>> Max file locks unlimited unlimited locks
>>> Max pending signals 256518 256518 signals
>>> Max msgqueue size 819200 819200 bytes
>>> Max nice priority 0 0
>>> Max realtime priority 0 0
>>> Max realtime timeout unlimited unlimited us
>>>
>>>
>>> Too may open files:
>>> # lsof -p 51798 | wc -l
>>> 6679
>>>
>>>
>>> Most of open files are TCP sockets:
>>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>>
>>>
>>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>>> 6413
>>
>> Some potentially dumb questions:
>>
>> 1. Did you upgrade anything recently (like Java VM)?
>>
>> 2. What is error you are seeing? A full stack trace would be helpful.
>>
>> 3. What is your <Connector> configuration?
>>
>> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
>> Different kernel versions count things differently.
>>
>> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi Chris,
Please indicate how to show lsof or netstat to *just the JVM* process.
Below is the list of running processes on my server:
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May07 ? 00:00:14 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
root 2 0 0 May07 ? 00:00:00 [kthreadd]
root 3 2 0 May07 ? 00:00:00 [rcu_gp]
root 4 2 0 May07 ? 00:00:00 [rcu_par_gp]
root 6 2 0 May07 ? 00:00:00 [kworker/0:0H-kblockd]
root 8 2 0 May07 ? 00:00:00 [mm_percpu_wq]
root 9 2 0 May07 ? 00:00:00 [ksoftirqd/0]
root 10 2 0 May07 ? 00:02:18 [rcu_sched]
root 11 2 0 May07 ? 00:00:00 [migration/0]
root 12 2 0 May07 ? 00:00:00 [watchdog/0]
root 13 2 0 May07 ? 00:00:00 [cpuhp/0]
root 14 2 0 May07 ? 00:00:00 [cpuhp/1]
root 15 2 0 May07 ? 00:00:00 [watchdog/1]
root 16 2 0 May07 ? 00:00:00 [migration/1]
root 17 2 0 May07 ? 00:00:00 [ksoftirqd/1]
root 19 2 0 May07 ? 00:00:00 [kworker/1:0H-kblockd]
root 20 2 0 May07 ? 00:00:00 [cpuhp/2]
root 21 2 0 May07 ? 00:00:00 [watchdog/2]
root 22 2 0 May07 ? 00:00:00 [migration/2]
root 23 2 0 May07 ? 00:00:00 [ksoftirqd/2]
root 25 2 0 May07 ? 00:00:00 [kworker/2:0H-kblockd]
root 26 2 0 May07 ? 00:00:00 [cpuhp/3]
root 27 2 0 May07 ? 00:00:00 [watchdog/3]
root 28 2 0 May07 ? 00:00:00 [migration/3]
root 29 2 0 May07 ? 00:00:00 [ksoftirqd/3]
root 31 2 0 May07 ? 00:00:00 [kworker/3:0H-kblockd]
root 32 2 0 May07 ? 00:00:00 [cpuhp/4]
root 33 2 0 May07 ? 00:00:00 [watchdog/4]
root 34 2 0 May07 ? 00:00:00 [migration/4]
root 35 2 0 May07 ? 00:00:00 [ksoftirqd/4]
root 37 2 0 May07 ? 00:00:00 [kworker/4:0H-kblockd]
root 38 2 0 May07 ? 00:00:00 [cpuhp/5]
root 39 2 0 May07 ? 00:00:00 [watchdog/5]
root 40 2 0 May07 ? 00:00:00 [migration/5]
root 41 2 0 May07 ? 00:00:00 [ksoftirqd/5]
root 43 2 0 May07 ? 00:00:00 [kworker/5:0H-kblockd]
root 44 2 0 May07 ? 00:00:00 [cpuhp/6]
root 45 2 0 May07 ? 00:00:00 [watchdog/6]
root 46 2 0 May07 ? 00:00:00 [migration/6]
root 47 2 0 May07 ? 00:00:00 [ksoftirqd/6]
root 49 2 0 May07 ? 00:00:00 [kworker/6:0H-kblockd]
root 50 2 0 May07 ? 00:00:00 [cpuhp/7]
root 51 2 0 May07 ? 00:00:00 [watchdog/7]
root 52 2 0 May07 ? 00:00:00 [migration/7]
root 53 2 0 May07 ? 00:00:00 [ksoftirqd/7]
root 55 2 0 May07 ? 00:00:00 [kworker/7:0H-kblockd]
root 57 2 0 May07 ? 00:00:00 [kdevtmpfs]
root 58 2 0 May07 ? 00:00:00 [netns]
root 59 2 0 May07 ? 00:00:00 [kauditd]
root 62 2 0 May07 ? 00:00:00 [khungtaskd]
root 63 2 0 May07 ? 00:00:00 [oom_reaper]
root 64 2 0 May07 ? 00:00:00 [writeback]
root 65 2 0 May07 ? 00:00:00 [kcompactd0]
root 66 2 0 May07 ? 00:00:00 [ksmd]
root 67 2 0 May07 ? 00:00:02 [khugepaged]
root 68 2 0 May07 ? 00:00:00 [crypto]
root 69 2 0 May07 ? 00:00:00 [kintegrityd]
root 70 2 0 May07 ? 00:00:00 [kblockd]
root 71 2 0 May07 ? 00:00:00 [tpm_dev_wq]
root 72 2 0 May07 ? 00:00:00 [md]
root 73 2 0 May07 ? 00:00:00 [edac-poller]
root 74 2 0 May07 ? 00:00:00 [watchdogd]
root 92 2 0 May07 ? 00:00:00 [kswapd0]
root 188 2 0 May07 ? 00:00:00 [kthrotld]
root 189 2 0 May07 ? 00:00:00 [irq/24-pciehp]
root 190 2 0 May07 ? 00:00:00 [irq/25-pciehp]
root 191 2 0 May07 ? 00:00:00 [irq/26-pciehp]
root 192 2 0 May07 ? 00:00:00 [irq/27-pciehp]
root 193 2 0 May07 ? 00:00:00 [irq/28-pciehp]
root 194 2 0 May07 ? 00:00:00 [irq/29-pciehp]
root 195 2 0 May07 ? 00:00:00 [irq/30-pciehp]
root 196 2 0 May07 ? 00:00:00 [irq/31-pciehp]
root 197 2 0 May07 ? 00:00:00 [irq/32-pciehp]
root 198 2 0 May07 ? 00:00:00 [irq/33-pciehp]
root 199 2 0 May07 ? 00:00:00 [irq/34-pciehp]
root 200 2 0 May07 ? 00:00:00 [irq/35-pciehp]
root 201 2 0 May07 ? 00:00:00 [irq/36-pciehp]
root 202 2 0 May07 ? 00:00:00 [irq/37-pciehp]
root 203 2 0 May07 ? 00:00:00 [irq/38-pciehp]
root 204 2 0 May07 ? 00:00:00 [irq/39-pciehp]
root 205 2 0 May07 ? 00:00:00 [irq/40-pciehp]
root 206 2 0 May07 ? 00:00:00 [irq/41-pciehp]
root 207 2 0 May07 ? 00:00:00 [irq/42-pciehp]
root 208 2 0 May07 ? 00:00:00 [irq/43-pciehp]
root 209 2 0 May07 ? 00:00:00 [irq/44-pciehp]
root 210 2 0 May07 ? 00:00:00 [irq/45-pciehp]
root 211 2 0 May07 ? 00:00:00 [irq/46-pciehp]
root 212 2 0 May07 ? 00:00:00 [irq/47-pciehp]
root 213 2 0 May07 ? 00:00:00 [irq/48-pciehp]
root 214 2 0 May07 ? 00:00:00 [irq/49-pciehp]
root 215 2 0 May07 ? 00:00:00 [irq/50-pciehp]
root 216 2 0 May07 ? 00:00:00 [irq/51-pciehp]
root 217 2 0 May07 ? 00:00:00 [irq/52-pciehp]
root 218 2 0 May07 ? 00:00:00 [irq/53-pciehp]
root 219 2 0 May07 ? 00:00:00 [irq/54-pciehp]
root 220 2 0 May07 ? 00:00:00 [irq/55-pciehp]
root 222 2 0 May07 ? 00:00:00 [acpi_thermal_pm]
root 223 2 0 May07 ? 00:00:00 [kmpath_rdacd]
root 224 2 0 May07 ? 00:00:00 [kaluad]
root 226 2 0 May07 ? 00:00:00 [ipv6_addrconf]
root 227 2 0 May07 ? 00:00:00 [kstrp]
root 508 2 0 May07 ? 00:00:00 [mpt_poll_0]
root 509 2 0 May07 ? 00:00:00 [ata_sff]
root 510 2 0 May07 ? 00:00:00 [mpt/0]
root 512 2 0 May07 ? 00:00:00 [scsi_eh_0]
root 516 2 0 May07 ? 00:00:00 [scsi_tmf_0]
root 517 2 0 May07 ? 00:00:00 [scsi_eh_1]
root 518 2 0 May07 ? 00:00:00 [scsi_tmf_1]
root 524 2 0 May07 ? 00:00:14 [irq/16-vmwgfx]
root 525 2 0 May07 ? 00:00:00 [ttm_swap]
root 531 2 0 May07 ? 00:00:00 [scsi_eh_2]
root 532 2 0 May07 ? 00:00:00 [scsi_tmf_2]
root 540 2 0 May07 ? 00:00:01 [kworker/7:1H-kblockd]
root 541 2 0 May07 ? 00:00:01 [kworker/5:1H-kblockd]
root 567 2 0 May07 ? 00:00:07 [jbd2/sda3-8]
root 568 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 611 2 0 May07 ? 00:00:49 [jbd2/sda2-8]
root 612 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 613 2 0 May07 ? 00:00:01 [kworker/6:1H-kblockd]
root 650 2 0 May07 ? 00:00:01 [kworker/4:1H-kblockd]
root 659 2 0 May07 ? 00:00:02 [kworker/3:1H-kblockd]
root 660 2 0 May07 ? 00:00:02 [kworker/2:1H-kblockd]
root 662 2 0 May07 ? 00:00:01 [kworker/0:1H-kblockd]
root 679 1 0 May07 ? 00:00:03 /usr/lib/systemd/systemd-journald
root 682 2 0 May07 ? 00:00:02 [kworker/1:1H-kblockd]
root 716 1 0 May07 ? 00:00:01 /usr/lib/systemd/systemd-udevd
root 723 2 0 May07 ? 00:00:18 [jbd2/sdb1-8]
root 724 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 733 2 0 May07 ? 00:00:14 [jbd2/sda6-8]
root 734 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 738 2 0 May07 ? 00:00:00 [jbd2/sdc1-8]
root 739 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 744 2 0 May07 ? 00:00:00 [jbd2/sda1-8]
root 745 2 0 May07 ? 00:00:00 [ext4-rsv-conver]
root 776 1 0 May07 ? 00:00:01 /sbin/auditd
root 790 2 0 May07 ? 00:00:00 [rpciod]
root 791 2 0 May07 ? 00:00:00 [kworker/u17:0]
root 792 2 0 May07 ? 00:00:00 [xprtiod]
dbus 883 1 0 May07 ? 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 885 1 0 May07 ? 00:02:42 /usr/sbin/NetworkManager --no-daemon
root 887 1 0 May07 ? 00:00:44 /usr/sbin/irqbalance --foreground
root 888 1 0 May07 ? 00:00:00 /usr/bin/VGAuthService -s
root 889 1 0 May07 ? 00:10:28 /usr/bin/vmtoolsd
root 890 1 0 May07 ? 00:00:00 /usr/sbin/sssd -i --logger=files
chrony 906 1 0 May07 ? 00:00:00 /usr/sbin/chronyd
rngd 944 1 0 May07 ? 00:00:49 /sbin/rngd -f --fill-watermark=0
root 946 890 0 May07 ? 00:00:06 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
root 954 1 0 May07 ? 00:00:00 /usr/sbin/iscsid -f -d2
root 962 890 0 May07 ? 00:00:12 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
root 963 1 0 May07 ? 00:00:00 /usr/sbin/gssproxy -D
root 970 1 0 May07 ? 00:00:02 /usr/lib/systemd/systemd-logind
root 971 2 0 May07 ? 00:00:00 [iscsi_eh]
root 998 1 0 May07 ? 00:00:33 /usr/sbin/rsyslogd -n
root 1007 1 0 May07 ? 00:00:00 /usr/sbin/crond -n
root 6223 1 0 May07 ? 00:07:27 /usr/local/vtl/bin/sanidmgr
root 6271 2 0 May07 ? 00:00:03 [bksc_exec]
root 6272 2 0 May07 ? 00:00:03 [bksc_done]
root 6274 2 0 May07 ? 00:00:00 [scsi_eh_3]
root 6275 2 0 May07 ? 00:00:00 [scsi_tmf_3]
root 6460 2 0 May07 ? 00:00:00 [kdownstream]
root 6461 2 0 May07 ? 00:00:07 [msgfilter]
root 6463 2 0 May07 ? 00:00:12 [iocore]
root 6466 2 0 May07 ? 00:00:02 [ioctl_mgr]
root 6467 2 0 May07 ? 00:00:02 [ioctl_mgr]
root 6505 1 0 May07 ? 00:00:00 fsnupd -d 0
root 6523 2 0 May07 ? 00:01:50 [trudpd]
root 6539 2 0 May07 ? 00:00:00 [fsnalias_worker]
root 6541 2 0 May07 ? 00:00:00 [fsnalias_check]
root 6551 2 0 May07 ? 00:00:00 [misc_thread]
root 6553 2 0 May07 ? 00:00:01 [mtcp2_listen]
root 6554 2 0 May07 ? 00:00:00 [listen_handle]
root 6557 2 0 May07 ? 00:00:00 [ipstord]
root 6579 2 0 May07 ? 00:00:00 [kfsvshostwq]
root 6580 2 0 May07 ? 00:00:00 [scsi_eh_4]
root 6582 2 0 May07 ? 00:00:00 [scsi_tmf_4]
root 6654 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorcomm -d 0
root 6655 6654 0 May07 ? 00:05:18 /usr/local/vtl/bin/ipstorcomm -d 0
root 6656 6655 0 May07 ? 00:00:36 /usr/local/vtl/bin/ipstorcomm -d 0
root 6693 1 0 May07 ? 00:00:00 rdesirkey
root 6699 1 0 May07 ? 00:00:12 ikmd
root 6721 1 0 May07 ? 00:00:35 /usr/local/vtl/bin/iscliproxy
root 6747 1 0 May07 ? 00:00:40 /usr/local/vtl/bin/ipstorlog
root 6780 1 0 May07 ? 00:04:50 /usr/local/vtl/bin/snmpd
root 7722 1 0 May07 ? 00:00:00 /usr/local/vtl/bin/ipstorsm
root 7723 7722 0 May07 ? 01:08:12 /usr/local/vtl/bin/ipstorsm
root 7901 1 0 May07 ? 00:00:00 /usr/lib/systemd/systemd --user
root 7904 7901 0 May07 ? 00:00:00 (sd-pam)
root 8179 1 0 May07 tty1 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root 14199 1 0 May08 ? 00:00:01 sshd: root [priv]
root 14201 14199 0 May08 ? 00:00:01 sshd: root@pts/0
root 14202 14201 0 May08 pts/0 00:00:00 -bash
root 14261 1 0 May08 ? 00:00:00 /usr/sbin/sshd -D -oCiphers=aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc -oMACs=hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512 -oGSSAPIKexAlgorithms=gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1- -oKexAlgorithms=curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 -oHostKeyAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypes=ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa,ssh-rsa-cert-v01@openssh.com -oCASignatureAlgorithms=ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
root 122905 2 0 May19 ? 00:00:27 [kworker/7:0-events]
root 126037 2 0 05:30 ? 00:00:00 [kworker/1:2-events]
root 128108 2 0 08:40 ? 00:00:00 [kworker/4:1-events_power_efficient]
root 128360 2 0 09:10 ? 00:00:00 [kworker/4:2-cgroup_destroy]
root 128430 2 0 09:30 ? 00:00:00 [kworker/6:0-mm_percpu_wq]
root 128738 2 0 11:00 ? 00:00:00 [kworker/3:3-mm_percpu_wq]
root 128761 2 0 11:00 ? 00:00:00 [kworker/2:1-mm_percpu_wq]
root 130244 1 1 11:01 ? 00:03:50 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
elastic+ 130516 1 1 11:01 ? 00:03:26 /bin/java -Xms8g -Xmx8g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.6.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
postgres 130548 1 0 11:01 ? 00:00:00 /usr/pgsql-9.6/bin/postgres -D /DB/postgres/data
postgres 130561 130548 0 11:01 ? 00:00:00 postgres: logger process
postgres 130563 130548 0 11:01 ? 00:00:00 postgres: checkpointer process
postgres 130564 130548 0 11:01 ? 00:00:00 postgres: writer process
postgres 130565 130548 0 11:01 ? 00:00:00 postgres: wal writer process
postgres 130566 130548 0 11:01 ? 00:00:00 postgres: autovacuum launcher process
postgres 130567 130548 0 11:01 ? 00:00:00 postgres: stats collector process
root 130570 1 0 11:01 ? 00:00:01 /usr/local/freestor/bin/fmsd
postgres 130675 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42660) idle
postgres 130676 130548 0 11:01 ? 00:00:04 postgres: postgres freestor 127.0.0.1(42662) idle
postgres 130677 130548 0 11:01 ? 00:00:03 postgres: postgres freestor 127.0.0.1(42664) idle
root 131206 2 0 13:00 ? 00:00:00 [kworker/7:2]
root 131269 2 0 13:20 ? 00:00:00 [kworker/6:2]
root 131347 2 0 13:49 ? 00:00:00 [kworker/5:1-mm_percpu_wq]
root 131364 2 0 13:56 ? 00:00:00 [kworker/u16:2-events_unbound]
root 131401 2 0 14:05 ? 00:00:00 [kworker/u16:1-events_unbound]
root 131438 2 0 14:20 ? 00:00:00 [kworker/2:2]
root 131465 2 0 14:30 ? 00:00:00 [kworker/0:2-events]
root 131490 2 0 14:40 ? 00:00:00 [kworker/3:0]
root 131491 2 0 14:40 ? 00:00:00 [kworker/0:1-events]
root 131515 2 0 14:49 ? 00:00:00 [kworker/1:0-events_freezable]
root 131520 2 0 14:50 ? 00:00:00 [kworker/5:0]
root 131525 14261 0 14:52 ? 00:00:00 sshd: root [priv]
root 131529 131525 0 14:52 ? 00:00:00 sshd: root@pts/1
root 131530 131529 0 14:52 pts/1 00:00:00 -bash
root 131600 131530 0 14:57 pts/1 00:00:00 ps -ef
Thanks
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, May 20, 2021 12:49 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/19/21 19:37, Yeggy Javadi wrote:
> Hi Chris,
> Thanks for your email. Below are replies to your questions:
>
> 1. Did you upgrade anything recently (like Java VM)?
> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
>
> 2. What is error you are seeing? A full stack trace would be helpful.
> [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
>
> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService -
> Error : The XML configuration file failed to be retrieved for server
> 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for
> server=10.1.6.25 error=java.io.FileNotFoundException:
> /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.
> 2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>
> 3. What is your <Connector> configuration?
> [YJ] It is as below in server.xml:
>
> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
> maxThreads="150"
> minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false" acceptCount="100"
> connectionTimeout="20000" disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml,application/json"
> redirectPort="443" />
> ...........
> <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> maxThreads="150" scheme="https" secure="true"
> clientAuth="false" sslEnabledProtocols="TLSv1.2"
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
> keystoreFile="conf/freestor" keystorePass="freestor"/>
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3"
> redirectPort="443" secretRequired="false" />
You have lots of settings on the HTTP connector but not the HTTPS one.
Is that intentional? Which one are you using? Or both?
Are you using AJP? If not, disable the connetor entirely. Definitely never enable it on 0.0.0.0 unless you really know what you are doing.
> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
> Different kernel versions count things differently.
> [YJ] I am just getting the process ID of tomcat and count open files
> for that process
Ok.
> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
> [YJ] Here is the netstat output:
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
That's not helpful. I want to see what the JVM process is holding, not what the whole OS is handling.
My guess is that lsof is going to be a better bet for inspecting the process. But if you get "too many open files" being thrown inside the JVM and you don't see a 1/4 million lines of "lsof" output on your process, then something else is going on. But remember to limit your lsof to *just the JVM* process.
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, May 17, 2021 5:18 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Yeggy,
>
> On 5/17/21 15:17, Yeggy Javadi wrote:
>> Hi,
>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>
>> I was running the same application on Linux 7.6 with the same
>> apache-tomcat version without seeing this issue. I saw that RedHat
>> has removed apache-tomcat fom their Linux 8 distribution. Please help
>> with throubleshooting and indicating if there is any changes in Linux
>> 8 that can explain this
>>
>> Configuration:
>> apache-tomcat-8.5.59
>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
>> using Postgres and elastic search databases
>>
>>
>> # ps -ef | grep tomcat
>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>>
>>
>> # cat /proc/51798/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 8388608 unlimited bytes
>> Max core file size unlimited unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 256518 256518 processes
>> Max open files 262144 262144 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 256518 256518 signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>>
>>
>> Too may open files:
>> # lsof -p 51798 | wc -l
>> 6679
>>
>>
>> Most of open files are TCP sockets:
>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>
>>
>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>> 6413
>
> Some potentially dumb questions:
>
> 1. Did you upgrade anything recently (like Java VM)?
>
> 2. What is error you are seeing? A full stack trace would be helpful.
>
> 3. What is your <Connector> configuration?
>
> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
> Different kernel versions count things differently.
>
> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Yeggy,
On 5/19/21 19:37, Yeggy Javadi wrote:
> Hi Chris,
> Thanks for your email. Below are replies to your questions:
>
> 1. Did you upgrade anything recently (like Java VM)?
> [YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
>
> 2. What is error you are seeing? A full stack trace would be helpful.
> [YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
>
> [InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The XML configuration file failed to be retrieved for server 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for server=10.1.6.25 error=java.io.FileNotFoundException: /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar (Too many open files) restatus=-1 output=
>
> 3. What is your <Connector> configuration?
> [YJ] It is as below in server.xml:
>
> <Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
> maxThreads="150"
> minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false" acceptCount="100"
> connectionTimeout="20000" disableUploadTimeout="true"
> compression="on"
> compressionMinSize="2048"
> noCompressionUserAgents="gozilla, traviata"
> compressableMimeType="text/html,text/xml,application/json"
> redirectPort="443" />
> ...........
> <Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> maxThreads="150" scheme="https" secure="true"
> clientAuth="false" sslEnabledProtocols="TLSv1.2"
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
> keystoreFile="conf/freestor" keystorePass="freestor"/>
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" redirectPort="443" secretRequired="false" />
You have lots of settings on the HTTP connector but not the HTTPS one.
Is that intentional? Which one are you using? Or both?
Are you using AJP? If not, disable the connetor entirely. Definitely
never enable it on 0.0.0.0 unless you really know what you are doing.
> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
> Different kernel versions count things differently.
> [YJ] I am just getting the process ID of tomcat and count open files for that process
Ok.
> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
> [YJ] Here is the netstat output:
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
> tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
> tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
That's not helpful. I want to see what the JVM process is holding, not
what the whole OS is handling.
My guess is that lsof is going to be a better bet for inspecting the
process. But if you get "too many open files" being thrown inside the
JVM and you don't see a 1/4 million lines of "lsof" output on your
process, then something else is going on. But remember to limit your
lsof to *just the JVM* process.
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Monday, May 17, 2021 5:18 PM
> To: users@tomcat.apache.org
> Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
>
> Yeggy,
>
> On 5/17/21 15:17, Yeggy Javadi wrote:
>> Hi,
>> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>
>> I was running the same application on Linux 7.6 with the same
>> apache-tomcat version without seeing this issue. I saw that RedHat has
>> removed apache-tomcat fom their Linux 8 distribution. Please help with
>> throubleshooting and indicating if there is any changes in Linux 8
>> that can explain this
>>
>> Configuration:
>> apache-tomcat-8.5.59
>> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
>> using Postgres and elastic search databases
>>
>>
>> # ps -ef | grep tomcat
>> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
>> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>>
>>
>> # cat /proc/51798/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 8388608 unlimited bytes
>> Max core file size unlimited unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 256518 256518 processes
>> Max open files 262144 262144 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 256518 256518 signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>>
>>
>> Too may open files:
>> # lsof -p 51798 | wc -l
>> 6679
>>
>>
>> Most of open files are TCP sockets:
>> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>>
>>
>> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
>> 6413
>
> Some potentially dumb questions:
>
> 1. Did you upgrade anything recently (like Java VM)?
>
> 2. What is error you are seeing? A full stack trace would be helpful.
>
> 3. What is your <Connector> configuration?
>
> 4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
> Different kernel versions count things differently.
>
> 5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Yeggy Javadi <Ye...@falconstor.com.INVALID>.
Hi Chris,
Thanks for your email. Below are replies to your questions:
1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too many open files) error when the limit of open files for tomcat has reached (262144 in my environment), for example, I can get a connectivity error when pulling info from a server as below :
[InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The XML configuration file failed to be retrieved for server 10.1.6.25. Check server connectivity.Error retrieving IPStorConfg for server=10.1.6.25 error=java.io.FileNotFoundException: /usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar (Too many open files) restatus=-1 output=
3. What is your <Connector> configuration?
[YJ] It is as below in server.xml:
<Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
maxThreads="150"
minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml,application/json"
redirectPort="443" />
...........
<Connector address="0.0.0.0" port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslEnabledProtocols="TLSv1.2"
ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
keystoreFile="conf/freestor" keystorePass="freestor"/>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" redirectPort="443" secretRequired="false" />
4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open files for that process
5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36850 ESTABLISHED
tcp 0 0 localhost.localdo:11753 localhost.localdo:51302 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36844 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48922 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48964 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48968 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60817 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48968 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48974 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60771 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48934 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48936 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48954 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48970 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48932 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48938 ESTABLISHED
tcp6 0 0 localhost.localdo:51302 localhost.localdo:11753 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48956 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48928 ESTABLISHED
tcp6 0 0 localhost.localdo:36844 localhost.loca:postgres ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48930 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 localhost.localdo:35202 localhost.localdo:vrace ESTABLISHED
tcp6 0 0 localhost.localdo:36850 localhost.loca:postgres ESTABLISHED
tcp6 0 0 localhost.localdo:vrace localhost.localdo:35202 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48966 ESTABLISHED
tcp6 0 0 localhost.localdo:51298 localhost.localdo:11753 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48954 ESTABLISHED
tcp6 0 1045 Yeggy-F8-FMSVA:54246 172.22.22.192:https ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48970 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48918 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48972 ESTABLISHED
tcp6 0 0 localhost.localdo:36846 localhost.loca:postgres ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48960 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48960 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48974 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60714 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48924 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48924 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48972 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48926 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48920 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60772 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:48976 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60864 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48916 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48920 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60818 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48976 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48964 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:36321 Yeggy-F8-FMSVA:35998 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48966 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60865 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48952 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:60713 TIME_WAIT
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48934 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48958 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48916 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48936 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48962 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48938 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:35998 Yeggy-F8-FMSVA:36321 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48918 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48962 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48926 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48922 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48928 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48930 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48952 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48940 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48958 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48956 ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:48940 Yeggy-F8-FMSVA:vrace ESTABLISHED
tcp6 0 0 Yeggy-F8-FMSVA:vrace Yeggy-F8-FMSVA:48932 ESTABLISHED
udp 0 0 localhost.localdo:41912 localhost.localdo:41912 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ] DGRAM 520 /var/run/chrony/chronyd.sock
unix 3 [ ] DGRAM 14708 /run/systemd/notify
unix 2 [ ] DGRAM 14710 /run/systemd/cgroups-agent
unix 12 [ ] DGRAM 14720 /run/systemd/journal/dev-log
unix 6 [ ] DGRAM 14734 /run/systemd/journal/socket
unix 2 [ ] DGRAM 31223 /run/user/0/systemd/notify
unix 2 [ ] STREAM CONNECTED 11157518
unix 2 [ ] STREAM CONNECTED 11152644
unix 3 [ ] STREAM CONNECTED 11657
unix 3 [ ] STREAM CONNECTED 22048 /var/lib/sss/pipes/private/sbus-monitor
unix 3 [ ] STREAM CONNECTED 545 /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 11974 /var/lib/sss/pipes/private/sbus-dp_implicit_files.946
unix 2 [ ] STREAM CONNECTED 11367562
unix 3 [ ] DGRAM 31226
unix 3 [ ] STREAM CONNECTED 22051 /var/lib/sss/pipes/private/sbus-dp_implicit_files.946
unix 3 [ ] STREAM CONNECTED 31229
unix 2 [ ] DGRAM 551
unix 3 [ ] STREAM CONNECTED 23869 /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 11879
unix 2 [ ] STREAM CONNECTED 800347
unix 2 [ ] DGRAM 23301
unix 3 [ ] STREAM CONNECTED 11819
unix 2 [ ] DGRAM 246
unix 3 [ ] STREAM CONNECTED 543
unix 3 [ ] STREAM CONNECTED 23109 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 11365631
unix 2 [ ] DGRAM 31213
unix 3 [ ] STREAM CONNECTED 11363838
unix 3 [ ] STREAM CONNECTED 12063
unix 3 [ ] STREAM CONNECTED 797101
unix 2 [ ] DGRAM 11659
unix 3 [ ] DGRAM 14712
unix 3 [ ] STREAM CONNECTED 23548
unix 3 [ ] STREAM CONNECTED 11880 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 23325
unix 3 [ ] STREAM CONNECTED 248
unix 3 [ ] STREAM CONNECTED 11973
unix 2 [ ] STREAM CONNECTED 32574
unix 3 [ ] STREAM CONNECTED 507
unix 3 [ ] STREAM CONNECTED 21947 /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 24095
unix 3 [ ] STREAM CONNECTED 18281 /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11001952
unix 2 [ ] DGRAM 11971
unix 3 [ ] STREAM CONNECTED 12134 /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11325964
unix 3 [ ] STREAM CONNECTED 23050
unix 2 [ ] STREAM CONNECTED 11365500
unix 3 [ ] STREAM CONNECTED 24096 /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11143790
unix 2 [ ] STREAM CONNECTED 793273
unix 3 [ ] STREAM CONNECTED 23868
unix 3 [ ] STREAM CONNECTED 11820 /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 11152638
unix 3 [ ] STREAM CONNECTED 24676 /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 23167
unix 3 [ ] STREAM CONNECTED 21912
unix 3 [ ] STREAM CONNECTED 543420
unix 3 [ ] DGRAM 31225
unix 3 [ ] STREAM CONNECTED 564
unix 3 [ ] STREAM CONNECTED 12133
unix 3 [ ] STREAM CONNECTED 12064 /var/lib/sss/pipes/private/sbus-monitor
unix 3 [ ] STREAM CONNECTED 18285 /run/systemd/journal/stdout
unix 2 [ ] STREAM CONNECTED 808539
unix 2 [ ] DGRAM 23542
unix 3 [ ] STREAM CONNECTED 22708
unix 2 [ ] DGRAM 11969
unix 3 [ ] STREAM CONNECTED 18279 /run/systemd/journal/stdout
unix 3 [ ] DGRAM 14713
unix 2 [ ] DGRAM 34975
unix 2 [ ] STREAM CONNECTED 11353853
unix 3 [ ] STREAM CONNECTED 11156644 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 21925
unix 2 [ ] DGRAM 22033
unix 3 [ ] STREAM CONNECTED 23168 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 24657
unix 3 [ ] STREAM CONNECTED 543421
unix 2 [ ] STREAM CONNECTED 793268
unix 3 [ ] STREAM CONNECTED 22047
unix 3 [ ] STREAM CONNECTED 782556 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 21490
unix 3 [ ] STREAM CONNECTED 23698
unix 2 [ ] DGRAM 31210
unix 3 [ ] STREAM CONNECTED 11141980
unix 2 [ ] STREAM CONNECTED 11344541
unix 2 [ ] STREAM CONNECTED 11353876
unix 2 [ ] STREAM CONNECTED 11367557
unix 3 [ ] STREAM CONNECTED 247
unix 3 [ ] STREAM CONNECTED 544
unix 3 [ ] STREAM CONNECTED 34094 /run/dbus/system_bus_socket
unix 2 [ ] STREAM CONNECTED 25346
unix 3 [ ] DGRAM 11664
unix 3 [ ] STREAM CONNECTED 21861
unix 3 [ ] STREAM CONNECTED 13539 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 565 /var/lib/sss/pipes/private/sbus-monitor
unix 2 [ ] DGRAM 24744
unix 3 [ ] STREAM CONNECTED 18283 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 13499
unix 2 [ ] STREAM CONNECTED 31209
unix 2 [ ] STREAM CONNECTED 32883
unix 3 [ ] DGRAM 11663
unix 3 [ ] STREAM CONNECTED 28573 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 561 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 11363839
unix 3 [ ] STREAM CONNECTED 11370377 /var/lib/sss/pipes/nss
unix 3 [ ] STREAM CONNECTED 22007
unix 2 [ ] DGRAM 28344
unix 2 [ ] DGRAM 23311
unix 3 [ ] STREAM CONNECTED 11932
unix 2 [ ] STREAM CONNECTED 11143887
unix 3 [ ] STREAM CONNECTED 31205
unix 2 [ ] DGRAM 32811
unix 2 [ ] DGRAM 513
Active Bluetooth connections (w/o servers)
Proto Destination Source State PSM DCID SCID IMTU OMTU Security
Proto Destination Source State Channel
Thanks
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/17/21 15:17, Yeggy Javadi wrote:
> Hi,
> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>
> I was running the same application on Linux 7.6 with the same
> apache-tomcat version without seeing this issue. I saw that RedHat has
> removed apache-tomcat fom their Linux 8 distribution. Please help with
> throubleshooting and indicating if there is any changes in Linux 8
> that can explain this
>
> Configuration:
> apache-tomcat-8.5.59
> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
> using Postgres and elastic search databases
>
>
> # ps -ef | grep tomcat
> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>
>
> # cat /proc/51798/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size unlimited unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 256518 256518 processes
> Max open files 262144 262144 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 256518 256518 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
>
> Too may open files:
> # lsof -p 51798 | wc -l
> 6679
>
>
> Most of open files are TCP sockets:
> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>
>
> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> 6413
Some potentially dumb questions:
1. Did you upgrade anything recently (like Java VM)?
2. What is error you are seeing? A full stack trace would be helpful.
3. What is your <Connector> configuration?
4. Are you counting all the open files for a single process or all of the sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
5. Running lsof, netstat, etc. can you see if some large number of those sockets are bound to any specific port (listen or connect)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Yeggy,
On 5/17/21 15:17, Yeggy Javadi wrote:
> Hi,
> I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>
> I was running the same application on Linux 7.6 with the same apache-tomcat version without seeing this issue. I saw that RedHat has remove apache-tomcat fom their Linux 8 distribution. Please help with throubleshooting and indicating if there is any changes in Linux 8 that can explain this
>
> Configuration:
> apache-tomcat-8.5.59
> OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64
> Web-based application using Postgres and elastic search databases
>
>
> # ps -ef | grep tomcat
> root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
> root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
>
>
> # cat /proc/51798/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size unlimited unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 256518 256518 processes
> Max open files 262144 262144 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 256518 256518 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
>
> Too may open files:
> # lsof -p 51798 | wc -l
> 6679
>
>
> Most of open files are TCP sockets:
> java 51798 root 6601u sock 0,9 0t0 3959342 protocol: TCPv6
>
>
> # lsof -p 51798 | grep "protocol: TCPv6" | wc -l
> 6413
Some potentially dumb questions:
1. Did you upgrade anything recently (like Java VM)?
2. What is error you are seeing? A full stack trace would be helpful.
3. What is your <Connector> configuration?
4. Are you counting all the open files for a single process or all of
the sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
5. Running lsof, netstat, etc. can you see if some large number of those
sockets are bound to any specific port (listen or connect)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org