You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Binay Paul (Jira)" <ji...@apache.org> on 2020/06/28 23:17:00 UTC
[jira] [Created] (SSHD-1025) "java.net.SocketException: Too many
open files" exception on long run of application with Apache Mina SSHD
2.1.0
Binay Paul created SSHD-1025:
--------------------------------
Summary: "java.net.SocketException: Too many open files" exception on long run of application with Apache Mina SSHD 2.1.0
Key: SSHD-1025
URL: https://issues.apache.org/jira/browse/SSHD-1025
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 2.1.0
Environment: Oracle Linux 7
Reporter: Binay Paul
This issue is observed during the performance testing of Oracle Communications Sessions Delivery Manager (OCSDM) release 8.2.1 with Apache Mina SSHD 2.1.0 version, after a long continuous run for about 15 to 16 hours of OCSDM with processing REST/SOAP API calls in a 3 node cluster setup.
Root Cause Analysis:
When OCSDMis running continuously with REST/SOAP consecutive requests, list of opened file descriptors count increases rapidly like some where around 23 Lakh and this leads to an exception "java.net.SocketException: Too many open files".
As a result of this DB read/write operations fails and OCSDM shutdown task is triggered by the application.
The lsof command shows FD named "PIPE" and "EventPoll" are been opened by SDM process in large number.
These FDs are opened by third party library Apache Mina which is used to make SSH/SFTP connections to 297 devices which are present in SDM (Used for SDM performance testing).
Walk around : As of now for OCSDM release 8.2.1 library Apache Mina is replaced with JSCH 0.1.55 which is been used in previous versions of SDM and application works as expected.
Console.out :
[Wed Nov 13 23:49:36 EST 2019] WARNING: RMI TCP Accept-1099: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=1099] throws
[Wed Nov 13 23:49:36 EST 2019] java.net.SocketException: Too many open files
[Wed Nov 13 23:49:36 EST 2019] at java.net.PlainSocketImpl.socketAccept(Native Method)
[Wed Nov 13 23:49:36 EST 2019] at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
[Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.implAccept(ServerSocket.java:545)
[Wed Nov 13 23:49:36 EST 2019] at java.net.ServerSocket.accept(ServerSocket.java:513)
[Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
[Wed Nov 13 23:49:36 EST 2019] at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
[Wed Nov 13 23:49:36 EST 2019] at java.lang.Thread.run(Thread.java:745)
BEServer.log :
11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutdown] Thread: [Thread-176494:334181] Msg:[##### BEServer shutdown was called ##### : Starting shutdown of BE server. ]
11/13 23:49:41.300 INFO [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer] Thread: [Thread-176494:334181] Msg:[##### WebServer shutdown was called ##### : Starting shutdown of http server]
11/13 23:49:41.300 ERROR [com.acmepacket.ems.server.main.BEServer] - Method: [shutDownApacheServer] Thread: [Thread-176494:334181] Msg:[Exception thrown in attempting to shutdown Web server. Reason : Cannot run program "../Apache/httpse
rver/bin/stopHttpServer.sh": error=24, Too many open files]
java.io.IOException: Cannot run program "../Apache/httpserver/bin/stopHttpServer.sh": error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_74]
at java.lang.Runtime.exec(Runtime.java:620) ~[?:1.8.0_74]
at java.lang.Runtime.exec(Runtime.java:450) ~[?:1.8.0_74]
at java.lang.Runtime.exec(Runtime.java:347) ~[?:1.8.0_74]
at com.acmepacket.ems.server.main.BEServer.shutDownApacheServer(BEServer.java:989) [ACMEPacketBEServer.jar:?]
at com.acmepacket.ems.server.main.BEServer.shutdownWebServers(BEServer.java:954) [ACMEPacketBEServer.jar:?]
at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:433) [ACMEPacketBEServer.jar:?]
at com.acmepacket.ems.server.main.BEServer.shutdown(BEServer.java:403) [ACMEPacketBEServer.jar:?]
at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.shutDownAfterFault(ReplicatedXMLDatabaseManagerImpl.java:2536) [ACMEPacketBEServer.jar:?]
at com.acmepacket.ems.server.services.database.ReplicatedXMLDatabaseManagerImpl.handlePanicEvent(ReplicatedXMLDatabaseManagerImpl.java:2568) [ACMEPacketBEServer.jar:?]
at com.sleepycat.db.internal.DbEnv.handle_panic_event_notify(DbEnv.java:115) [db.jar:?]
Caused by: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_74]
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[?:1.8.0_74]
at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_74]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_74]
... 10 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org