You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "pengyue-hw (via GitHub)" <gi...@apache.org> on 2023/03/01 03:55:41 UTC
[GitHub] [mina-sshd] pengyue-hw opened a new issue, #323: Threads like sshd-SshClient[35a0cde2]-nio2-thread-1 lead to memory leak
pengyue-hw opened a new issue, #323:
URL: https://github.com/apache/mina-sshd/issues/323
### Version
2.9.2 and hitstory version
### Bug description
I have read #270 Issue, but it doesn't help me
In my project, I use `SshClient` and `ClientChanne`l to create an ssh client connection. After connecting to the server, the connection thread will simply operate the `pwd` command, and then disconnect and terminate the thread through the `close` method. This process simulates the process of continuous users connecting to the server through the ssh client. I found that for each connection, mina-sshd will create 6 threads similar to `"sshd-SshClient[35a0cde2]-nio2-thread-1"`. After calling the close method, these threads will not disappear and remain in the `WAITING (parking)` state.
### Actual behavior
As the number of connections increases, thousands of `"sshd-SshClient[35a0cde2]-nio2-thread-1"` will eventually lead to memory leaks, making it impossible to create new connection threads and causing the process to crash. In my test, I allocated 1GB of memory to the process, and the exception `"java.lang.OutOfMemoryError: unable to create new native thread"` occurred after nearly 2000 connections
### Expected behavior
Can such thread be terminated after closing the connection? Otherwise, if the client connects 2000 times, the memory will overflow and the program will crash. The reliability of such a program is too poor.
### Relevant log output
```Shell
Exception in thread "Thread-3979" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at sun.nio.ch.AsynchronousChannelGroupImpl$2.run(AsynchronousChannelGroupImpl.java:123)
at sun.nio.ch.AsynchronousChannelGroupImpl$2.run(AsynchronousChannelGroupImpl.java:118)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.AsynchronousChannelGroupImpl.startInternalThread(AsynchronousChannelGroupImpl.java:118)
at sun.nio.ch.AsynchronousChannelGroupImpl.threadExit(AsynchronousChannelGroupImpl.java:164)
at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:302)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "ConnThread1988" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at sun.nio.ch.AsynchronousChannelGroupImpl$2.run(AsynchronousChannelGroupImpl.java:123)
at sun.nio.ch.AsynchronousChannelGroupImpl$2.run(AsynchronousChannelGroupImpl.java:118)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.AsynchronousChannelGroupImpl.startInternalThread(AsynchronousChannelGroupImpl.java:118)
at sun.nio.ch.AsynchronousChannelGroupImpl.startThreads(AsynchronousChannelGroupImpl.java:132)
at sun.nio.ch.EPollPort.start(EPollPort.java:113)
at sun.nio.ch.LinuxAsynchronousChannelProvider.openAsynchronousChannelGroup(LinuxAsynchronousChannelProvider.java:64)
at java.nio.channels.AsynchronousChannelGroup.withThreadPool(AsynchronousChannelGroup.java:273)
at org.apache.sshd.common.io.nio2.Nio2ServiceFactory.<init>(Nio2ServiceFactory.java:45)
at org.apache.sshd.common.io.nio2.Nio2ServiceFactoryFactory.create(Nio2ServiceFactoryFactory.java:50)
at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.create(DefaultIoServiceFactoryFactory.java:53)
at org.apache.sshd.common.helpers.AbstractFactoryManager.getIoServiceFactory(AbstractFactoryManager.java:113)
at org.apache.sshd.client.SshClient.createConnector(SshClient.java:814)
at org.apache.sshd.client.SshClient.start(SshClient.java:456)
at com.wnt.connection.terminal.SshdConnTerminal.login(SshdConnTerminal.java:56)
at com.wnt.connection.ConnTerminalFactory$ConnThread.run(ConnTerminalFactory.java:37)
Exception in thread "ConnThread1989" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1587)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:573)
at org.apache.sshd.common.helpers.AbstractFactoryManager.setupSessionTimeout(AbstractFactoryManager.java:480)
at org.apache.sshd.client.SshClient.start(SshClient.java:454)
at com.wnt.connection.terminal.SshdConnTerminal.login(SshdConnTerminal.java:56)
at com.wnt.connection.ConnTerminalFactory$ConnThread.run(ConnTerminalFactory.java:37)
```
### Other information
The following is my main source code:
ConnTerminalFactory.java Create 5000 ssh connections, 1 per second on average
```
import com.wnt.connection.model.ConnParams;
import com.wnt.connection.terminal.SshdConnTerminal;
public class ConnTerminalFactory {
private final static int SSH_CONN = 2;
private final static String SERVER_IP = "1.1.1.1";
private final static int SERVER_PORT = 22;
private final static String USERNAME = "root";
private final static String PASSWD = "root";
public static void main(String[] args) throws Exception {
for (int i = 0; i < 5000; i++) {
Thread.sleep(1000);
ConnParams params = new ConnParams(SSH_CONN, SERVER_IP, SERVER_PORT, USERNAME, PASSWD);
ConnThread connThread = new ConnThread(params);
connThread.setName("ConnThread" + i);
connThread.start();
}
System.out.println("Thread over--------------------------------");
Thread.sleep(10000000);
}
private static class ConnThread extends Thread {
private ConnParams params;
public ConnThread(ConnParams params) {
this.params = params;
}
@Override
public void run() {
try {
SshdConnTerminal it = new SshdConnTerminal(params);
it.login();
it.execCmd("pwd");
System.out.println(this.getName() + ",exec pwd result:" + it.readResponse());
sleep(1000);
it.logout();
System.out.println(this.getName() + ",logout");
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
```
SshdConnTerminal.java Terminal operations such as login, execute command, and logout
```
import com.wnt.connection.ConnTerminalUtil;
import com.wnt.connection.model.CmdResponse;
import com.wnt.connection.model.ConnParams;
import com.wnt.connection.model.TerminalStream;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.sshd.client.ClientBuilder;
import org.apache.sshd.client.SshClient;
import org.apache.sshd.client.channel.ClientChannel;
import org.apache.sshd.client.session.ClientSession;
import org.apache.sshd.common.NamedFactory;
import org.apache.sshd.common.SshException;
import org.apache.sshd.common.channel.PtyChannelConfiguration;
import org.apache.sshd.common.kex.BuiltinDHFactories;
import org.apache.sshd.common.signature.BuiltinSignatures;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.ArrayList;
public class SshdConnTerminal implements IConnTerminal {
private static final Logger LOGGER = LoggerFactory.getLogger(SshdConnTerminal.class);
private final ConnParams params;
private ClientSession session;
private OutputStream out;
private PrintStream pOut;
InputStream in;
ClientChannel channel;
public SshdConnTerminal(ConnParams params) {
this.params = params;
}
@Override
public TerminalStream login() {
TerminalStream ts = new TerminalStream();
try {
SshClient client = SshClient.setUpDefaultClient();
// 解决server使用diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1算法,Client端默认算法不支持导致key exchange失败的问题
client.setKeyExchangeFactories(NamedFactory.setUpTransformedFactories(
false,
BuiltinDHFactories.VALUES,
ClientBuilder.DH2KEX
));
client.setSignatureFactories(new ArrayList<>(BuiltinSignatures.VALUES));
client.start();
session = client.connect(params.getUsername(), params.getAddress(), params.getPort()).verify(IConnTerminal.CONN_TIMEOUT).getSession();
session.addPasswordIdentity(params.getPassword());
session.auth().verify(IConnTerminal.CONN_TIMEOUT);
// 设置终端类型为xterms,避免部分指令不识别终端类型
PtyChannelConfiguration ptyConfig = new PtyChannelConfiguration();
ptyConfig.setPtyType(params.getTerminalType());
channel = session.createShellChannel(ptyConfig, null);
channel = session.createShellChannel();
channel.open().verify(IConnTerminal.CONN_TIMEOUT);
out = channel.getInvertedIn();
pOut = new PrintStream(out);
in = channel.getInvertedOut();
ts.fillContent(in, TerminalStream.CODE_SUCCESS, true, IConnTerminal.SUCCESS_LOGIN_MSG);
} catch (Exception e) {
LOGGER.error("error when ssh login.params:{}", params, e);
ts.fillContent(null, processSshdException(e), false, "Login failed." + e.getMessage());
}
return ts;
}
private int processSshdException(Exception e) {
if (!(e instanceof SshException)) {
return TerminalStream.CODE_FAIL_OTHER;
}
String msg = e.getMessage();
int disconnectCode = ((SshException) e).getDisconnectCode();
if (StringUtils.containsIgnoreCase(msg, "信号灯超时时间已到") || StringUtils.containsIgnoreCase(msg, "Failed to get operation result within specified timeout")) {
return TerminalStream.CODE_DST_UNREACHABLE;
} else if (StringUtils.containsIgnoreCase(msg, "远程计算机拒绝网络连接") || StringUtils.containsIgnoreCase(msg, "The remote computer refused the network connection")) {
return TerminalStream.CODE_CONN_REFUSED;
} else if (disconnectCode == 14) {
return TerminalStream.CODE_AUTH_FAIL;
}
return TerminalStream.CODE_FAIL_OTHER;
}
@Override
public void rowWrite(byte[] buf) throws IOException {
ConnTerminalUtil.rowWrite(buf, pOut);
}
@Override
public String readResponse() throws IOException {
return ConnTerminalUtil.readResponse(in);
}
@Override
public void execCmd(String cmd) {
ConnTerminalUtil.sendCommand(cmd, pOut);
}
@Override
public CmdResponse execCmdWithResponse(String cmd) {
return ConnTerminalUtil.sendCommandWithResponse(cmd, pOut, in);
}
@Override
public void logout() {
try {
IOUtils.close(in);
IOUtils.close(out);
if (channel != null && channel.isOpen()) {
channel.close(true);
}
if (session != null && session.isOpen()) {
session.close(true);
}
} catch (Exception e) {
LOGGER.error("error when ssh logout.params:{}", params, e);
} finally {
session = null;
channel = null;
}
}
}
```
ConnTerminalUtil.java Tool class for reading and writing ssh input and output streams
```
import com.wnt.connection.model.CmdResponse;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.PrintStream;
import java.nio.charset.StandardCharsets;
public class ConnTerminalUtil {
private static final Logger logger = LoggerFactory.getLogger(ConnTerminalUtil.class);
public static final int READ_WAIT_COUNT = 5;
public static final int READ_WAIT_DURATION_MS = 600;
public static void sleep(int interval) {
try {
Thread.sleep(interval);
} catch (Exception e) {
logger.error("error when sleep.", e);
}
}
public static String readResponse(InputStream in) throws IOException {
int readable = 0;
int last = 0;
int waitCount = 0;
while (waitCount < READ_WAIT_COUNT || readable == 0) {
sleep(READ_WAIT_DURATION_MS);
readable = in.available();
if (readable == last) {
waitCount++;
} else {
waitCount = 0;
last = readable;
}
if (waitCount >= 20 || readable > 1024) {
break;
}
}
byte[] buf = new byte[readable];
in.read(buf);
return new String(buf, StandardCharsets.UTF_8);
}
public static CmdResponse sendCommandWithResponse(String command, PrintStream out, InputStream in) {
CmdResponse response = new CmdResponse();
try {
sendCommand(command, out);
response.setStdout(readResponse(in));
} catch (Exception e) {
response.setReturnCode(-1);
response.setStdout("execCmd error, cmd:" + command);
response.setStderr(e.getMessage());
}
return response;
}
public static void rowWrite(byte[] buf, OutputStream out) throws IOException {
out.write(buf);
out.flush();
}
public static void sendCommand(String command, PrintStream out) {
out.println(command);
out.flush();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] pengyue-hw closed issue #323: Threads like sshd-SshClient[35a0cde2]-nio2-thread-1 lead to memory leak
Posted by "pengyue-hw (via GitHub)" <gi...@apache.org>.
pengyue-hw closed issue #323: Threads like sshd-SshClient[35a0cde2]-nio2-thread-1 lead to memory leak
URL: https://github.com/apache/mina-sshd/issues/323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] pengyue-hw commented on issue #323: Threads like sshd-SshClient[35a0cde2]-nio2-thread-1 lead to memory leak
Posted by "pengyue-hw (via GitHub)" <gi...@apache.org>.
pengyue-hw commented on issue #323:
URL: https://github.com/apache/mina-sshd/issues/323#issuecomment-1455786270
I tried to close the SSH Client connection normally through `client.stop();` and the corresponding `"sshd-SshClient[35a0cde2]-nio2-thread-1"` thread was also closed normally.
I confirmed that it was caused by my improper use。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org