[jira] [Commented] (SSHD-921) OutOfMemory Error in extreme situations

["Goldstein Lyor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SSHD-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856332#comment-16856332 ] 

Goldstein Lyor commented on SSHD-921:
-------------------------------------

I am not sure that the "trick" you played on the code really helps simulate the problem since it may introduce a synthetic error:

The code detects an authentication timeout and attempts to send an {{SSH_MSG_DISCONNECT}} message. It attaches a {{Future}} listener to close the session once message is sent. However, the message is never sent because of the {{writeData}} change. It is however appended to the receive buffer.

This occurs every 1-2 seconds since the session is still "active" as far as the thread that checks the timeouts is concerned - so it attempts to send another {{SSH_MSG_DISCONNECT}} (and another, and another....) - as the log clearly seems to indicate. These messages keep accumulating in the receive buffer, but no one is sending them. Also, the peer does not know that the session is closed so it does not disconnect - so OOM is likely.

That being said, I will keep looking at the code since the scenario I just described might not explain 100% the behavior you experienced - there are a few points of interest that I need to research further.


> OutOfMemory Error in extreme situations
> ---------------------------------------
>
>                 Key: SSHD-921
>                 URL: https://issues.apache.org/jira/browse/SSHD-921
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 1.7.0, 2.0.0
>            Reporter: Zhenliang Su
>            Assignee: Goldstein Lyor
>            Priority: Major
>         Attachments: Main.java, jstack.txt
>
>
> The demo code run in my local machine.
>  When I 'ssh root@127.0.0.1 -p 2233', my demo will use the agent forwarding feature to login 10.10.16.201.
>  In most cases, it's ok. But in few cases, for example, client disconnect suddenly, it will lead to OutOfMemory Error.
>  To simulate this scenario of OutOfMemory Error, I did some trick to the sshd code:
>  1. In org.apache.sshd.common.future.DefaultSshFuture#setValue:
> {code:java}
> public void setValue(Object newValue) {
>     	// to simulate race condition
>     	try {
>     		Thread.sleep(10);
>     	} catch (Exception e) {
>     		// ignore
>     	}
>         synchronized (lock) {
>             // Allow only once.
>             if (result != null) {
>                 return;
>             }
>             result = (newValue != null) ? newValue : GenericUtils.NULL;
>             lock.notifyAll();
>         }
>         notifyListeners();
>     }
> {code}
> 2. In org.apache.sshd.agent.local.AgentForwardedChannel#doWriteData:
> {code:java}
>  protected void doWriteData(byte[] data, int off, long len) throws IOException {
>         ValidateUtils.checkTrue(len <= Integer.MAX_VALUE, "Data length exceeds int boundaries: %d", len);
>         Buffer message = null;
>         synchronized (receiveBuffer) {
>             receiveBuffer.putBuffer(new ByteArrayBuffer(data, off, (int) len));
>             // to simulate client disconnect suddenly
>             if (false && receiveBuffer.available() >= 4) {
>                 off = receiveBuffer.rpos();
>                 len = receiveBuffer.getInt();
>                 receiveBuffer.rpos(off);
>                 if (receiveBuffer.available() >= (4 + len)) {
>                     message = new ByteArrayBuffer(receiveBuffer.getBytes());
>                     receiveBuffer.compact();
>                 }
>             }
>         }
>         if (message != null) {
>             synchronized (messages) {
>                 messages.offer(message);
>                 messages.notifyAll();
>             }
>         }
>     }
> {code}
> After two munites, my console displayed '[sshd-SshClient[720b3c53]-timer-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Disconnecting(ClientSessionImpl[root@/10.10.16.201:8022]): SSH2_DISCONNECT_PROTOCOL_ERROR - Session has timed out waiting for authentication after 120000 ms.' every second, and finally lead to OutOfMemory Error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org