[jira] [Comment Edited] (SSHD-85) Port Forward closes connection before all bytes are sent

["Bill Kuker (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SSHD-85?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16055741#comment-16055741 ] 

Bill Kuker edited comment on SSHD-85 at 6/20/17 1:20 PM:
---------------------------------------------------------

[~lgoldstein] I understand your concern with the test code.

My understanding is that the close() call waits until all bytes have been handed off to the kernel and shuffled off onto the network before closing the connection. There is a small chance the final packet(s) of data could be lost, and since the sender's tcp stack has discarded the buffer and can not retransmit, data can be lost, but this is unlikely, SO_LINGER was introduced for exactly this. Most protocols implemented on TCP do something smart like Http's content-length header so the client KNOWS all the data has been received. I am specifically dealing with a dumb http 1.0 server via sshd.

I can say that in 2010 with similar code the bytes that go "missing" were already unambiguously present in a buffer in the SSHD server when SSHD closed the socket to the client. I strongly believe that this remains the case in 2017, and I am working towards a "smoking gun".

Regardless, I hope you will take a look at my re-written test server per the example, https://github.com/bkuker/mina-sshd/blob/6c462112c3a97d03b1edfeae463a1d7618ed98cb/sshd-core/src/test/java/org/apache/sshd/common/forward/PortForwardingServerCloseTest.java#L119

On the test server end I am pretty sure that by waiting for write() to complete before I close() we can agree that all the data has been written by the server before the connection closes. This is necessary but not yet sufficient to prove that the kind of write/close race you mention is taking place within the SSHD server.





was (Author: bkuker1):
@lgoldstein I understand your concern with the test code.

My understanding is that the close() call waits until all bytes have been handed off to the kernel and shuffled off onto the network before closing the connection. There is a small chance the final packet(s) of data could be lost, and since the sender's tcp stack has discarded the buffer and can not retransmit, data can be lost, but this is unlikely, SO_LINGER was introduced for exactly this. Most protocols implemented on TCP do something smart like Http's content-length header so the client KNOWS all the data has been received. I am specifically dealing with a dumb http 1.0 server via sshd.

I can say that in 2010 with similar code the bytes that go "missing" were already unambiguously present in a buffer in the SSHD server when SSHD closed the socket to the client. I strongly believe that this remains the case in 2017, and I am working towards a "smoking gun".

Regardless, I hope you will take a look at my re-written test server per the example, https://github.com/bkuker/mina-sshd/blob/6c462112c3a97d03b1edfeae463a1d7618ed98cb/sshd-core/src/test/java/org/apache/sshd/common/forward/PortForwardingServerCloseTest.java#L119

On the test server end I am pretty sure that by waiting for write() to complete before I close() we can agree that all the data has been written by the server before the connection closes. This is necessary but not yet sufficient to prove that the kind of write/close race you mention is taking place within the SSHD server.




> Port Forward closes connection before all bytes are sent
> --------------------------------------------------------
>
>                 Key: SSHD-85
>                 URL: https://issues.apache.org/jira/browse/SSHD-85
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 0.2.0, 1.4.0, 1.6.0
>         Environment: Windows or Linux, JSCH Client
>            Reporter: Bill Kuker
>            Assignee: Guillaume Nodet
>         Attachments: MINA-Port-Forward.txt, SSHD-85_testcase.txt
>
>
> When I am forwarding connections from a Client to a port on the SSH Server through the SSH Client to a Remote Server on the SSH Client network:
> Client --(a)---> SSH Server <===[ssh]===> SSH Client --(b)--> Remote Server
>  if the Remote Server sends some bytes and immediately closes the connection (b) to the SSH Client the SSH Server will close the connection (a) to the client before all of those bytes are sent to the SSH Client.
> This is probably a race condition. If I run one connection at a time it works 99% of the time, but if I use a load generator (a completely separate process) to max out all 4 cores I can make it fail (I do not get all the bytes from Remote Server at Client) 99% of the time. Also by running about 100 simultaneous requests I get maybe a 20% failure rate.
> If the Remote Server is something like Telnet (Open for a while, user closes connection) everything seems fine, but the the Remote Server is a web server with keep alive disabled, and many small requests are made, many images, htmls, css are truncated.
> Sorry I can't be more precise with this, but I hope anyone else having a similar trouble can help fill out this bug with details.
> *Update in 2017*
> I have provided a maven project with a unit test that has a 100% failure rate on my desktop without resorting to creating artificial load. It is a lot simpler:
> https://github.com/bkuker/sshd-85
> https://github.com/bkuker/sshd-85/blob/master/src/main/java/com/billkuker/sshd/sshd85/TheTest.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)