You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Thomas Wolf (Jira)" <ji...@apache.org> on 2022/06/08 20:32:00 UTC
[jira] [Comment Edited] (SSHD-1055) Remote port forwarding mode does not handle EOF properly
[ https://issues.apache.org/jira/browse/SSHD-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551764#comment-17551764 ]
Thomas Wolf edited comment on SSHD-1055 at 6/8/22 8:31 PM:
-----------------------------------------------------------
This is still reproducible on current master, using OpenSSH 7.9. Though I have to start the SSH client (with remote port forwarding) as
{code:java}
ssh -o 'ExitOnForwardFailure yes' -o 'StrictHostKeyChecking off' -vvv -p 12133 -f -x -N -T -R 127.0.0.1:4567:127.0.0.1:23645 test5@127.0.0.1 {code}
and then
{code:java}
curl 127.0.0.1:4567 {code}
to get the example to work.
The hang is indeed caused by not shutting down the output when an EOF is received in {{{}TcpipClientChannel{}}}. If the code removed in [https://github.com/apache/mina-sshd/commit/385f63731e74d0c73b6624170bfd1ad385ebffc6] is re-instatiated, behavior is as expected.
I see the gRPC server logging:
{code:java}
20:50:30.079 [grpc-nio-worker-ELG-3-1] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler - [id: 0xe7c489a5, L:/127.0.0.1:23645 - R:/127.0.0.1:58972] OUTBOUND GO_AWAY: lastStreamId=0 errorCode=1 length=35 bytes=556e657870656374656420485454502f312e7820726571756573743a20474554202f20
20:50:30.081 [grpc-nio-worker-ELG-3-1] DEBUG io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler - [id: 0xe7c489a5, L:/127.0.0.1:23645 - R:/127.0.0.1:58972] Sent GOAWAY: lastStreamId '0', errorCode '1', debugData 'Unexpected HTTP/1.x request: GET / '. Forcing shutdown of the connection. {code}
In response to that, the OpenSSH client sends the EOF, which in the Apache MINA sshd server's {{TcpipClientChannel}} only sets a flag. It seems to me propagating this EOF to whatever is connected on the socket on that end ({{{}curl{}}} in the example case) by calling {{serverSession.shutdownOutputStream()}} might not be totally wrong. (But see below: should we get not only an EOF but also a CLOSE?)
Note that if {{TcpipClientChannel}} needs to have this {{shutDownOutputStream()}} re-added, then probably {{TcpipServerChannel}} also needs to do so to handle the inverse case properly.
The removal of this code in SSHD-964 was accompanied by much uncertainty, and unfortunately SSHD-964 doesn't explain why it was removed. The issue itself was about EOF not being sent... why exactly was it removed?? (Or phrased differently: what breaks if it is re-added?) (It was originally added in SSHD-902.)
OTOH: can someone try this example with a modern OpenSSH (9.0)? Maybe OpenSSH should not only send an EOF but also a SSH_MSG_CHANNEL_CLOSE? That would be a somewhat more logical moment to shut down the socket output.
was (Author: wolft):
This is still reproducible on current master, using OpenSSH 7.9. Though I have to start the SSH client (with remote port forwarding) as
{code:java}
ssh -o 'ExitOnForwardFailure yes' -o 'StrictHostKeyChecking off' -vvv -p 12133 -f -x -N -T -R 127.0.0.1:4567:127.0.0.1:23645 test5@127.0.0.1 {code}
and then
{code:java}
curl 127.0.0.1:4567 {code}
to get the example to work.
The hang is indeed caused by not shutting down the output when an EOF is received in {{{}TcpipClientChannel{}}}. If the code removed in [https://github.com/apache/mina-sshd/commit/385f63731e74d0c73b6624170bfd1ad385ebffc6] is re-instatiated, behavior is as expected.
I see the gRPC server logging:
{code:java}
20:50:30.079 [grpc-nio-worker-ELG-3-1] DEBUG io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler - [id: 0xe7c489a5, L:/127.0.0.1:23645 - R:/127.0.0.1:58972] OUTBOUND GO_AWAY: lastStreamId=0 errorCode=1 length=35 bytes=556e657870656374656420485454502f312e7820726571756573743a20474554202f20
20:50:30.081 [grpc-nio-worker-ELG-3-1] DEBUG io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler - [id: 0xe7c489a5, L:/127.0.0.1:23645 - R:/127.0.0.1:58972] Sent GOAWAY: lastStreamId '0', errorCode '1', debugData 'Unexpected HTTP/1.x request: GET / '. Forcing shutdown of the connection. {code}
In response to that, the OpenSSH client sends the EOF, which in the Apache MINA sshd server's {{TcpipClientChannel}} only sets a flag. It seems to me propagating this EOF to whatever is connected on the socket on that end ({{{}curl{}}} in the example case) by calling {{serverSession.shutdownOutputStream()}} might not be totally wrong. (But see below: should we get not only an EOF but also a CLOSE?)
Note that if {{TcpipClientChannel}} needs to have this {{shutDownOutputStream()}} re-added, then probably {{TcpipServerChannel}} also needs to do so to handle the inverse case properly.
The removal of this code in SSHD-964 was accompanied by much uncertainty, and unfortunately SSHD-964 doesn't explain why it was removed. The issue itself was about EOF not being sent... why exactly was it removed?? (Or phrased differently: what breaks if it is re-added?)
OTOH: can someone try this example with a modern OpenSSH (9.0)? Maybe OpenSSH should not only send an EOF but also a SSH_MSG_CHANNEL_CLOSE? That would be a somewhat more logical moment to shut down the socket output.
> Remote port forwarding mode does not handle EOF properly
> --------------------------------------------------------
>
> Key: SSHD-1055
> URL: https://issues.apache.org/jira/browse/SSHD-1055
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.5.1
> Reporter: Feng Jiajie
> Priority: Major
>
> I want to call the remote server's gRPC service locally through an SSH tunnel.
> MyApp -> MINA SSHD -> \{Internet} -> gRPC Server
> It works just fine with OpenSSH, but there is a small problem(no problems with core functions, only in unusual circumstances) with Mina SSHD.
> I think the problem is Mina SSHD's handling of EOF.
> Here is the example:
> Step 1. Start a gRPC server:
> Because we only need a gRPC server to reproduce the problem, so I write a simple version without any service:
> {code:java}
> <dependency>
> <groupId>io.grpc</groupId>
> <artifactId>grpc-netty-shaded</artifactId>
> <version>1.27.2</version>
> </dependency>
> <dependency>
> <groupId>io.grpc</groupId>
> <artifactId>grpc-protobuf</artifactId>
> <version>1.27.2</version>
> </dependency>
> <dependency>
> <groupId>io.grpc</groupId>
> <artifactId>grpc-stub</artifactId>
> <version>1.27.2</version>
> </dependency>
> {code}
> main:
> {code:java}
> import io.grpc.Server;
> import io.grpc.ServerBuilder;
> public class EmptyGrpcServer {
> public static void main(String[] args) throws Exception {
> Server server = ServerBuilder.forPort(23645).build().start();
> server.awaitTermination();
> }
> }
> {code}
> Full example can be fond here: [https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java]
> Step 2. Start a MINA SSHD server:
> {code:java}
> import org.apache.sshd.server.SshServer;
> import org.apache.sshd.server.forward.AcceptAllForwardingFilter;
> import org.apache.sshd.server.keyprovider.SimpleGeneratorHostKeyProvider;
> import java.nio.file.Paths;
> public class Example1 {
> public static void main(String[] args) throws Exception {
> SshServer sshd = SshServer.setUpDefaultServer();
> sshd.setPort(12133);
> sshd.setKeyPairProvider(new SimpleGeneratorHostKeyProvider(Paths.get("/tmp/a.ser")));
> sshd.setPasswordAuthenticator((username, password, session) -> true);
> sshd.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
> sshd.start();
> Thread.sleep(10000000);
> }
> }
> {code}
> Step 3. Create a channel using ssh client
> {code:java}
> ssh -o 'ExitOnForwardFailure yes' -vvv -p 12133 -f -x -N -T -R 0.0.0.0:0:127.0.0.1:23645 test5@127.0.0.1
> {code}
> Step 4. Reproduce
> If I connect directly to the gRPC server using curl, cause gRPC using http/2, I would get error output like this:
> {code:java}
> $ curl 127.0.0.1:23645
> ���+Unexpected HTTP/1.x request: GET /
> $
> {code}
> Then if I do step 3 with an OpenSSH server, I would get same error output:
> {code:java}
> $ ssh -o 'ExitOnForwardFailure yes' -f -x -N -T -R 0.0.0.0:0:127.0.0.1:23645 work@dev.kbyte.cn
> Allocated port 13525 for remote forward to 127.0.0.1:23645
> $
> $ curl dev.kbyte.cn:13525
> ���+Unexpected HTTP/1.x request: GET /
> $
> {code}
> But when I do step 3 with MINA SSHD, curl would stuck without any output:
> {code:java}
> $ curl 127.0.0.1:55604
> {code}
> I found MINA SSHD had already got and wrote the package with the string "Unexpected.HTTP/1.x.request:.GET", and received SSH_MSG_CHANNEL_EOF.
> So I think handleEof should do more? like send SSH_MSG_CHANNEL_EOF to curl?
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org