svn commit: r891122 - /mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java

[sp...@apache.org]

Author: spearce
Date: Wed Dec 16 04:20:04 2009
New Revision: 891122

URL: http://svn.apache.org/viewvc?rev=891122&view=rev
Log:
SSHD-54: Use OS default receive buffer to improve throughput

MINA itself is actually forcing the receive buffer size to be 1024
bytes, rather than the operating system default.  By default the
NioAcceptor is created with a DefaultSocketSessionConfig [1] which
sets both the send and receive buffer sizes to 1024 bytes.

NioSocketAcceptor [2] then turns around and sets this receive
buffer size into the accept socket, so all child sockets created
for incoming connections inherit this low default value.

However, in the case of the send buffer, because the configured
value of 1024 matches the default of 1024 for the send buffer,
doSetAll [3] never actually changes the send buffer size away from
the operating system default.

Unfortunately this means we have the following situation for our
NIO based sockets:

       MINA  | OS Default
      +------+-----------
  RCV | 1024 | 43690
  SND | 8192 |  8192

So the receive buffer size under MINA is 42x smaller than the OS
default.  But the send buffer size is the OS default, as MINA does
not directly set the send buffer size itself.  So send throughput
is actually reasonable, but receive throughput is horrible.

There is a good article [4] which explains how these settings impact
connection throughput.  If we want any decent TCP/IP throughput we
need reasonably large buffers to increase the amount of data one
side can send before stopping and waiting for an ACK from the peer.
If our receive window is only 1024 bytes, our peer can't even send
a full TCP/IP packet on Ethernet without stopping and waiting for
an ACK from us.

Of course one could try to argue that the MINA default of 1024 bytes
for the receive buffer is suitable, especially on a server with a
very large number of idle connections, or connections where a human
is typing all of the input data. But it is not suitable for a high
traffic data receiver, like the scp command.

Unfortunately asking applications to raise the receive buffer size
on each individual connection through the SessionFactory doesn't
work on all platforms.  On Linux with OpenJDK 6 the following
application code appears to work:

  setSessionFactory(new SessionFactory() {
      @Override
      protected ServerSession createSession(final IoSession io)
          throws Exception {
          final SocketSessionConfig c = (SocketSessionConfig) io.getConfig();
          c.setKeepAlive(keepAlive);
          c.setReceiveBufferSize(43690);
        }
    }
  });

but the aggregate throughput is still horrible (3 MB/s).  So the
receive buffer didn't actually increase with the peer.

Linux seems to be capping the connection at the upper bound when
the connection starts.  If we set the receive buffer size of the
NioSocketAcceptor to 43690, later drop the receive buffer size of
the individual connection to 1024 in the session factory, we can
later boost it when a specific command starts, and performance
behaves as expected.

To match developer expectations we now reset the receive buffer
size to the operating system's default size before the server
starts accepting connections.  This allows applications to get good
throughput by default, or raise/lower the receive buffer within
reasonable limits based on the command executed.

References:
  [1] http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/DefaultSocketSessionConfig.java?view=markup#l44
  [2] http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/nio/NioSocketAcceptor.java?view=markup#l248
  [3] http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/AbstractSocketSessionConfig.java?view=markup#l58
  [4] http://www.ibm.com/developerworks/linux/library/l-hisock.html

Modified:
    mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java

Modified: mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java
URL: http://svn.apache.org/viewvc/mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java?rev=891122&r1=891121&r2=891122&view=diff
==============================================================================
--- mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java (original)
+++ mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java Wed Dec 16 04:20:04 2009
@@ -21,6 +21,7 @@
 import java.io.File;
 import java.io.IOException;
 import java.net.InetSocketAddress;
+import java.net.Socket;
 import java.security.InvalidKeyException;
 import java.security.PublicKey;
 import java.util.ArrayList;
@@ -31,6 +32,9 @@
 import java.util.List;
 import java.util.concurrent.CountDownLatch;
 
+import org.slf4j.LoggerFactory;
+import org.slf4j.Logger;
+
 import org.apache.mina.core.service.IoAcceptor;
 import org.apache.mina.core.session.IoSession;
 import org.apache.mina.core.session.IoSessionConfig;
@@ -107,6 +111,8 @@
  */
 public class SshServer extends AbstractFactoryManager implements ServerFactoryManager {
 
+    private final Logger log = LoggerFactory.getLogger(getClass());
+
     protected IoAcceptor acceptor;
     protected String host;
     protected int port;
@@ -326,8 +332,26 @@
 
     protected void configure(IoAcceptor acceptor) {
         if (acceptor instanceof NioSocketAcceptor) {
-            ((NioSocketAcceptor) acceptor).setReuseAddress(reuseAddress);
-            ((NioSocketAcceptor) acceptor).setBacklog(backlog);
+            final NioSocketAcceptor nio = (NioSocketAcceptor) acceptor;
+            nio.setReuseAddress(reuseAddress);
+            nio.setBacklog(backlog);
+
+            // MINA itself forces our socket receive buffer to 1024 bytes
+            // by default, despite what the operating system defaults to.
+            // This limits us to about 3 MB/s incoming data transfer.  By
+            // forcing back to the operating system default we can get a
+            // decent transfer rate again.
+            //
+            final Socket s = new Socket();
+            try {
+              try {
+                  nio.getSessionConfig().setReceiveBufferSize(s.getReceiveBufferSize());
+              } finally {
+                  s.close();
+              }
+            } catch (IOException e) {
+                log.warn("cannot adjust SO_RCVBUF back to system default", e);
+            }
         }
         if (sessionConfig != null) {
             acceptor.getSessionConfig().setAll(sessionConfig);