[jira] [Created] (HDFS-9861) Add SO_REUSEPORT into HDFS

["Yingqi Lu (JIRA)" <ji...@apache.org>]

Yingqi Lu created HDFS-9861:
-------------------------------

             Summary: Add SO_REUSEPORT into HDFS
                 Key: HDFS-9861
                 URL: https://issues.apache.org/jira/browse/HDFS-9861
             Project: Hadoop HDFS
          Issue Type: New Feature
            Reporter: Yingqi Lu


Hello All,

SO_REUSEPORT is a kernel feature which enables multiple listen sockets to listen to the same IP:PORT pair. This can potentially improve the scalability and parallelism of network traffic handling, especially for big core count systems or large sized clusters. The feature is enabled for both TCP and UDP sockets. Linux, BSD and majority of other UNIX based OSes already have the feature supported. Windows has not enabled it yet. For more details on the feature itself, please refer to https://lwn.net/Articles/542629/. 

Recently, SO_REUSEPORT is added into OpenJDK9 as one of the socket options. For more details, please refer to http://hg.openjdk.java.net/jdk9/dev/jdk/rev/f92af283ab18#l13.1.

Here, we are proposing to add SO_REUSEPORT support into HDFS for its NameNode and DataNode implementations. We have a quick prototype which shows good throughput and latency improvements on given micro workload (purely HDFS workload, not MapReduce based). We will wrap up all the details together with a patch and send them here soon for further discussion and review. At the meantime, please let us know your comment and feedback.

Thanks,
Lucy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)