[jira] [Created] (HDFS-17286) Add UDP as a transfer protocol for HDFS

["Xing Lin (Jira)" <ji...@apache.org>]

Xing Lin created HDFS-17286:
-------------------------------

             Summary: Add UDP as a transfer protocol for HDFS
                 Key: HDFS-17286
                 URL: https://issues.apache.org/jira/browse/HDFS-17286
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs
            Reporter: Xing Lin


Right now, every connection in HDFS is based on RPC/IPC which is based on TCP. Connection is re-used based on ConnectionID, which includes RpcTimeout as part of the key to identify a connection. The consequence is if we want to use a different rpc timeout between two hosts, this would create different TCP connections. 

A use case which motivated us to consider UDP is getHAServiceState() in ObserverReadProxyProvider. We'd like getHAServiceState() to time out with a much smaller timeout threshold and move to probe next Namenode. To support this, we used an executorService and set a timeout for the task in HDFS-17030. This implementation can be improved by using UDP to query HAServiceState. getHAServiceState() does not have to be very reliable, as we can always fall back to the active.

Another motivation is it seems 5~10% of RPC calls hitting our active/observers are GetHAServiceState(). If we can move them off to the UDP server, that can hopefully improve RPC latency.

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org