[jira] [Updated] (HDFS-17999) NameNode fails to parse IPv6 DataNode addresses in WebHDFS excludedatanodes and favored nodes

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDFS-17999:
------------------------------
    Status: Patch Available  (was: Open)

> NameNode fails to parse IPv6 DataNode addresses in WebHDFS excludedatanodes and favored nodes
> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17999
>                 URL: https://issues.apache.org/jira/browse/HDFS-17999
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, webhdfs
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>              Labels: ipv6, pull-request-available
>
> Two NameNode code paths parse a DataNode address sent by a client by splitting it at the first colon, so an IPv6 address fails with {{NumberFormatException}}.
> *WebHDFS excludedatanodes.* {{NamenodeWebHdfsMethods.chooseDatanode}} ([NamenodeWebHdfsMethods.java:298|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L298]):
> {code:java}
> int idx = host.indexOf(':');
> ...
> excludeNode = bm.getDatanodeManager().getDatanodeByXferAddr(
>   host.substring(0, idx), Integer.parseInt(host.substring(idx + 1)));
> {code}
> The WebHDFS client fills this parameter itself. The NameNode redirects to a DataNode with a bracketed URI such as {{http://[2001:db8::1]:9864/webhdfs/v1/...}}, and if the client then fails to reach that DataNode it adds {{[2001:db8::1]:9864}} to {{excludedatanodes}} ([WebHdfsFileSystem.java:744|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L744], {{URL.getHost()}} keeps the brackets). On the retry the NameNode throws {{NumberFormatException: For input string: "db8::1]:9864"}}, which {{ExceptionHandler}} maps to 400 Bad Request. The client does not retry a {{RemoteException}}, so the read or write fails instead of moving to another DataNode. The client sends the parameter on retries, which happen when HA failover is configured or {{dfs.http.client.retry.policy.enabled}} is true. A REST caller that passes an IPv6 {{excludedatanodes}} value hits it directly.
> *Favored nodes.* {{DistributedFileSystem#create(..., InetSocketAddress[] favoredNodes)}} sends each favored node as {{getHostName() + ":" + getPort()}} ([DFSClient.java:1334|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1334]). For an IPv6 DataNode without a PTR record that is {{2001:db8:0:0:0:0:0:1:9866}}. On addBlock, {{BlockManager#chooseTarget4NewBlock}} resolves favored nodes through {{DatanodeManager#getDatanodeDescriptor}}, whose {{parseDNFromHostsEntry}} ([DatanodeManager.java:1610|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java#L1610]) splits at the first colon and throws {{NumberFormatException}}. The addBlock RPC fails and the client cannot write the file.
> Removing the brackets is not enough for an entry to match. {{Host2NodesMap}} is keyed by {{DatanodeID.ipAddr}}, the expanded form from {{InetAddress.getHostAddress()}} (such as {{2001:db8:0:0:0:0:0:1}}), so a compressed literal also has to be normalized.
> *Fix:* add {{DatanodeManager#parseIpv6Address}} and use it on both paths. It only handles addresses with more than one colon:
> * {{[addr]:port}} and {{[addr]}} are bracketed IPv6 addresses.
> * An unbracketed {{addr:port}} is split at the last colon when the part before it is an IP literal; otherwise the whole string is the address.
> * The address is normalized with {{InetAddresses.forString(addr).getHostAddress()}}.
> Addresses with at most one colon (IPv4 and host names) go through the existing code unchanged. A malformed entry still fails the request, as a malformed IPv4 entry does today. A bare IPv6 address with no port is ambiguous when its last group could be a port (for example {{2001:db8::5:1}}); callers should bracket it.
> Not addressed here: the WebHDFS client puts the DataNode HTTP port in {{excludedatanodes}} (the redirect uses {{dn.getInfoPort()}}, [NamenodeWebHdfsMethods.java:463|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L463]), while {{chooseDatanode}} compares it with the transfer port, so client generated exclusions with a port do not match on IPv4 either. With this fix IPv6 entries behave like IPv4 ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org