[jira] [Commented] (HDFS-17999) NameNode fails to parse IPv6 DataNode addresses in WebHDFS excludedatanodes and favored nodes

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129842#comment-18129842 ] 

ASF GitHub Bot commented on HDFS-17999:
---------------------------------------

smengcl opened a new pull request, #8853:
URL: https://github.com/apache/hadoop/pull/8853

   Generated-by: Claude Code (Opus 5.5)
   
   ### Description of PR
   
   The NameNode split client supplied DataNode addresses at the first `:` in two places, so IPv6 addresses failed with `NumberFormatException`:
   
   - WebHDFS `excludedatanodes` (`NamenodeWebHdfsMethods#chooseDatanode`): the WebHDFS client retries with entries such as `[2001:db8::1]:9864`, and the request failed with 400.
   - Favored nodes (`DatanodeManager#parseDNFromHostsEntry`, reached from `addBlock`): `DFSClient` sends `2001:db8:0:0:0:0:0:1:9866`, and `addBlock` failed.
   
   This adds `DatanodeManager#parseIpv6Address`, used by both. It handles only addresses with more than one `:`: `[addr]:port`, `[addr]`, `addr:port` (split at the last `:` when the prefix is an IP literal) and `addr`, and normalizes the address to the expanded form DataNodes register with. IPv4 addresses and host names take the existing code path unchanged.
   
   ### How was this patch tested?
   
   - `TestWebHdfsDataLocality#testExcludeIpv6DataNodes`: passes bracketed, plain and portless IPv6 entries plus an IPv4 entry to `chooseDatanode` and checks that all four DataNodes end up in the exclude set. Fails without the fix with `NumberFormatException: For input string: "db8::1]:9866"`.
   - `TestDatanodeManager#testGetDatanodeDescriptorForIpv4AndIpv6`: checks that `getDatanodeDescriptor` resolves IPv4 with and without a port, and IPv6 as `addr:port`, `[addr]:port`, `[addr]`, `addr` and compressed `[addr]:port`. Fails without the fix with `NumberFormatException`.
   - `TestDatanodeManager` and `TestWebHdfsDataLocality` pass. Not run on an IPv6 MiniDFSCluster.
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   




> NameNode fails to parse IPv6 DataNode addresses in WebHDFS excludedatanodes and favored nodes
> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17999
>                 URL: https://issues.apache.org/jira/browse/HDFS-17999
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, webhdfs
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>              Labels: ipv6
>
> Two NameNode code paths parse a DataNode address sent by a client by splitting it at the first colon, so an IPv6 address fails with {{NumberFormatException}}.
> *WebHDFS excludedatanodes.* {{NamenodeWebHdfsMethods.chooseDatanode}} ([NamenodeWebHdfsMethods.java:298|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L298]):
> {code:java}
> int idx = host.indexOf(':');
> ...
> excludeNode = bm.getDatanodeManager().getDatanodeByXferAddr(
>   host.substring(0, idx), Integer.parseInt(host.substring(idx + 1)));
> {code}
> The WebHDFS client fills this parameter itself. The NameNode redirects to a DataNode with a bracketed URI such as {{http://[2001:db8::1]:9864/webhdfs/v1/...}}, and if the client then fails to reach that DataNode it adds {{[2001:db8::1]:9864}} to {{excludedatanodes}} ([WebHdfsFileSystem.java:744|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L744], {{URL.getHost()}} keeps the brackets). On the retry the NameNode throws {{NumberFormatException: For input string: "db8::1]:9864"}}, which {{ExceptionHandler}} maps to 400 Bad Request. The client does not retry a {{RemoteException}}, so the read or write fails instead of moving to another DataNode. The client sends the parameter on retries, which happen when HA failover is configured or {{dfs.http.client.retry.policy.enabled}} is true. A REST caller that passes an IPv6 {{excludedatanodes}} value hits it directly.
> *Favored nodes.* {{DistributedFileSystem#create(..., InetSocketAddress[] favoredNodes)}} sends each favored node as {{getHostName() + ":" + getPort()}} ([DFSClient.java:1334|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1334]). For an IPv6 DataNode without a PTR record that is {{2001:db8:0:0:0:0:0:1:9866}}. On addBlock, {{BlockManager#chooseTarget4NewBlock}} resolves favored nodes through {{DatanodeManager#getDatanodeDescriptor}}, whose {{parseDNFromHostsEntry}} ([DatanodeManager.java:1610|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java#L1610]) splits at the first colon and throws {{NumberFormatException}}. The addBlock RPC fails and the client cannot write the file.
> Removing the brackets is not enough for an entry to match. {{Host2NodesMap}} is keyed by {{DatanodeID.ipAddr}}, the expanded form from {{InetAddress.getHostAddress()}} (such as {{2001:db8:0:0:0:0:0:1}}), so a compressed literal also has to be normalized.
> *Fix:* add {{DatanodeManager#parseIpv6Address}} and use it on both paths. It only handles addresses with more than one colon:
> * {{[addr]:port}} and {{[addr]}} are bracketed IPv6 addresses.
> * An unbracketed {{addr:port}} is split at the last colon when the part before it is an IP literal; otherwise the whole string is the address.
> * The address is normalized with {{InetAddresses.forString(addr).getHostAddress()}}.
> Addresses with at most one colon (IPv4 and host names) go through the existing code unchanged. A malformed entry still fails the request, as a malformed IPv4 entry does today. A bare IPv6 address with no port is ambiguous when its last group could be a port (for example {{2001:db8::5:1}}); callers should bracket it.
> Not addressed here: the WebHDFS client puts the DataNode HTTP port in {{excludedatanodes}} (the redirect uses {{dn.getInfoPort()}}, [NamenodeWebHdfsMethods.java:463|https://github.com/apache/hadoop/blob/35bbaa10e886f10947f35f1ae32e86e6c4c52ef1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L463]), while {{chooseDatanode}} compares it with the transfer port, so client generated exclusions with a port do not match on IPv4 either. With this fix IPv6 entries behave like IPv4 ones.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org