[jira] [Created] (HDFS-18002) RBF: Router builds unbracketed IPv6 host:port strings for heartbeats and the State Store

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDFS-18002:
---------------------------------

             Summary: RBF: Router builds unbracketed IPv6 host:port strings for heartbeats and the State Store
                 Key: HDFS-18002
                 URL: https://issues.apache.org/jira/browse/HDFS-18002
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: rbf
            Reporter: Siyao Meng
            Assignee: Siyao Meng


The Router builds {{host:port}} strings by concatenation in two places. For an IPv6 host the result has no brackets, for example {{2001:db8:0:0:0:0:0:10:8020}}, and {{NetUtils.createSocketAddr}} rejects it with "Does not contain a valid host:port authority".

*1. NamenodeHeartbeatService with name resolution* ([NamenodeHeartbeatService.java:206|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L206])
{code:java}
rpcAddress = resolvedHost + ":"
    + NetUtils.getPortFromHostPortString(rpcAddress);
serviceAddress = resolvedHost + ":"
    + NetUtils.getPortFromHostPortString(serviceAddress);
{code}
With {{dfs.federation.router.monitor.namenode.nameservice.resolution-enabled.<ns>=true}}, the Router resolves the configured NameNode name to its addresses and creates one heartbeat service per address with {{resolvedHost = isa.getHostName()}} ([Router.java:577|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java#L577]). For an AAAA record with no PTR record that is the IPv6 literal. {{lifelineAddress}} and {{webAddress}} are built the same way. {{serviceInit}} then creates an {{NNHAServiceTarget}} from {{serviceAddress}}, whose {{createSocketAddr}} call throws. The heartbeat service is a child service of the Router, so the Router fails to start. {{getRpcAddress()}} has the same pattern in its fallback ([NamenodeHeartbeatService.java:299|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L299]), reached when a nameservice is monitored without an nnId and lists one NameNode. Without resolution the addresses are the configured, bracketed strings, and these paths work.

*2. Router admin addresses in the State Store* ([StateStoreUtils.java:137|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L137], [StateStoreUtils.java:152|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L152])
{code:java}
return hostName + ":" + address.getPort();
...
return inet.getHostAddress() + ":" + address.getPort();
{code}
{{RouterHeartbeatService}} stores each Router's admin address with {{getHostPortString}} (default) or {{getIpPortString}} ({{dfs.federation.router.heartbeat.with.ip.enable=true}}). {{getIpPortString}} always gives an unbracketed IPv6 string; {{getHostPortString}} does when there is no PTR record. {{getHostPortString}} also only replaces the IPv4 wildcard {{0.0.0.0}}, so {{dfs.federation.router.admin-address=[::]:8111}} is stored as {{0:0:0:0:0:0:0:0:8111}}. With {{dfs.federation.router.mount-table.cache.update=true}} and two or more Routers, {{MountTableRefresherService}} passes these addresses to {{createSocketAddr}} ([MountTableRefresherService.java:170|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/MountTableRefresherService.java#L170]). The {{IllegalArgumentException}} reaches the caller wrapped in Guava's {{UncheckedExecutionException}}, which the {{catch (ExecutionException)}} at line 248 does not catch, so add, update and remove mount table calls return an error after the entry has been written, and the other Routers are not refreshed.

Verified by reading the code, and on JDK 21: {{getHostAddress()}} and the no-PTR {{getHostName()}} return unbracketed literals, and a copy of the {{createSocketAddr}} parsing rejects them. The Guava wrapping is from the documented {{LoadingCache.get}} contract. Not run on a Router or with real DNS.

*Proposed fix:* use {{NetUtils.getHostPortString(host, port)}} for the four heartbeat addresses, the {{getRpcAddress()}} fallback, and both {{StateStoreUtils}} helpers. In {{getHostPortString}}, treat any {{isAnyLocalAddress()}} address as the wildcard. Optionally catch {{RuntimeException}} per Router in {{MountTableRefresherService}} so one bad record cannot fail the admin call. Add unit tests with IPv6 hosts.

POC: https://github.com/smengcl/hadoop/commit/daab9184f1b0818161b3c53a50753aaaefb9f568



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org