[jira] [Created] (YARN-12032) RM builds node HTTP and RPC addresses without brackets for IPv6 NodeId hosts

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created YARN-12032:
---------------------------------

             Summary: RM builds node HTTP and RPC addresses without brackets for IPv6 NodeId hosts
                 Key: YARN-12032
                 URL: https://issues.apache.org/jira/browse/YARN-12032
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
            Reporter: Siyao Meng
            Assignee: Siyao Meng


[RMNodeImpl.java:404|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java#L404] builds:
{code:java}
this.nodeAddress = hostName + ":" + cmPort;
this.httpAddress = hostName + ":" + httpPort;
{code}

On a node without reverse DNS, the NodeManager's NodeId host is a numeric IPv6 literal such as {{2001:db8:0:0:0:0:0:21}}, because {{InetAddress.getCanonicalHostName()}} returns the literal ([ContainerManagerImpl.java:753|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java#L753]). Clusters whose NodeManager hosts reverse-resolve to names are not affected. {{httpAddress}} is then {{2001:db8:0:0:0:0:0:21:8042}}. This value becomes {{Container.getNodeHttpAddress()}}, {{NodeReport.getHttpAddress()}} and the REST {{nodeHTTPAddress}}, and {{new URL("http://2001:db8:0:0:0:0:0:21:8042")}} throws {{MalformedURLException}}.

*MapReduce jobs fail.* The MR AM calls {{NetUtils.createSocketAddr(container.getNodeHttpAddress())}} ([TaskAttemptImpl.java:2008|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java#L2008], also L1730). Trunk's {{createSocketAddr}} rejects unbracketed IPv6, so it throws {{IllegalArgumentException}} inside a state machine transition and the AM's dispatcher shuts the AM down.

*Web UI and redirects are broken:*
* node links in [NodesPage.java:192|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NodesPage.java#L192] and [RMAppBlock.java:156|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L156], plus {{AppBlock}}, {{AppAttemptBlock}} and {{ContainerBlock}};
* AM container log links in [AppInfo.java:213|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java#L213] and {{AppAttemptInfo.java:90}};
* the 307 redirect to the NodeManager in [LogServlet.java:378|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogServlet.java#L378].

Verified by reading the code, and by running copies of the trunk composition code and {{createSocketAddr}} on JDK 21. Not run on a cluster.

*Proposed fix:* build both addresses with {{NetUtils.getHostPortString(hostName, port)}}, which brackets IPv6 and leaves host names and IPv4 unchanged. Then check consumers that split the address on {{:}}: {{TaskImpl.java:700}} takes the shuffle host with {{getNodeHttpAddress().split(":")[0]}}, which would give {{[2001}}, and needs a bracket-aware host in the same change or a MAPREDUCE follow-up.

POC (fixes some consumers only; does not change {{RMNodeImpl}}): https://github.com/smengcl/hadoop/commit/d4009c263cd14a6f3ebe6a259996c901eeac4f6e



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org