[jira] [Created] (YARN-12033) NodeId.fromString cannot parse an IPv6 NodeId, breaking aggregated log pages and node APIs

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created YARN-12033:
---------------------------------

             Summary: NodeId.fromString cannot parse an IPv6 NodeId, breaking aggregated log pages and node APIs
                 Key: YARN-12033
                 URL: https://issues.apache.org/jira/browse/YARN-12033
             Project: Hadoop YARN
          Issue Type: Bug
          Components: api
            Reporter: Siyao Meng
            Assignee: Siyao Meng


On a node without reverse DNS, the NodeManager's NodeId host is a numeric IPv6 literal such as {{2001:db8:0:0:0:0:0:21}}, because {{InetAddress.getCanonicalHostName()}} returns the literal ([ContainerManagerImpl.java:753|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java#L753]). Clusters whose NodeManager hosts reverse-resolve to names are not affected. {{NodeId.toString()}} then gives {{2001:db8:0:0:0:0:0:21:8041}}, which {{NodeId.fromString()}} cannot parse back ([NodeId.java:119|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java#L119]):

{code:java}
String[] parts = nodeIdStr.split(":");
if (parts.length != 2) {
  throw new IllegalArgumentException("Invalid NodeId [" + nodeIdStr
      + "]. Expected host:port");
{code}

The bracketed form {{[2001:db8::21]:8041}} is rejected too.

*Aggregated logs.* A finished attempt's log link is {{<yarn.log.server.url>/<NodeId>/<containerId>/...}} ([AppAttemptInfo.java:85|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppAttemptInfo.java#L85]). The NodeManager redirect ([NMWebAppFilter.java:124|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebAppFilter.java#L124]), [WebAppUtils.java:464|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java#L464] and [TimelineEntityV2Converter.java:138|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineEntityV2Converter.java#L138] build links the same way. The log page parses the segment in [LogAggregationWebUtils.java:103|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationWebUtils.java#L103] and shows "Cannot get container logs. Invalid nodeId". The URL itself is valid: colons are allowed in a path segment.

*Other callers of the same parser:*
* {{GET /ws/v1/cluster/nodes/{nodeid}}} and the node resource update ([RMWebServices.java:518|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L518], L581);
* the activities REST call, which also splits at the first colon ([RMWebServices.java:780|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L780]) before {{ActivitiesManager}} calls {{fromString}};
* {{yarn node -status}} and {{rmadmin -updateNodeResource}};
* MR AM recovery of completed task attempts ([TaskAttemptImpl.java:1454|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java#L1454]).

Verified by running a copy of the trunk parsing on JDK 21, and {{getCanonicalHostName()}} for a literal with no PTR record. Callers traced in code. Not run on a cluster.

*Proposed fix:* accept {{[ipv6]:port}} in {{fromString}}, and split an unbracketed value at the last colon; the port is always present, so the {{toString()}} form is unambiguous. Keep {{toString()}} unchanged for compatibility. Do not put brackets in URL path segments: RFC 3986 does not allow them there, and {{java.net.URI}} rejects them.

YARN-4283 changed {{fromString}} on the unmerged HADOOP-17800 branch only, and its Guava {{HostAndPort}} version still rejects the unbracketed {{toString()}} form.

POC (do not port its {{getRunningLogURL}} change, which brackets scheme-prefixed input as {{[http://host:8042]}} and breaks IPv4 links): https://github.com/smengcl/hadoop/commit/190839820bac7f3356dc3f52671eeb8363edb6e2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org