[jira] [Updated] (HDDS-17205) OM and SCM HA use reverse DNS names instead of configured IP literals for Ratis peer addresses

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17205:
------------------------------
    Description: 
When OM or SCM HA node addresses are configured as IP literals (for example {{ozone.scm.address.scmservice.scm1=[fd00:0:0:6::11]}}), the Ratis peer addresses are not the literals. They are whatever reverse DNS returns for the literal on the host that builds them. Different hosts can get different names for the same address (short name from {{/etc/hosts}} on the node itself, FQDN from DNS on the others), and a name may not be usable at all. Both SCM HA and OM HA fail to form a ring in the IPv6 compose cluster with literal addresses.

h3. Cause

{{NodeDetails}} builds the Ratis address from {{getRpcAddress().getHostName()}}:

{code:java}
public String getRatisHostPortStr() {
  return HddsUtils.getHostPortString(getHostName(), ratisPort);
}
{code}

For an {{InetSocketAddress}} created from a literal, {{getHostName()}} does a reverse lookup and caches the result in the address, so later {{getHostString()}} calls return the looked up name too. The same lookup happens earlier in {{NodeDetails}}' constructor ({{this.hostAddress = rpcAddress.getHostName()}}), the "Found matching ... address" log lines in {{SCMHANodeDetails}} and {{OMHANodeDetails}}, and the HTTP address helpers in {{OMHANodeDetails}} / {{OMNodeDetails}}. Host names in the configuration are not affected, because {{getHostName()}} returns the configured name without a lookup.

Lookups of the same address in the compose cluster:

||lookup of fd00:0:0:6::11 in||result||
|scm1 (itself)|7fdcd1080e27|
|scm2|ozone-ha-scm1-1.ozone-ha_default|

h3. Symptoms

*SCM HA:* the leader announces itself as {{7fdcd1080e27:9894}}. {{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks the leader up by comparing that string with {{getRatisHostPortStr()}} of each peer ({{ozone-ha-scm1-1.ozone-ha_default:9894}}), finds nothing, and fails {{Preconditions.checkState(leaderDetails.isPresent())}} with no message. Ratis sends the next install snapshot notification immediately, so the bootstrapping SCM loops forever (scm1 wrote about 90 MB of log in 4 minutes) and SCM stays in safe mode. {{SCMClientProtocolServer#resolveTargetPeerId}} (leader transfer by node ID) uses the same string comparison.

{noformat}
ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
java.lang.IllegalStateException
	at com.google.common.base.Preconditions.checkState(Preconditions.java:519)
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
{noformat}

*OM HA:* the peers are {{om2|ozone-ha-om2-1.ozone-ha_default:9872}} and so on, and no leader is elected. In this environment gRPC rejects the name because of the underscore in the Docker network name ({{Invalid DNS name: ozone-ha-om2-1.ozone-ha_default:9872}}). That part is specific to Docker, but the general problem is that the configured literal is replaced by a name that must then be resolved again. On a dual-stack host, that name can resolve to an IPv4 address even though an IPv6 literal was configured.

h3. Reproduction

{{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses for scm1..3 and om1..3, and {{ozone.scm.address.scmservice.scmN}} / {{ozone.om.address.omservice.omN}} set to the bracketed literals. Found while testing the HDDS-15763 patches (including HDDS-17135) against Hadoop branch-3.4.4 on Docker Desktop.

h3. Proposed fix

Keep the configured host string (literal or name) for Ratis peer addresses and for matching the leader, and do not call {{getHostName()}} on these addresses before it is captured. Only changing {{getRatisHostPortStr()}} to {{getHostString()}} is not enough: it was tried locally and still returned the looked up name because of the earlier {{getHostName()}} calls. Matching the leader by resolved socket address (IP and port) instead of by string would also make the SCM lookup robust. The tight retry loop after a failed notification is tracked separately in HDDS-17127.

  was:
When SCM HA addresses are configured as IP literals ({{ozone.scm.address.<service>.<node>=[fd00:0:0:6::11]}}), a bootstrapping SCM cannot install a snapshot from the leader and never joins the ring. SCM stays in safe mode and the leader and follower log thousands of lines per second.

h3. Cause

{{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks up the leader by comparing the leader's Ratis address string with {{getRatisHostPortStr()}} of each configured peer:

{code:java}
.filter(p -> p.getRatisHostPortStr().equals(leaderAddress))
...
Preconditions.checkState(leaderDetails.isPresent());
{code}

{{NodeDetails#getRatisHostPortStr}} uses {{getRpcAddress().getHostName()}}. For an address configured as a literal, that is a reverse lookup, and its result depends on the host doing the lookup. The leader resolves its own address through {{/etc/hosts}} (short host name), while the follower resolves the same address through DNS (FQDN). In the IPv6 compose cluster:

||lookup of fd00:0:0:6::11 in||result||
|scm1 (leader, itself)|7fdcd1080e27|
|scm2 (follower)|ozone-ha-scm1-1.ozone-ha_default|

So the leader announces {{7fdcd1080e27:9894}}, the follower finds no matching peer, and {{checkState}} throws an {{IllegalStateException}} with no message. Ratis immediately sends the next install snapshot notification, which fails the same way, so this becomes a tight loop (scm1 wrote about 90 MB of log in 4 minutes).

Host names in the configuration are not affected, because {{getHostName()}} returns the configured name without a lookup. Any literal (IPv4 or IPv6) is affected when reverse lookups disagree between hosts, which is common (short name in {{/etc/hosts}}, FQDN in DNS). IPv6 deployments without DNS names for every node are the most likely to configure literals.

h3. Reproduction

{{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses for scm1..3 and {{ozone.scm.address.scmservice.scmN}} set to the bracketed literals. Found while testing the HDDS-15763 patches (including HDDS-17135) on Docker Desktop.

{noformat}
ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
java.lang.IllegalStateException
	at com.google.common.base.Preconditions.checkState(Preconditions.java:519)
	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
{noformat}

h3. Proposed fix

Match the leader by resolved socket address (IP and port) instead of the host name string, and fail with a message naming the leader address when no peer matches. The tight retry loop after a failed notification is tracked separately in HDDS-17127.

        Summary: OM and SCM HA use reverse DNS names instead of configured IP literals for Ratis peer addresses  (was: SCM follower cannot install a snapshot when SCM addresses are configured as IP literals)

> OM and SCM HA use reverse DNS names instead of configured IP literals for Ratis peer addresses
> ----------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17205
>                 URL: https://issues.apache.org/jira/browse/HDDS-17205
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> When OM or SCM HA node addresses are configured as IP literals (for example {{ozone.scm.address.scmservice.scm1=[fd00:0:0:6::11]}}), the Ratis peer addresses are not the literals. They are whatever reverse DNS returns for the literal on the host that builds them. Different hosts can get different names for the same address (short name from {{/etc/hosts}} on the node itself, FQDN from DNS on the others), and a name may not be usable at all. Both SCM HA and OM HA fail to form a ring in the IPv6 compose cluster with literal addresses.
> h3. Cause
> {{NodeDetails}} builds the Ratis address from {{getRpcAddress().getHostName()}}:
> {code:java}
> public String getRatisHostPortStr() {
>   return HddsUtils.getHostPortString(getHostName(), ratisPort);
> }
> {code}
> For an {{InetSocketAddress}} created from a literal, {{getHostName()}} does a reverse lookup and caches the result in the address, so later {{getHostString()}} calls return the looked up name too. The same lookup happens earlier in {{NodeDetails}}' constructor ({{this.hostAddress = rpcAddress.getHostName()}}), the "Found matching ... address" log lines in {{SCMHANodeDetails}} and {{OMHANodeDetails}}, and the HTTP address helpers in {{OMHANodeDetails}} / {{OMNodeDetails}}. Host names in the configuration are not affected, because {{getHostName()}} returns the configured name without a lookup.
> Lookups of the same address in the compose cluster:
> ||lookup of fd00:0:0:6::11 in||result||
> |scm1 (itself)|7fdcd1080e27|
> |scm2|ozone-ha-scm1-1.ozone-ha_default|
> h3. Symptoms
> *SCM HA:* the leader announces itself as {{7fdcd1080e27:9894}}. {{SCMStateMachine#notifyInstallSnapshotFromLeader}} looks the leader up by comparing that string with {{getRatisHostPortStr()}} of each peer ({{ozone-ha-scm1-1.ozone-ha_default:9894}}), finds nothing, and fails {{Preconditions.checkState(leaderDetails.isPresent())}} with no message. Ratis sends the next install snapshot notification immediately, so the bootstrapping SCM loops forever (scm1 wrote about 90 MB of log in 4 minutes) and SCM stays in safe mode. {{SCMClientProtocolServer#resolveTargetPeerId}} (leader transfer by node ID) uses the same string comparison.
> {noformat}
> ERROR impl.SnapshotInstallationHandler: ...: installSnapshot failed
> java.lang.IllegalStateException
> 	at com.google.common.base.Preconditions.checkState(Preconditions.java:519)
> 	at org.apache.hadoop.hdds.scm.ha.SCMStateMachine.notifyInstallSnapshotFromLeader(SCMStateMachine.java:285)
> {noformat}
> *OM HA:* the peers are {{om2|ozone-ha-om2-1.ozone-ha_default:9872}} and so on, and no leader is elected. In this environment gRPC rejects the name because of the underscore in the Docker network name ({{Invalid DNS name: ozone-ha-om2-1.ozone-ha_default:9872}}). That part is specific to Docker, but the general problem is that the configured literal is replaced by a name that must then be resolved again. On a dual-stack host, that name can resolve to an IPv4 address even though an IPv6 literal was configured.
> h3. Reproduction
> {{compose/ozone-ha}} on an IPv6-only network (HDDS-16969) with static addresses for scm1..3 and om1..3, and {{ozone.scm.address.scmservice.scmN}} / {{ozone.om.address.omservice.omN}} set to the bracketed literals. Found while testing the HDDS-15763 patches (including HDDS-17135) against Hadoop branch-3.4.4 on Docker Desktop.
> h3. Proposed fix
> Keep the configured host string (literal or name) for Ratis peer addresses and for matching the leader, and do not call {{getHostName()}} on these addresses before it is captured. Only changing {{getRatisHostPortStr()}} to {{getHostString()}} is not enough: it was tried locally and still returned the looked up name because of the earlier {{getHostName()}} calls. Matching the leader by resolved socket address (IP and port) instead of by string would also make the SCM lookup robust. The tight retry loop after a failed notification is tracked separately in HDDS-17127.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org