You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "kerneltime (via GitHub)" <gi...@apache.org> on 2026/06/11 04:37:47 UTC
[PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
kerneltime opened a new pull request, #10485:
URL: https://github.com/apache/ozone/pull/10485
## What changes were proposed in this pull request?
This is **PR 1 of 4** splitting [HDDS-15514](https://issues.apache.org/jira/browse/HDDS-15514) (originally proposed as a single ~160KB patch in #10473, split per @szetszwo's review feedback).
This PR fixes the Ratis-replication paths only:
- **OM ↔ OM Ratis replication**: `OzoneManagerRatisServer.createRaftPeer` now always passes a `hostname:port` string to `RaftPeer.setAddress(...)`, never a resolved `InetSocketAddress`. Two of the three pre-existing `createRaftPeer` branches called `new InetSocketAddress(omNode.getInetAddress(), ratisPort)`, baking the resolved IP into `RaftPeer.address`. The two overloads collapse into one.
- **SCM ↔ SCM Ratis replication**: comment-only — replaces the misleading `// TODO : Should we use IP instead of hostname??` markers in `SCMRatisServerImpl.buildRaftGroup` and `SCMHAManagerImpl#start` with explanatory comments citing HDDS-15514. Both call sites already passed `getRatisHostPortStr()`; the TODO falsely implied IPs would be a fine alternative.
## Why this matters
Ratis builds gRPC channels via `NettyChannelBuilder.forTarget(address)`. The default `DnsNameResolver` re-resolves hostnames on connection failure, so a peer-pod restart in Kubernetes (where the pod's IP changes but the DNS name stays stable) is recovered automatically — **as long as `RaftPeer.address` is a hostname**. If the address is a numeric IP (because Ozone resolved it at construction and passed the resolved form to Ratis), the channel stays bound to the now-defunct IP forever and the only recovery is a parent-process restart.
## How was this patch tested?
- Unit test: `TestOzoneManagerRatisServer.testCreateRaftPeerUsesHostnameAddress` — asserts that `RaftPeer.getAddress()` for an OM peer is the literal `hostname:port` string and is not an IPv4 numeric form. This guards the invariant against future regressions that re-introduce `InetSocketAddress` at this seam.
- Existing `TestOzoneManagerRatisServer` tests (5 prior + 1 new) all pass.
- `mvn clean install` of `hadoop-ozone/ozone-manager` and `hadoop-hdds/server-scm` modules.
## Scope of this PR
- Pure hostname-string discipline. No new flags, no new exception classifier, no atomic-replace machinery, no `RPC.stopProxy` calls. Those arrive with the next three PRs.
- Zero behavior change in non-K8s deployments where DNS-to-IP is stable for the process lifetime.
## Follow-up PRs
Per the [4-PR split](https://github.com/apache/ozone/pull/10473#issuecomment-4677191510):
2. **HDDS-15514. DNS refresh on connection failure for Client → OM** — introduces `ConnectionFailureUtils`, `ozone.client.failover.resolve-needed` flag, `OMProxyInfo.refreshAddressIfChanged`, and the `OMFailoverProxyProviderBase.shouldRetry` hook.
3. **HDDS-15514. DNS refresh on connection failure for OM → SCM** — same shape against `SCMFailoverProxyProviderBase` / `SCMProxyInfo`.
4. **HDDS-15514. DNS refresh on heartbeat failure for DN → SCM** — `EndpointStateMachine.resolveLatestAddress`, `SCMConnectionManager.refreshSCMServer` (4-phase atomic-replace), `StateContext.migrateEndpoint`, and the `ozone.datanode.scm.heartbeat.address.refresh.threshold` knob.
## What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-15514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10485:
URL: https://github.com/apache/ozone/pull/10485#discussion_r3400891098
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:
##########
@@ -145,6 +145,37 @@ public void testStartOMRatisServer() throws Exception {
"Ratis Server should be in running state");
}
+ /**
+ * RaftPeer.address must always be a hostname:port string, never an
+ * IP:port string. This is what allows gRPC's DnsNameResolver to
+ * re-resolve the peer when its underlying IP changes (Kubernetes pod
+ * restart). If a resolved IP were baked in, recovery would require
+ * a full OM restart.
+ */
+ @Test
+ public void testCreateRaftPeerUsesHostnameAddress() {
+ String hostname = "om-2.om.example.svc.cluster.local";
+ int ratisPort = 9872;
Review Comment:
nit:
int rpcPort = 9862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10485:
URL: https://github.com/apache/ozone/pull/10485#issuecomment-4687876136
Done in the latest force-push. Collapsed both `createRaftPeer` overloads into the single form you suggested, and updated `newOMRatisServer` to iterate `peerNodes.values()` directly.
Also reworded the Javadoc and the SCM-side comments per Copilot's earlier feedback: the invariant is now described as "pass the configured host string through verbatim; don't pre-resolve into an `InetSocketAddress`" rather than "must be hostname". IP-literal configs are explicitly noted as still honored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "amaliujia (via GitHub)" <gi...@apache.org>.
amaliujia commented on code in PR #10485:
URL: https://github.com/apache/ozone/pull/10485#discussion_r3400891098
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:
##########
@@ -145,6 +145,37 @@ public void testStartOMRatisServer() throws Exception {
"Ratis Server should be in running state");
}
+ /**
+ * RaftPeer.address must always be a hostname:port string, never an
+ * IP:port string. This is what allows gRPC's DnsNameResolver to
+ * re-resolve the peer when its underlying IP changes (Kubernetes pod
+ * restart). If a resolved IP were baked in, recovery would require
+ * a full OM restart.
+ */
+ @Test
+ public void testCreateRaftPeerUsesHostnameAddress() {
+ String hostname = "om-2.om.example.svc.cluster.local";
+ int ratisPort = 9872;
Review Comment:
int rpcPort = 9862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10485:
URL: https://github.com/apache/ozone/pull/10485#discussion_r3397140251
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:
##########
@@ -145,6 +145,37 @@ public void testStartOMRatisServer() throws Exception {
"Ratis Server should be in running state");
}
+ /**
+ * RaftPeer.address must always be a hostname:port string, never an
+ * IP:port string. This is what allows gRPC's DnsNameResolver to
+ * re-resolve the peer when its underlying IP changes (Kubernetes pod
+ * restart). If a resolved IP were baked in, recovery would require
+ * a full OM restart.
+ */
Review Comment:
The Javadoc says RaftPeer.address must never be an IP:port string, but the actual requirement for HDDS-15514 is to preserve the configured host string and avoid pre-resolving a hostname to a numeric IP before handing it to Ratis/gRPC. Rewording this avoids misleading readers into thinking numeric IP configuration is invalid.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMRatisServerImpl.java:
##########
@@ -396,8 +396,14 @@ private static RaftGroup buildRaftGroup(SCMNodeDetails details,
final RaftGroupId groupId = buildRaftGroupId(clusterId);
RaftPeerId selfPeerId = getSelfPeerId(scmId);
+ // The peer address is intentionally a hostname:port string, never a
+ // resolved IP. Ratis hands this string to gRPC's NettyChannelBuilder,
+ // whose default DnsNameResolver re-resolves hostnames on connection
+ // failure. Baking a resolved IP would freeze the peer at one IP,
+ // breaking recovery from peer pod restarts in Kubernetes-style
+ // environments where DNS names are stable but IPs are not. See
+ // HDDS-15514 (DNS-refresh-on-failure for all RPC paths).
Review Comment:
The new comment says the peer address is never a resolved IP; the important invariant is that the configured host string is passed through (so a hostname stays a hostname) rather than constructing an InetSocketAddress from an InetAddress. Consider rewording to avoid implying that numeric IP configuration is inherently invalid.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAManagerImpl.java:
##########
@@ -118,9 +118,11 @@ public void start() throws IOException {
final boolean success = HAUtils.addSCM(ozoneConf,
new AddSCMRequest.Builder().setClusterId(scm.getClusterId())
.setScmId(scm.getScmId())
- .setRatisAddr(nodeDetails
- // TODO : Should we use IP instead of hostname??
- .getRatisHostPortStr()).build(), scm.getSCMNodeId());
+ // Hostname is intentional: gRPC's DnsNameResolver re-resolves
+ // hostnames on connection failure, so SCM peer pod restarts
+ // are recovered automatically. See HDDS-15514.
Review Comment:
This comment implies hostname is always required. The main requirement is to avoid baking a resolved IP into the Ratis address; passing through the configured host string preserves hostnames when used, while still allowing explicit IP configuration when desired.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:
##########
@@ -435,44 +435,29 @@ private void updateRatisConfiguration(List<RaftPeer> followers, List<RaftPeer> l
}
}
- private static RaftPeer createRaftPeer(OMNodeDetails omNode) {
- String nodeId = omNode.getNodeId();
- RaftPeerId raftPeerId = RaftPeerId.valueOf(nodeId);
- InetSocketAddress ratisAddr = new InetSocketAddress(
- omNode.getHostAddress(), omNode.getRatisPort());
- RaftPeerRole startRole = omNode.isRatisListener() ?
- RaftPeerRole.LISTENER : RaftPeerRole.FOLLOWER;
+ /**
+ * Build a RaftPeer for the given OM node. The peer address is always set
+ * as a hostname:port string (never a resolved IP). Ratis ultimately hands
+ * this string to gRPC's NettyChannelBuilder.forTarget(...), whose default
+ * DnsNameResolver re-resolves hostnames on connection failure / refresh.
+ * Baking a resolved IP into RaftPeer.address would freeze the peer at one
+ * IP for the channel's lifetime, breaking recovery from peer pod restarts
+ * in environments like Kubernetes where DNS names are stable but IPs are
+ * not. See HDDS-15514 (DNS-refresh-on-failure for all RPC paths).
+ */
Review Comment:
This Javadoc states the peer address is always a hostname:port string, which is stronger than what the implementation guarantees/needs. The key invariant is: don’t pass a resolved InetAddress/InetSocketAddress (which bakes in a numeric IP); instead, pass the configured host string through to gRPC so it can re-resolve when needed. Rewording will better match the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10485:
URL: https://github.com/apache/ozone/pull/10485#issuecomment-4699331468
Thank you for the quick turn around. There are 3 more to go, the next in the chain is https://github.com/apache/ozone/pull/10486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10485:
URL: https://github.com/apache/ozone/pull/10485#discussion_r3398265758
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:
##########
@@ -435,44 +435,29 @@ private void updateRatisConfiguration(List<RaftPeer> followers, List<RaftPeer> l
}
}
- private static RaftPeer createRaftPeer(OMNodeDetails omNode) {
- String nodeId = omNode.getNodeId();
- RaftPeerId raftPeerId = RaftPeerId.valueOf(nodeId);
- InetSocketAddress ratisAddr = new InetSocketAddress(
- omNode.getHostAddress(), omNode.getRatisPort());
- RaftPeerRole startRole = omNode.isRatisListener() ?
- RaftPeerRole.LISTENER : RaftPeerRole.FOLLOWER;
+ /**
+ * Build a RaftPeer for the given OM node. The peer address is always set
+ * as a hostname:port string (never a resolved IP). Ratis ultimately hands
+ * this string to gRPC's NettyChannelBuilder.forTarget(...), whose default
+ * DnsNameResolver re-resolves hostnames on connection failure / refresh.
+ * Baking a resolved IP into RaftPeer.address would freeze the peer at one
+ * IP for the channel's lifetime, breaking recovery from peer pod restarts
+ * in environments like Kubernetes where DNS names are stable but IPs are
+ * not. See HDDS-15514 (DNS-refresh-on-failure for all RPC paths).
+ */
+ static RaftPeer createRaftPeer(OMNodeDetails omNode) {
+ return createRaftPeer(omNode, omNode.getNodeId());
+ }
+ static RaftPeer createRaftPeer(OMNodeDetails omNode, String nodeId) {
Review Comment:
Combine these two createRaftPeer methods:
```java
static RaftPeer createRaftPeer(OMNodeDetails omNode) {
return RaftPeer.newBuilder()
.setId(RaftPeerId.valueOf(omNode.getNodeId()))
.setAddress(omNode.getRatisHostPortStr())
.setStartupRole(omNode.isRatisListener() ? RaftPeerRole.LISTENER : RaftPeerRole.FOLLOWER)
.build();
}
```
```diff
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java
@@ -227,10 +227,8 @@ public static OzoneManagerRatisServer newOMRatisServer(
// On regular startup, add all OMs to Ratis ring
raftPeers.add(localRaftPeer);
- for (Map.Entry<String, OMNodeDetails> peerInfo : peerNodes.entrySet()) {
- String peerNodeId = peerInfo.getKey();
- OMNodeDetails peerNode = peerInfo.getValue();
- RaftPeer raftPeer = OzoneManagerRatisServer.createRaftPeer(peerNode, peerNodeId);
+ for (OMNodeDetails peerInfo : peerNodes.values()) {
+ RaftPeer raftPeer = OzoneManagerRatisServer.createRaftPeer(peerInfo);
// Add other OM nodes belonging to the same OM service to the Ratis ring
raftPeers.add(raftPeer);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10485:
URL: https://github.com/apache/ozone/pull/10485#issuecomment-4697620525
Thanks @kerneltime for the patch, @amaliujia, @smengcl, @szetszwo for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10485:
URL: https://github.com/apache/ozone/pull/10485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. Pass Ratis peer addresses as hostnames [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #10485:
URL: https://github.com/apache/ozone/pull/10485#discussion_r3401239934
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:
##########
@@ -145,6 +145,37 @@ public void testStartOMRatisServer() throws Exception {
"Ratis Server should be in running state");
}
+ /**
+ * RaftPeer.address must always be a hostname:port string, never an
+ * IP:port string. This is what allows gRPC's DnsNameResolver to
+ * re-resolve the peer when its underlying IP changes (Kubernetes pod
+ * restart). If a resolved IP were baked in, recovery would require
+ * a full OM restart.
+ */
+ @Test
+ public void testCreateRaftPeerUsesHostnameAddress() {
+ String hostname = "om-2.om.example.svc.cluster.local";
+ int ratisPort = 9872;
Review Comment:
Done — extracted `int rpcPort = 9862` to match the existing local-variable style for `ratisPort`. Cascaded the rebase through PR-2 (#10486), PR-3 (#10487), and PR-4 (#10488).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org