You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "kerneltime (via GitHub)" <gi...@apache.org> on 2026/06/11 05:06:22 UTC
[PR] HDDS-15514. DNS refresh on connection failure for Client to OM [ozone]
kerneltime opened a new pull request, #10486:
URL: https://github.com/apache/ozone/pull/10486
## What changes were proposed in this pull request?
This is **PR 2 of 4** splitting [HDDS-15514](https://issues.apache.org/jira/browse/HDDS-15514) (originally proposed as a single ~160KB patch in #10473, split per @szetszwo's review feedback).
This PR fixes the **Client → OM Hadoop-RPC path** and introduces the shared infrastructure (connection-failure classifier, opt-in flag) that the remaining proxy-provider PRs in this series will reuse.
> **Stacked on [#10485](https://github.com/apache/ozone/pull/10485)** (PR-1 of 4 — Ratis peer addresses as hostnames). Please review PR-1 first; that branch is the merge base for this PR. The diff against `master` includes PR-1's hunks until PR-1 lands.
## Why this matters
`OMProxyInfo` — the per-OM peer record inside `OMFailoverProxyProvider` — constructs an `InetSocketAddress` from the configured `host:port` once at process start, then re-uses it for the proxy's lifetime. `InetSocketAddress` performs a one-shot DNS lookup at construction and **freezes** the resolved IP. When an OM pod is rescheduled in Kubernetes (or any environment where pod IPs change while DNS names remain stable), every subsequent client RPC dials the now-defunct IP forever. The failover ring walks past the broken peer to the next OM, but the broken peer never recovers without a process restart.
Note: the gRPC OM-client variant (`GrpcOMFailoverProxyProvider`) already passes a placeholder `InetSocketAddress(0)` and lets gRPC's `NameResolver` re-resolve hostnames on its own schedule — so it is unaffected by this bug and unchanged in this PR.
## What this PR does
### 1. Shared infrastructure
| Artifact | Purpose |
|---|---|
| `org.apache.hadoop.hdds.utils.ConnectionFailureUtils` | Classifies a `Throwable` (with cause-chain walk bounded to depth 16) as a connection-class failure or not. Connection-class types: `ConnectException`, `SocketTimeoutException`, `NoRouteToHostException`, `UnknownHostException`, `EOFException`, `SocketException`. Application-level errors (`OMException`, `OMNotLeaderException`, `AccessControlException`, `RetryAction`-coded responses) are NOT classified as connection failures, so DNS load is not amplified by logical failures. |
| `ozone.client.failover.resolve-needed` | Single opt-in flag for the entire DNS-refresh-on-failure feature, defaulting **`false`** so existing non-K8s deployments see zero behaviour change. Mirrors `dfs.client.failover.resolve-needed` (HDFS-14118) and `hbase.resolve.hostnames.on.failure`. |
The flag and classifier ship with this PR but are also used (in subsequent PRs) by the SCM proxy provider and the DN heartbeat task. Landing them with the OM PR follows @szetszwo's "OM first" guidance.
### 2. `OMProxyInfo` becomes refreshable
| Change | Why |
|---|---|
| Preserves the original `host:port` string (`rpcAddrStr`) alongside the resolved `InetSocketAddress` | The string is the source of truth for re-resolution; the `InetSocketAddress` is now a derived cache. |
| `refreshAddressIfChanged()` re-resolves `rpcAddrStr` outside the entry monitor, then atomically swaps the cached address, the derived `dtService`, and nulls the cached `proxy` under the monitor | DNS lookup must not happen under any lock (a slow/dead resolver would freeze concurrent readers). Building `dtService` first means a `SecurityUtil.buildTokenService` throw (rare but possible) aborts cleanly with no half-swap. |
| Old proxy is stopped via `RPC.stopProxy(staleProxy)` **outside** the monitor | Avoid blocking concurrent readers behind socket teardown. |
| `setCachedAddressForTest` (`@VisibleForTesting`) | Test seam to inject a deliberately stale cached address without a real OM peer. |
### 3. `OMFailoverProxyProviderBase.shouldRetry` calls the refresh on connection-class exceptions
After the existing `shouldFailover` filter and before the failover-index advance, when the flag is enabled and `ConnectionFailureUtils.isConnectionFailure(exception)` matches, the provider:
1. Calls `OMProxyInfo.refreshAddressIfChanged()` for the current peer.
2. On a successful refresh, returns `FAILOVER_AND_RETRY` but **pins `nextProxyIndex` to the current node** so `RetryInvocationHandler.performFailover()` does NOT advance past the just-refreshed peer. Without this pin, an N-peer HA cluster would skip the now-fixed peer for up to N-1 attempts.
### 4. `HadoopRpcOMFailoverProxyProvider` wiring
Pass the preserved hostname string into `OMProxyInfo` at construction. `HadoopRpcOMFollowerReadFailoverProxyProvider` adjusts to match.
## Atomic-replace pattern
Every refresh path in this series follows the same ordering: **build the new resource → atomically install it → close the old resource**. Never remove-then-build (a build failure would leave the system without a peer). Never close-then-build (same). For `OMProxyInfo.refreshAddressIfChanged`:
1. Re-resolve `rpcAddrStr` (no lock).
2. Build the new `dtService` (no lock).
3. Capture the stale proxy reference, swap the address/dtService/proxy=null fields under the entry monitor.
4. Stop the stale proxy outside the monitor.
## Secure-cluster prerequisite
When `ozone.client.failover.resolve-needed=true` is enabled on a Kerberos-secured cluster, operators **must** also set `hadoop.security.token.service.use_ip=false` in `core-site.xml`. The Hadoop delegation-token service identifier defaults to an `IP:port` string; after a refresh, the per-OM service identifier built from the new IP no longer matches the IP-based service captured on long-lived tokens, and token selection silently fails for the refreshed peer. With `use_ip=false` the service identifier is the stable `hostname:port`, which survives any IP change. Same prerequisite HADOOP-17068 carries. Documented inline in `ozone-default.xml`.
## How was this patch tested?
| Test class | Count | Coverage |
|---|---|---|
| `TestConnectionFailureUtils` (new) | 20 | Filter classification: bare types, `IOException`-wrapped, deeply nested cause chains (3 levels), application-level negative cases, length-2 cause-chain cycles (terminates), 1024-deep non-matching chains (cost bound). |
| `TestOMProxyInfoDnsRefresh` (new) | 4 | Per-instance refresh: no-op preserves cached proxy, swap on IP change, rebuilt proxy is dialed with the freshly-resolved address (not a sentinel), `dtService` updates. Uses the `@VisibleForTesting` setter rather than reflection. |
| `TestOMFailoverProxyProviderRefreshWired` (new) | 5 | End-to-end retry path. `SocketTimeoutException` triggers `maybeRefreshCurrentOmAddress` (the AWS EC2 silent-drop case end-to-end). `ConnectException` triggers refresh. `OMException` does NOT. Flag-off does NOT. After a successful refresh, `performFailover` stays on the same `nodeId`. |
Existing regression suites confirmed non-regressed: `TestOMFailoverProxyProvider` (8), `TestOMFailovers` (1).
## Scope of this PR
- **Includes:** Client → OM Hadoop RPC. Includes the shared `ConnectionFailureUtils` and the `ozone.client.failover.resolve-needed` flag.
- **Free transitive benefit:** OM ↔ OM control-plane RPC (`OMInterServiceProtocol`) rides on the same `OMFailoverProxyProvider`, so OM-to-OM Hadoop-RPC recovery is also fixed by this PR.
- **Out of scope (later PRs):**
- **PR-3:** OM → SCM (`SCMFailoverProxyProviderBase`, `SCMProxyInfo`). Same shape, different provider.
- **PR-4:** DN → SCM heartbeat (`EndpointStateMachine`, `SCMConnectionManager`, `StateContext`). Brings the `ozone.datanode.scm.heartbeat.address.refresh.threshold` knob.
## What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-15514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on PR #10486:
URL: https://github.com/apache/ozone/pull/10486#issuecomment-4730363665
@kerneltime did anyone else review this patch beside Copilot? I see @adoroszlai fixed one javadoc, but other than that I don't see reviews. I didn't check the change much, but it's 1000+ lines, not sure about the impact of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime merged PR #10486:
URL: https://github.com/apache/ozone/pull/10486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10486:
URL: https://github.com/apache/ozone/pull/10486#issuecomment-4687996963
Addressed Copilot's three findings in the latest force-push:
1. `OMProxyInfo.setCachedAddressForTest` now rejects null via `Objects.requireNonNull`, so a null injection surfaces immediately as a test bug rather than as a confusing NPE inside `refreshAddressIfChanged`.
2. The `LOG.warn` at the `IllegalArgumentException` catch and the one at `RPC.stopProxy` failure now pass the exception object (not `.getMessage()`), so SLF4J emits the full stack trace.
3. `testRefreshUpdatesDelegationTokenService` was indeed vacuous: the original assertion would pass even if the refresh code forgot to rebuild `dtService`. Added a `setCachedDtServiceForTest` hook, stale the field with a sentinel value before refresh, and assert post-refresh that `dtService` equals `SecurityUtil.buildTokenService(info.getAddress())`. A regression that skipped the `dtService` swap would now fail this test.
Also retitled the PR to HDDS-15531 per @szetszwo's subtask request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10486:
URL: https://github.com/apache/ozone/pull/10486#discussion_r3408509958
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ConnectionFailureUtils.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import java.io.EOFException;
+import java.net.ConnectException;
+import java.net.NoRouteToHostException;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.net.UnknownHostException;
+
+/**
+ * Shared classifier for exceptions where the cached peer IP is no longer
+ * reachable and DNS re-resolution is the only plausible recovery path.
+ * <p>
+ * Used by both {@code SCMFailoverProxyProviderBase} and
+ * {@code OMFailoverProxyProviderBase} to gate the DNS-refresh-on-failure
+ * code path so that application-level errors (NotLeader, AccessControl,
+ * OMException, RetryAction) do not trigger spurious DNS lookups.
+ * <p>
+ * The classifier must match the failure shapes seen in production
+ * Kubernetes deployments where the peer pod has been rescheduled to a
+ * new IP under a stable hostname:
+ * <ul>
+ * <li>{@link ConnectException} -- the TCP SYN was refused. Seen on
+ * OpenStack / fast-RST environments. </li>
+ * <li>{@link SocketTimeoutException} (and its IPC subclass
+ * {@code ConnectTimeoutException}) -- the SYN was dropped silently.
+ * This is the dominant failure shape on AWS EC2 / EKS where the
+ * network silently drops packets to a defunct pod IP. The PR that
+ * introduced this helper (HDDS-15514) is sold on this case; it
+ * must be in the filter. </li>
+ * <li>{@link NoRouteToHostException} -- routing table no longer
+ * reaches the cached IP. </li>
+ * <li>{@link UnknownHostException} -- the hostname itself failed to
+ * resolve at the time the IPC layer reconstructed the address. </li>
+ * <li>{@link EOFException} -- a load balancer or iptables RST closed
+ * the half-open connection cleanly. Common in Kubernetes when an
+ * IP is reassigned to an unrelated pod that rejects the RPC
+ * handshake. </li>
+ * <li>{@link SocketException} (e.g. "Connection reset") -- the peer
+ * sent RST mid-stream. </li>
+ * </ul>
+ * The walk is bounded to {@value #MAX_CAUSE_DEPTH} levels to defend
+ * against cause chains that have been constructed (in violation of
+ * {@code Throwable.initCause}'s contract) into a cycle of length > 1.
Review Comment:
```
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ConnectionFailureUtils.java:61: warning - invalid usage of tag >
```
```suggestion
* {@code Throwable.initCause}'s contract) into a cycle of length > 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10486:
URL: https://github.com/apache/ozone/pull/10486#issuecomment-4705631322
Thank you @adoroszlai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #10486:
URL: https://github.com/apache/ozone/pull/10486#discussion_r3430323286
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java:
##########
@@ -604,6 +604,19 @@ public final class OzoneConfigKeys {
public static final boolean OZONE_JVM_NETWORK_ADDRESS_CACHE_ENABLED_DEFAULT =
true;
+ /**
+ * When true, RPC clients (DN heartbeat, OM client, SCM client) re-resolve
+ * cached hostnames on connection failure and rebuild the proxy if the
+ * resolved IP has changed. Set to true in environments where server pod
+ * IPs may change while DNS names remain stable, such as Kubernetes.
+ * Default false preserves pre-fix behavior. Mirrors the design intent of
+ * HADOOP-17068 / HDFS-14118.
+ */
+ public static final String OZONE_CLIENT_FAILOVER_RESOLVE_NEEDED_KEY =
+ "ozone.client.failover.resolve-needed";
Review Comment:
Move it next to "ozone.client.failover.max.attempts".
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/ConnectionFailureUtils.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import java.io.EOFException;
+import java.net.ConnectException;
+import java.net.NoRouteToHostException;
+import java.net.SocketException;
+import java.net.SocketTimeoutException;
+import java.net.UnknownHostException;
+
+/**
+ * Shared classifier for exceptions where the cached peer IP is no longer
+ * reachable and DNS re-resolution is the only plausible recovery path.
+ * <p>
+ * Used by both {@code SCMFailoverProxyProviderBase} and
+ * {@code OMFailoverProxyProviderBase} to gate the DNS-refresh-on-failure
+ * code path so that application-level errors (NotLeader, AccessControl,
+ * OMException, RetryAction) do not trigger spurious DNS lookups.
+ * <p>
+ * The classifier must match the failure shapes seen in production
+ * Kubernetes deployments where the peer pod has been rescheduled to a
+ * new IP under a stable hostname:
+ * <ul>
+ * <li>{@link ConnectException} -- the TCP SYN was refused. Seen on
+ * OpenStack / fast-RST environments. </li>
+ * <li>{@link SocketTimeoutException} (and its IPC subclass
+ * {@code ConnectTimeoutException}) -- the SYN was dropped silently.
+ * This is the dominant failure shape on AWS EC2 / EKS where the
+ * network silently drops packets to a defunct pod IP. The PR that
+ * introduced this helper (HDDS-15514) is sold on this case; it
+ * must be in the filter. </li>
+ * <li>{@link NoRouteToHostException} -- routing table no longer
+ * reaches the cached IP. </li>
+ * <li>{@link UnknownHostException} -- the hostname itself failed to
+ * resolve at the time the IPC layer reconstructed the address. </li>
+ * <li>{@link EOFException} -- a load balancer or iptables RST closed
+ * the half-open connection cleanly. Common in Kubernetes when an
+ * IP is reassigned to an unrelated pod that rejects the RPC
+ * handshake. </li>
+ * <li>{@link SocketException} (e.g. "Connection reset") -- the peer
+ * sent RST mid-stream. </li>
+ * </ul>
+ * The walk is bounded to {@value #MAX_CAUSE_DEPTH} levels to defend
+ * against cause chains that have been constructed (in violation of
+ * {@code Throwable.initCause}'s contract) into a cycle of length > 1.
+ */
+public final class ConnectionFailureUtils {
+
+ /**
+ * Maximum depth of the {@code Throwable.getCause()} chain we walk
+ * before giving up. Matches Hadoop's own walkers in
+ * {@code RemoteException} handling.
+ */
+ static final int MAX_CAUSE_DEPTH = 16;
+
+ private ConnectionFailureUtils() {
+ }
+
+ /**
+ * Returns true when any link in {@code t}'s cause chain (up to
+ * {@link #MAX_CAUSE_DEPTH} levels) is one of the connection-class
+ * exceptions documented on this class.
+ *
+ * @param t the throwable to classify. {@code null} returns false.
+ */
+ public static boolean isConnectionFailure(Throwable t) {
+ Throwable cause = t;
+ for (int depth = 0; cause != null && depth < MAX_CAUSE_DEPTH; depth++) {
+ if (cause instanceof ConnectException
+ || cause instanceof SocketTimeoutException
+ || cause instanceof NoRouteToHostException
Review Comment:
ConnectException and NoRouteToHostException are subclasses of SocketException. Let remove them and add a comment.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMFailoverProxyProviderBase.java:
##########
@@ -477,4 +509,63 @@ public static ReadException getReadException(Exception exception) {
protected ConfigurationSource getConf() {
return conf;
}
+
+ /**
+ * Asks the current proxy's {@link OMProxyInfo} to re-resolve its
+ * configured hostname. If DNS now returns a different IP, the
+ * OMProxyInfo replaces its cached address and discards the cached
+ * proxy so the next dial happens against the new IP.
+ * <p>
+ * Calls {@link #onAddressRefreshed(String)} when a swap occurs so
+ * subclasses (specifically {@code HadoopRpcOMFailoverProxyProvider})
+ * can refresh derived state such as the aggregated delegation-token
+ * service identifier.
+ * <p>
+ * The DNS lookup performed inside {@link OMProxyInfo#refreshAddressIfChanged}
+ * is run OUTSIDE this provider's monitor. {@link OMProxyInfo} maintains
+ * its own monitor for the swap commit; if we held the provider monitor
+ * across the resolve, a slow / dead resolver would freeze every
+ * concurrent caller of synchronized provider methods (e.g.
+ * {@link #performFailover}, {@link #selectNextOmProxy}). The provider
+ * monitor is only re-acquired briefly to invoke the refresh hook.
+ *
+ * @return true if a swap actually happened.
+ */
+ @VisibleForTesting
+ boolean maybeRefreshCurrentOmAddress() {
+ final String nodeId;
+ final OMProxyInfo<T> info;
+ synchronized (this) {
+ nodeId = getCurrentProxyOMNodeId();
+ if (nodeId == null) {
+ return false;
+ }
+ info = omProxies.get(nodeId);
+ if (info == null) {
+ return false;
+ }
+ }
Review Comment:
- nodeId is never null
- info is never null
- getCurrentProxyOMNodeId() is already synchronized
- the omProxies map is unmodifiable
So, I suggest the change below:
```java
final String nodeId = Objects.requireNonNull(getCurrentProxyOMNodeId(), "Current proxy node id is null");
final OMProxyInfo<T> info = Objects.requireNonNull(omProxies.get(nodeId), "Current proxy info is null");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15531. DNS refresh on connection failure for Client to OM [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #10486:
URL: https://github.com/apache/ozone/pull/10486#issuecomment-4738101498
Ack. Will address these comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15514. DNS refresh on connection failure for Client to OM [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #10486:
URL: https://github.com/apache/ozone/pull/10486#discussion_r3397197937
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMProxyInfo.java:
##########
@@ -106,10 +135,94 @@ public synchronized void createProxyIfNeeded(CheckedFunction<InetSocketAddress,
}
/**
- * A {@link OMProxyInfo} map with a particular order,
+ * Re-resolve {@link #rpcAddrStr} via DNS and, if the resolved IP has
+ * changed, replace the cached {@link #rpcAddr} (and the derived
+ * delegation-token service) and discard the cached proxy so that the
+ * next {@link #createProxyIfNeeded} call dials the new IP. The stale
+ * RPC proxy is closed via {@link RPC#stopProxy} so the underlying
+ * Hadoop {@code Client} connection thread and authenticated SASL
+ * session against the gone-away peer are not leaked.
+ * <p>
+ * Returns true when a swap occurred. Off the failure path this is a
+ * no-op (returns false): unchanged IP, unresolved lookup, or
+ * malformed host string.
+ * <p>
+ * The DNS lookup and the {@code RPC.stopProxy} call are performed
+ * outside the entry monitor so that a slow / dead resolver or a
+ * blocking proxy teardown does not freeze concurrent readers of
+ * {@link #getAddress()} / {@link #getProxy()}.
+ */
+ public boolean refreshAddressIfChanged() {
+ final InetSocketAddress refreshed;
+ try {
+ refreshed = NetUtils.createSocketAddr(rpcAddrStr);
+ } catch (IllegalArgumentException ex) {
+ LOG.warn("Failed to re-resolve OM address {}: {}", rpcAddrStr,
+ ex.getMessage());
Review Comment:
Similarly here, logging only ex.getMessage() loses the stack trace for malformed addresses / resolution errors. Include the exception so operators can see the full failure details.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMProxyInfo.java:
##########
@@ -83,11 +103,20 @@ public String getAddressString() {
return rpcAddrStr;
}
- public InetSocketAddress getAddress() {
+ public synchronized InetSocketAddress getAddress() {
return rpcAddr;
}
- public Text getDelegationTokenService() {
+ /**
+ * Test-only: inject a deliberately stale cached address to drive
+ * the DNS-refresh code path without standing up a real OM.
+ */
+ @VisibleForTesting
+ synchronized void setCachedAddressForTest(InetSocketAddress address) {
+ this.rpcAddr = address;
+ }
Review Comment:
The test-only setter currently allows null, which can lead to a confusing NPE later in refreshAddressIfChanged() when it dereferences rpcAddr. Enforcing non-null here keeps OMProxyInfo invariants intact and makes failures clearer.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/ha/OMProxyInfo.java:
##########
@@ -106,10 +135,94 @@ public synchronized void createProxyIfNeeded(CheckedFunction<InetSocketAddress,
}
/**
- * A {@link OMProxyInfo} map with a particular order,
+ * Re-resolve {@link #rpcAddrStr} via DNS and, if the resolved IP has
+ * changed, replace the cached {@link #rpcAddr} (and the derived
+ * delegation-token service) and discard the cached proxy so that the
+ * next {@link #createProxyIfNeeded} call dials the new IP. The stale
+ * RPC proxy is closed via {@link RPC#stopProxy} so the underlying
+ * Hadoop {@code Client} connection thread and authenticated SASL
+ * session against the gone-away peer are not leaked.
+ * <p>
+ * Returns true when a swap occurred. Off the failure path this is a
+ * no-op (returns false): unchanged IP, unresolved lookup, or
+ * malformed host string.
+ * <p>
+ * The DNS lookup and the {@code RPC.stopProxy} call are performed
+ * outside the entry monitor so that a slow / dead resolver or a
+ * blocking proxy teardown does not freeze concurrent readers of
+ * {@link #getAddress()} / {@link #getProxy()}.
+ */
+ public boolean refreshAddressIfChanged() {
+ final InetSocketAddress refreshed;
+ try {
+ refreshed = NetUtils.createSocketAddr(rpcAddrStr);
+ } catch (IllegalArgumentException ex) {
+ LOG.warn("Failed to re-resolve OM address {}: {}", rpcAddrStr,
+ ex.getMessage());
+ return false;
+ }
+ if (refreshed.isUnresolved()) {
+ LOG.warn("OM hostname {} re-resolved to an unresolved address; "
+ + "leaving cached entry in place.", rpcAddrStr);
+ return false;
+ }
+ // Compute the new delegation-token service identifier OUTSIDE the
+ // entry monitor. SecurityUtil.buildTokenService is unlikely to throw
+ // for a resolved address, but if it ever did inside the swap block
+ // we'd be left with rpcAddr=new but dtService=old and proxy=non-null
+ // pointing at the old IP -- and the equality short-circuit at the
+ // top of the synchronized block below would skip every subsequent
+ // refresh attempt because rpcAddr already matches the new IP.
+ // Building first means a throw here aborts the whole refresh with
+ // no state change.
+ final Text newDtService = SecurityUtil.buildTokenService(refreshed);
+ final T staleProxy;
+ final InetSocketAddress old;
+ synchronized (this) {
+ // Null-safe IP comparison. The constructor accepts (with a warn)
+ // an unresolved rpcAddr -- in that case rpcAddr.getAddress() is
+ // null, and a successful re-resolution is genuinely a change so
+ // we MUST proceed to swap rather than NPE on .equals().
+ InetAddress cachedIp = rpcAddr.getAddress();
+ InetAddress refreshedIp = refreshed.getAddress();
+ if (cachedIp != null && refreshedIp != null
+ && refreshedIp.equals(cachedIp)) {
+ return false;
+ }
+ old = rpcAddr;
+ staleProxy = this.proxy;
+ this.rpcAddr = refreshed;
+ this.dtService = newDtService;
+ this.proxy = null;
+ }
+ if (staleProxy != null) {
+ try {
+ RPC.stopProxy(staleProxy);
+ } catch (RuntimeException stopEx) {
+ LOG.warn("Failed to stop stale OM proxy for nodeId {}: {}",
+ nodeId, stopEx.getMessage());
Review Comment:
Logging only the exception message drops the stack trace, which can make proxy-stop failures harder to diagnose. Log the exception itself so the stack trace is preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org