You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/07 22:08:00 UTC

[jira] [Commented] (HADOOP-19987) ObserverReadProxyProvider never falls back to the active NameNode when its probing pool saturates

    [ https://issues.apache.org/jira/browse/HADOOP-19987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18112450#comment-18112450 ] 

ASF GitHub Bot commented on HADOOP-19987:
-----------------------------------------

joseluisll opened a new pull request, #8724:
URL: https://github.com/apache/hadoop/pull/8724

   ### Description of PR
   
   https://issues.apache.org/jira/browse/HADOOP-19987
   
   `getHAServiceStateWithTimeout` catches `RejectedExecutionException` to fall back to the active NameNode, but that fallback is unreachable: the probing pool was a `BlockingThreadPoolExecutorService`, whose `submit()` blocks on a semaphore instead of rejecting. A caller that should have been shed onto the active NN is parked instead — while holding the provider's monitor, since the probe runs inside the `synchronized changeProxy()`.
   
   Latent today: the only production caller is inside `changeProxy()`, so there is at most one probe in flight per provider. Reaching saturation needs `dfs.client.failover.namenode.ha-state.probe.timeout` set above its `0` default plus probes that don't return on interrupt — but when it happens the monitor is held forever.
   
   Fix: use a plain `ThreadPoolExecutor` with the same shape (4 threads, 128-deep queue, 10s idle timeout, daemon threads) and the default `AbortPolicy`. Capacity is unchanged at 132; the field type widens to `ExecutorService`. Thread names keep the `nn-ha-state-probing-pool<N>-t<M>` shape, with the per-thread counter now starting at `t0` and `<N>` counting probing pools alone.
   
   ### How was this patch tested?
   
   Two new tests in `TestObserverReadProxyProvider`, each running the call under test on its own thread with an explicit deadline so a hang names the stuck call:
   
   - `testProbingPoolRejectsOnceSaturated` — `submit()` past capacity throws instead of blocking.
   - `testFallsBackToActiveWhenProbingPoolSaturated` — the HA-state probe returns and falls back to the active NN.
   
   Both fail against the old pool, blocking for the full 20s deadline.
   
   Full CI green on the fork (`common`, `hdfs - other`, `hdfs - slow`, `hdfs-rbf`, `mr`, `other`, `yarn-server-rm` on Java 17; build-only on Java 21 and 25): https://github.com/joseluisll/hadoop/actions/runs/34147776162
   
   `common` needed a rerun for `TestZKDelegationTokenSecretManager#testNodesLoadedAfterRestart`, an unrelated pre-existing flake in `hadoop-common`; this patch only touches `hadoop-hdfs-client`.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   Contains content generated by Claude Code.
   




> ObserverReadProxyProvider never falls back to the active NameNode when its probing pool saturates
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-19987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19987
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: ha, hdfs-client
>            Reporter: Jose Luis López
>            Priority: Minor
>
> {{ObserverReadProxyProvider.getHAServiceStateWithTimeout}} catches
> {{RejectedExecutionException}} and returns null so the caller falls back to the
> active NameNode:
> {code:java}
> } catch (RejectedExecutionException e) {
>   LOG.warn("Run out of threads to submit the request to query HA state. "
>       + "Ok to return null and we will fallback to use active NN to serve "
>       + "this request.");
>   return null;
> }
> {code}
> That fallback is unreachable.
> h3. Cause
> The pool is a {{BlockingThreadPoolExecutorService}}, which wraps the real executor
> in a {{SemaphoredDelegatingExecutor}}. Its {{submit()}} blocks on
> {{queueingPermits.acquire()}} rather than rejecting, so it never throws
> {{RejectedExecutionException}} -- the handler above is dead code.
> Once the pool saturates, a caller that should have been shed onto the active NN is
> instead parked. It parks while holding this provider's monitor, since the probe is
> reached through the {{synchronized changeProxy()}}, so every other thread needing
> that monitor waits behind it.
> h3. Fix
> Use a plain {{ThreadPoolExecutor}} with the same shape -- 4 threads, 128-deep queue,
> 10s idle timeout via {{allowCoreThreadTimeOut}}, daemon threads -- and the default
> {{AbortPolicy}}, so saturation reaches the fallback that was written for it. The
> field type widens to {{ExecutorService}}.
> Behaviour is otherwise unchanged; the only observable difference is that probing
> thread names gain a numeric suffix.
> h3. Verification
> Two tests added to {{TestObserverReadProxyProvider}}. Against the current
> {{BlockingThreadPoolExecutorService}} both fail, blocking for the full 20s
> deadline:
>   submit past capacity did not return within 20000ms: it blocked instead of failing fast
>   HA state probe on a saturated pool did not return within 20000ms: it blocked instead of failing fast
> With the fix the class passes 18/18, repeated three times.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org