You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jose Luis López (Jira)" <ji...@apache.org> on 2026/09/07 16:31:00 UTC

[jira] [Updated] (HADOOP-19987) ObserverReadProxyProvider never falls back to the active NameNode when its probing pool saturates

     [ https://issues.apache.org/jira/browse/HADOOP-19987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jose Luis López updated HADOOP-19987:
-------------------------------------
    Description: 
{{ObserverReadProxyProvider.getHAServiceStateWithTimeout}} catches
{{RejectedExecutionException}} and returns null so the caller falls back to the
active NameNode:

{code:java}
} catch (RejectedExecutionException e) {
  LOG.warn("Run out of threads to submit the request to query HA state. "
      + "Ok to return null and we will fallback to use active NN to serve "
      + "this request.");
  return null;
}
{code}

That fallback is unreachable.

h3. Cause

The pool is a {{BlockingThreadPoolExecutorService}}, which wraps the real executor
in a {{SemaphoredDelegatingExecutor}}. Its {{submit()}} blocks on
{{queueingPermits.acquire()}} rather than rejecting, so it never throws
{{RejectedExecutionException}} -- the handler above is dead code.

Once the pool saturates, a caller that should have been shed onto the active NN is
instead parked. It parks while holding this provider's monitor, since the probe is
reached through the {{synchronized changeProxy()}}, so every other thread needing
that monitor waits behind it.

h3. Fix

Use a plain {{ThreadPoolExecutor}} with the same shape -- 4 threads, 128-deep queue,
10s idle timeout via {{allowCoreThreadTimeOut}}, daemon threads -- and the default
{{AbortPolicy}}, so saturation reaches the fallback that was written for it. The
field type widens to {{ExecutorService}}.

Behaviour is otherwise unchanged; the only observable difference is that probing
thread names gain a numeric suffix.

  was:
pending

 


> ObserverReadProxyProvider never falls back to the active NameNode when its probing pool saturates
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-19987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19987
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: ha, hdfs-client
>            Reporter: Jose Luis López
>            Priority: Minor
>
> {{ObserverReadProxyProvider.getHAServiceStateWithTimeout}} catches
> {{RejectedExecutionException}} and returns null so the caller falls back to the
> active NameNode:
> {code:java}
> } catch (RejectedExecutionException e) {
>   LOG.warn("Run out of threads to submit the request to query HA state. "
>       + "Ok to return null and we will fallback to use active NN to serve "
>       + "this request.");
>   return null;
> }
> {code}
> That fallback is unreachable.
> h3. Cause
> The pool is a {{BlockingThreadPoolExecutorService}}, which wraps the real executor
> in a {{SemaphoredDelegatingExecutor}}. Its {{submit()}} blocks on
> {{queueingPermits.acquire()}} rather than rejecting, so it never throws
> {{RejectedExecutionException}} -- the handler above is dead code.
> Once the pool saturates, a caller that should have been shed onto the active NN is
> instead parked. It parks while holding this provider's monitor, since the probe is
> reached through the {{synchronized changeProxy()}}, so every other thread needing
> that monitor waits behind it.
> h3. Fix
> Use a plain {{ThreadPoolExecutor}} with the same shape -- 4 threads, 128-deep queue,
> 10s idle timeout via {{allowCoreThreadTimeOut}}, daemon threads -- and the default
> {{AbortPolicy}}, so saturation reaches the fallback that was written for it. The
> field type widens to {{ExecutorService}}.
> Behaviour is otherwise unchanged; the only observable difference is that probing
> thread names gain a numeric suffix.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org