You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yue Wang (Jira)" <ji...@apache.org> on 2026/09/28 02:12:00 UTC
[jira] [Updated] (HDFS-17982) RBF should return StandbyException when no Active NameNode is known
[ https://issues.apache.org/jira/browse/HDFS-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yue Wang updated HDFS-17982:
----------------------------
Description:
A Router may have an incomplete view of the NameNodes in a nameservice and know only about Observer NameNodes. When a request must be forwarded to the Active NameNode, the Router skips those Observer candidates and makes no RPC calls. As a result, the collected IOException map is empty.
Exceptions:
{code:java}
Caused by: org.apache.hadoop.ipc.RemoteException: No namenode available to invoke create [/xxx/xxx/xxx, { masked: rw-rw-r--, unmasked: rw-rw-rw- }, DFSClient_NONMAPREDUCE_1160536643_94, [CREATE], true, 3, 268435456, , ] in [ip-xxx->tl11:ob21:ip-xxx-OBSERVER, ip-XXX->tl11:ob31:ip-XXX-OBSERVER] from ip-xxx
at org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.handlerAllNamenodeFail(RouterRpcClient.java:677)
at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.lambda$invokeMethodAsync$10(RouterAsyncRpcClient.java:312)
at org.apache.hadoop.hdfs.server.federation.router.async.utils.ApplyFunction.lambda$apply$0(ApplyFunction.java:65)
at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684)
at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662)
at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2200)
at org.apache.hadoop.hdfs.server.federation.router.async.utils.ApplyFunction.apply(ApplyFunction.java:63)
at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncUtil.asyncApply(AsyncUtil.java:196)
at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.invokeMethodAsync(RouterAsyncRpcClient.java:308)
at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.lambda$invokeMethod$3(RouterAsyncRpcClient.java:215)
at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncApplyFunction.async(AsyncApplyFunction.java:107)
at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncApplyFunction.lambda$apply$1(AsyncApplyFunction.java:156)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583) {code}
The current failure handling can treat this empty map as if every candidate failed to connect and return ConnectException. Instead, the Router should return StandbyException so the client can retry the request through another Router.
Add a regression test that invokes a request with only Observer candidates when Observer reads are disabled, and verifies that the Router returns StandbyException when no NameNode RPC is attempted.
> RBF should return StandbyException when no Active NameNode is known
> -------------------------------------------------------------------
>
> Key: HDFS-17982
> URL: https://issues.apache.org/jira/browse/HDFS-17982
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.3.4
> Reporter: Yue Wang
> Assignee: Yue Wang
> Priority: Major
>
> A Router may have an incomplete view of the NameNodes in a nameservice and know only about Observer NameNodes. When a request must be forwarded to the Active NameNode, the Router skips those Observer candidates and makes no RPC calls. As a result, the collected IOException map is empty.
> Exceptions:
> {code:java}
> Caused by: org.apache.hadoop.ipc.RemoteException: No namenode available to invoke create [/xxx/xxx/xxx, { masked: rw-rw-r--, unmasked: rw-rw-rw- }, DFSClient_NONMAPREDUCE_1160536643_94, [CREATE], true, 3, 268435456, , ] in [ip-xxx->tl11:ob21:ip-xxx-OBSERVER, ip-XXX->tl11:ob31:ip-XXX-OBSERVER] from ip-xxx
> at org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.handlerAllNamenodeFail(RouterRpcClient.java:677)
> at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.lambda$invokeMethodAsync$10(RouterAsyncRpcClient.java:312)
> at org.apache.hadoop.hdfs.server.federation.router.async.utils.ApplyFunction.lambda$apply$0(ApplyFunction.java:65)
> at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684)
> at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662)
> at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2200)
> at org.apache.hadoop.hdfs.server.federation.router.async.utils.ApplyFunction.apply(ApplyFunction.java:63)
> at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncUtil.asyncApply(AsyncUtil.java:196)
> at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.invokeMethodAsync(RouterAsyncRpcClient.java:308)
> at org.apache.hadoop.hdfs.server.federation.router.async.RouterAsyncRpcClient.lambda$invokeMethod$3(RouterAsyncRpcClient.java:215)
> at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncApplyFunction.async(AsyncApplyFunction.java:107)
> at org.apache.hadoop.hdfs.server.federation.router.async.utils.AsyncApplyFunction.lambda$apply$1(AsyncApplyFunction.java:156)
> at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583) {code}
> The current failure handling can treat this empty map as if every candidate failed to connect and return ConnectException. Instead, the Router should return StandbyException so the client can retry the request through another Router.
> Add a regression test that invokes a request with only Observer candidates when Observer reads are disabled, and verifies that the Router returns StandbyException when no NameNode RPC is attempted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org