You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Aleksei Ieshin (Jira)" <ji...@apache.org> on 2026/08/14 04:48:00 UTC
[jira] [Updated] (HDFS-17964) HDFS delegation token not found in cache errors seen on Observer NameNodes
[ https://issues.apache.org/jira/browse/HDFS-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksei Ieshin updated HDFS-17964:
----------------------------------
Description:
Observer NameNodes fatally reject delegation tokens they have not yet tailed.
`DelegationTokenSecretManager.retriableRetrievePassword` and `FSNamesystem.verifyToken` wrap `InvalidToken` in a `RetriableException` only when `inTransitionToActive()`. An observer in steady state never is, so a token the active issued moments ago fails with
{code:java}
InvalidToken: Token for real user: <user>, can't be found in cache
{code}
HDFS-5322 added that guard in 2.3.0, before Observer NameNodes existed (HDFS-12943). Widening it to cover observers fixes both paths.
h6. Details:
The common trigger is WebHDFS op=OPEN, the one operation with a second hop: The common trigger is WebHDFS op=OPEN, the one operation with a second hop: the NameNode redirects to a DataNode, and that DataNode validates the token against a NameNode chosen by its own ConfiguredFailoverProxyProvider. Nothing the client configures influences that choice, so in a three-NameNode HA setup with one observer a substantial fraction of reads land on the observer and fail with HTTP 403.
Fix: treat an observer as possibly behind on tailing at both sites, so a failed lookup yields RetriableException. Callers already handle it - that is what HDFS-5322 relies on. No config, no new mechanism.
was:
Observer NameNodes fatally reject delegation tokens they have not yet tailed.
`DelegationTokenSecretManager.retriableRetrievePassword` and `FSNamesystem.verifyToken` wrap `InvalidToken` in a `RetriableException` only when `inTransitionToActive()`. An observer in steady state never is, so a token the active issued moments ago fails with
```
InvalidToken: Token for real user: <user>, can't be found in cache
```
HDFS-5322 added that guard in 2.3.0, before Observer NameNodes existed (HDFS-12943). Widening it to cover observers fixes both paths.
h6. Details:
The common trigger is WebHDFS op=OPEN, the one operation with a second hop: The common trigger is WebHDFS op=OPEN, the one operation with a second hop: the NameNode redirects to a DataNode, and that DataNode validates the token against a NameNode chosen by its own ConfiguredFailoverProxyProvider. Nothing the client configures influences that choice, so in a three-NameNode HA setup with one observer a substantial fraction of reads land on the observer and fail with HTTP 403.
Fix: treat an observer as possibly behind on tailing at both sites, so a failed lookup yields RetriableException. Callers already handle it - that is what HDFS-5322 relies on. No config, no new mechanism.
> HDFS delegation token not found in cache errors seen on Observer NameNodes
> --------------------------------------------------------------------------
>
> Key: HDFS-17964
> URL: https://issues.apache.org/jira/browse/HDFS-17964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Reporter: Aleksei Ieshin
> Priority: Major
>
> Observer NameNodes fatally reject delegation tokens they have not yet tailed.
> `DelegationTokenSecretManager.retriableRetrievePassword` and `FSNamesystem.verifyToken` wrap `InvalidToken` in a `RetriableException` only when `inTransitionToActive()`. An observer in steady state never is, so a token the active issued moments ago fails with
> {code:java}
> InvalidToken: Token for real user: <user>, can't be found in cache
> {code}
> HDFS-5322 added that guard in 2.3.0, before Observer NameNodes existed (HDFS-12943). Widening it to cover observers fixes both paths.
> h6. Details:
> The common trigger is WebHDFS op=OPEN, the one operation with a second hop: The common trigger is WebHDFS op=OPEN, the one operation with a second hop: the NameNode redirects to a DataNode, and that DataNode validates the token against a NameNode chosen by its own ConfiguredFailoverProxyProvider. Nothing the client configures influences that choice, so in a three-NameNode HA setup with one observer a substantial fraction of reads land on the observer and fail with HTTP 403.
>
> Fix: treat an observer as possibly behind on tailing at both sites, so a failed lookup yields RetriableException. Callers already handle it - that is what HDFS-5322 relies on. No config, no new mechanism.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org