[jira] [Commented] (HDFS-16842) DatanodeHttpServer cannot get the new configuration of the new nameservice even after refreshNamenodes is run

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062059#comment-18062059 ] 

ASF GitHub Bot commented on HDFS-16842:
---------------------------------------

balodesecurity opened a new pull request, #8291:
URL: https://github.com/apache/hadoop/pull/8291

   ## Problem
   
   When `DataNode.refreshNamenodes()` is called to add a new nameservice it reloads configuration via `setConf(new Configuration())`, but `DatanodeHttpServer` kept holding a stale reference to the old `Configuration`. Subsequent WebHDFS requests for blocks on the newly-added nameservice would fail because the `DFSClient` created per-request could not resolve the namenode addresses for that nameservice from the stale config.
   
   Root cause: `DatanodeHttpServer.conf` and `confForCreate` were `final` fields set once at construction time. `DataNode.refreshNamenodes()` never called any update method on `httpServer`.
   
   ## Fix
   
   - Make `conf` and `confForCreate` in `DatanodeHttpServer` `volatile` (non-final) so Netty I/O threads see updates atomically.
   - Add `DatanodeHttpServer.updateConf(Configuration newConf)` that refreshes both fields and the embedded Jetty `infoServer`'s servlet-context attributes (`CONF_CONTEXT_ATTRIBUTE`, `CURRENT_CONF`).
   - Call `httpServer.updateConf(getConf())` at the end of `DataNode.refreshNamenodes()` after the new configuration has been loaded.
   
   The Netty `ChannelInitializer` creates a fresh `URLDispatcher` per connection and reads `DatanodeHttpServer.conf` at that point, so new connections automatically pick up the updated configuration without requiring a pipeline or server restart.
   
   ## Testing
   
   - Added `TestDatanodeHttpServerUpdateConf.testUpdateConfRefreshesFields` — a unit test (no cluster required) that:
     - Creates a `DatanodeHttpServer` with an initial configuration.
     - Calls `updateConf(newConf)`.
     - Asserts (via reflection) that the `conf` and `confForCreate` instance fields now reference the updated configuration.
     - Asserts that the Jetty `infoServer` servlet-context attributes have been updated.
   
   ```
   Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
   ```
   
   ## Related
   
   - JIRA: https://issues.apache.org/jira/browse/HDFS-16842
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)




> DatanodeHttpServer cannot get the new configuration of the new nameservice even after refreshNamenodes is run
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-16842
>                 URL: https://issues.apache.org/jira/browse/HDFS-16842
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.1.2
>            Reporter: Jihye Seo
>            Priority: Major
>              Labels: patch-available
>             Fix For: 3.1.2
>
>         Attachments: HDFS-16842.001.patch
>
>
> 'refreshNamenodes' should be run when a new nameservice is added.
> However, a 'DatanodeHttpServer' cannot get the new configuration of the new nameservice even after 'refreshNamenodes' is run.
> Therefore, when a client tries to access the new namespace's data with a swebhdfs or REST API request, an error occurs as follows.
> {code:java}
> Does not contain a valid host:port authority: NEW_NAMESPACE{code}
> When 'refreshNamenodes' is executed, a Datanode object reads the new configuration file and updates its configuration with 'setConf(new Configuration)'.
> {code:java}
> public void refreshNamenodes() throws IOException {
>   checkSuperuserPrivilege();
>   setConf(new Configuration());
>   refreshNamenodes(getConf());
> }{code}
> On the other hand, the configuration of a DatanodeHttpServer object remains unchanged after the DatanodeHttpServer object is created.
> When a Datanode object is created, a DatanodeHttpServer is created by copying the Datanode's configuration. Since then, there's no way to update its configuration.
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
>       final DataNode datanode,
>       final ServerSocketChannel externalHttpChannel)
>     throws IOException {
>     this.restCsrfPreventionFilter = createRestCsrfPreventionFilter(conf);
>     this.conf = conf;
>     ...
> } {code}
> This causes a DatanodeHttpServer to be unable to interpret a request for the new name service.
> Therefore, a DatanodeHttpServer also needs to update its configuration to handle a request for a new name service.
> I attached a patch based on the trunk.
>  
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org