[jira] [Commented] (HDFS-17840) The "Nodes in Service" information is incorrect under DFS Storage Types in NameNode UI

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063951#comment-18063951 ] 

ASF GitHub Bot commented on HDFS-17840:
---------------------------------------

balodesecurity opened a new pull request, #8326:
URL: https://github.com/apache/hadoop/pull/8326

   ## Summary
   
   The **Nodes in Service** count displayed per storage type in the NameNode UI (DFS Storage Types section) could become grossly incorrect — e.g. showing 3 nodes when the cluster has 26.
   
   ### Root Cause
   
   `DatanodeStats.StorageTypeStatsMap` maintains a `StorageTypeStats` entry per storage type with an incremental `nodesInService` counter. The map entry was removed whenever `nodesInService` dropped to 0 — even when decommissioning/maintenance nodes still used the same storage type.
   
   The premature removal caused a cascade:
   
   1. Node A (last in-service node for DISK) starts decommissioning → `nodesInService` drops to 0 → **DISK entry removed**.
   2. Next heartbeat from any node recreates the entry fresh (`nodesInService = 0`).
   3. When in-service node B heartbeats: `subtract(B)` runs against the fresh entry → `nodesInService: 0→-1`. Then `add(B)` → `nodesInService: -1→0`. **B's in-service contribution is lost**.
   4. After enough such cycles the reported count is far below the real number.
   
   ### Fix
   
   Add a `totalNodes` counter to `StorageTypeStats` that tracks **all** nodes using a storage type (in-service + decommissioning + maintenance). Change the map-entry removal condition from `nodesInService == 0` to `totalNodes == 0`. An entry is now only removed when no node of any admin state still uses that storage type.
   
   **Changed files:**
   - `StorageTypeStats.java` — new `totalNodes` field; `addNode`/`subtractNode` always update it; new `getTotalNodes()` accessor
   - `DatanodeStats.java` — removal condition updated to `getTotalNodes() == 0`
   - `TestStorageTypeStatsMap.java` — 4 new unit tests (new file)
   
   ## Test plan
   
   - [x] `TestStorageTypeStatsMap` (4 tests) — PASS
     - `testBasicAddRemove` — basic correctness
     - `testEntryNotRemovedWhenDecommissioningNodeRemains` — entry survives when a decommissioning node still uses the storage type; nodesInService stays correct
     - `testEntryNotRemovedWhenLastInServiceDecommissions` — entry survives when the last in-service node decommissions; new in-service node is counted correctly
     - `testEntryRemovedOnlyWhenAllNodesGone` — entry removed only after all nodes (including decommissioning) are gone
   - [ ] Full blockmanagement test suite (CI)




> The "Nodes in Service" information is incorrect under DFS Storage Types in NameNode UI
> --------------------------------------------------------------------------------------
>
>                 Key: HDFS-17840
>                 URL: https://issues.apache.org/jira/browse/HDFS-17840
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.2.2
>            Reporter: terrytlu
>            Priority: Major
>         Attachments: before_cdp_upgrade_namenode.png, image-2025-09-23-11-34-26-420.png, latest_nodes_in_service_namenode.png
>
>
> We found in our hadoop cluster, which has 26 nodes, while in NameNode UI, only show 3 nodes
> !image-2025-09-23-11-34-26-420.png|width=720,height=253!
> the problem still exists after merge https://issues.apache.org/jira/browse/HDFS-16402 ,this metric modification involves multiple data fields and heartbeat logic, it is difficult to guarantee atomicity.
>  
> And the metrics are calculated using real-time increments and decrements, once it is corrupted, it cannot be restored. We recommend change the implementation to retrieve the data from org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager#datanodeMap to ensure that data is neither lost nor duplicated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org