You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/02/12 05:50:00 UTC

[jira] [Commented] (HDFS-17372) CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command

    [ https://issues.apache.org/jira/browse/HDFS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18058037#comment-18058037 ] 

ASF GitHub Bot commented on HDFS-17372:
---------------------------------------

hfutatzhanghb opened a new pull request, #6530:
URL: https://github.com/apache/hadoop/pull/6530

   ### Description of PR
   Refer to HDFS-17372.
   
     Recently, we met a critical problem in our production cluster which have lots of small files. In that cluster, per datanode has almost six million blocks. 
     After deleting large dir or recommision, some datanodes's SumOfActorCommandQueueLength metrics became very high as below picture shows.
   
   ![image](https://github.com/apache/hadoop/assets/25115709/a8754f73-7779-400b-835a-4c0044959f8c)
   
   After a while, we found some datanodes's write block ops became zero. That means client can not write to those datanodes. We found some logs on those datanodes:
   
   ```
   2024-02-05 17:30:11,657 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block token verification failed: op=WRITE_BLOCK, remoteAddress=/x.x.x.x:63990, message=Can't re-compute password for block_token_identifier (expiryDate=1707161411033, keyId=596193458, userId=hdfs, blockPoolId=BP-1169917699-x.x.x.x-1678688680604, blockId=11354988786, access modes=[WRITE], storageTypes= [DISK, DISK, DISK, DISK, DISK, DISK, DISK], storageIds= [DS-2664b73d-1cc9-4613-b6e3-cd58ec5ae8d5, DS-8f18621e-46c1-49b3-8bf5-7f5b8e47fe80, DS-527817e1-c5d0-4085-929f-9100a423223f, DS-d3f850c9-18ab-43cc-86d9-779b9d71a79f, DS-e4aa15ce-0c5f-4645-a582-e187e46e7183, DS-3fbbd401-58d2-49ac-a258-da7769417593, DS-d76a51e4-73db-4db6-a4e0-f82a38c1ad82]), since the required block key (keyID=596193458) doesn't exist.
   ```
   
   That is to say,  DNA_ACCESSKEYUPDATE command was blocked in CommandProcessingThread#queue. This can be deadly.
   
   So, we should guarantee that command with high-priority should be processed in time.
   
   
   




> CommandProcessingThread#queue should use LinkedBlockingDeque to prevent high priority command blocked by low priority command
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17372
>                 URL: https://issues.apache.org/jira/browse/HDFS-17372
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Haobo Zhang
>            Assignee: Haobo Zhang
>            Priority: Major
>              Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org