You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ruiliang (Jira)" <ji...@apache.org> on 2024/11/22 08:29:00 UTC
[jira] [Commented] (HDFS-17589) hdfs EC data new blk reconstruct old blk not delete
[ https://issues.apache.org/jira/browse/HDFS-17589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17900282#comment-17900282 ]
ruiliang commented on HDFS-17589:
---------------------------------
[~yuyanlei] I used a clunky way to clear this blk file
1:Create an API on the datanode to check disk blk file info. python file:[^DiskBlkAPI.py]
2:The client analyzes the EC reconstruct striped block log (which needs to be collected separately), compares the disk blk on the ip with the namenode blk, and then goes back if there are too many BLKS. python file: [^FindDirtyBlk.py]
^case:^
^!https://rte.weiyun.baidu.com/wiki/attach/image/api/imageDownloadAddress?attachId=39900719750b4e0a8041ef4974cbf0b5&docGuid=8WuVFWCqJw3yXz&sign=eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiYXBwSWQiOjEsInVpZCI6IjNOVEZhejVrT1giLCJkb2NJZCI6IjhXdVZGV0NxSnczeVh6In0..TpGVchjIee0exDgK.Tlrt3kCoLtePC8ZLP28r7QkUNT9WgZa36FzOb3lhQ-QhxyIsSAMVS2CGnWDX8GL4MDgX8q0rudXldMmGenw9ewod52s26beRINgPw6zQ9aPE3luENVW4Ms_rwagDAUe4VKSKDJK-HNNH_vInMuR-nJMBvy_rfkxkaLJIl3rlfjwLy1I7GxaDuBep9ju0aYhXdCToDYDQrZW0vf76XGBjAfFUzQ.RCh7lfsQ_p3bJyOL2jfZIQ!^
^After treatment:^
^!https://rte.weiyun.baidu.com/wiki/attach/image/api/imageDownloadAddress?attachId=f50c1a93291949ec9a1a8de8e935519b&docGuid=8WuVFWCqJw3yXz&sign=eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiYXBwSWQiOjEsInVpZCI6IjNOVEZhejVrT1giLCJkb2NJZCI6IjhXdVZGV0NxSnczeVh6In0..TpGVchjIee0exDgK.Tlrt3kCoLtePC8ZLP28r7QkUNT9WgZa36FzOb3lhQ-QhxyIsSAMVS2CGnWDX8GL4MDgX8q0rudXldMmGenw9ewod52s26beRINgPw6zQ9aPE3luENVW4Ms_rwagDAUe4VKSKDJK-HNNH_vInMuR-nJMBvy_rfkxkaLJIl3rlfjwLy1I7GxaDuBep9ju0aYhXdCToDYDQrZW0vf76XGBjAfFUzQ.RCh7lfsQ_p3bJyOL2jfZIQ!^
> hdfs EC data new blk reconstruct old blk not delete
> ------------------------------------------------------
>
> Key: HDFS-17589
> URL: https://issues.apache.org/jira/browse/HDFS-17589
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.1.1
> Reporter: ruiliang
> Priority: Major
> Attachments: DiskBlkAPI.py, FindDirtyBlk.py
>
>
> The reason is that the cluster was faulty before, and Datanodes kept losing connections and recovering, resulting in a lot of EC data reconstruct, but a lot of old blk failed to clean up correctly. Has this been repaired? What patch do I need to add, thank you
> The following is a detailed check log
>
> ok: blk_-9223372036371044652 in 10.12.66.225
> {color:#de350b}error: blk_-9223372036371044652 in 10.12.66.154(/data3/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652) {color}
> {color:#de350b}Why didn't you delete it?{color}
>
> {code:java}
> ====datanode delete data ec blk ?
> grep blk_-9223372036371044656 hadoop-hdfs-root-datanode-fs-hiido-dn-12-66-111.hiido.host.xxx.com.log
> 2024-07-18 17:25:07,879 INFO datanode.DataNode (DataXceiver.java:writeBlock(738)) - Receiving BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036371044656_1688858793 src: /10.12.66.111:25066 dest: /10.12.66.111:1019
> 2024-07-18 17:25:17,396 INFO datanode.DataNode (StripedBlockReconstructor.java:run(86)) - ok EC reconstruct striped block: BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036371044656_1688858793 blockId: -9223372036371044656
> 2024-07-18 17:25:17,396 INFO datanode.DataNode (DataXceiver.java:writeBlock(914)) - Received BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036371044656_1688858793 src: /10.12.66.111:25066 dest: /10.12.66.111:1019 of size 193986560
> 2024-07-18 17:25:25,465 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_-9223372036371044656_1688858793 replica FinalizedReplica, blk_-9223372036371044656_1688858793, FINALIZED
> getBlockURI() = file:/data4/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044656 for deletion
> 2024-07-18 17:25:25,746 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1822992414-10.12.65.48-1660893388633 blk_-9223372036371044656_1688858793 URI file:/data4/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044656=============my config
> dfs.blockreport.intervalMsec =21600000============namenode3 log
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-18 04:34:39,523 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-18 04:34:40,131 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-18 10:34:38,950 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-18 10:34:39,559 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log:2024-07-18 16:34:38,564 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log:2024-07-18 16:34:39,190 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 04:34:39,462 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 04:34:40,083 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 10:34:39,686 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 10:34:40,295 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 16:34:39,667 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 16:34:40,301 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 22:34:38,187 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn3.hiido.host.xxxyy.com.log.1:2024-07-17 22:34:38,794 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> =====namenode2 log active
> grep blk_-9223372036371044656 hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn2.hiido.host.xxx.com.log.10
> 2024-07-18 17:25:04,786 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> 2024-07-18 17:25:05,703 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> ========namenode2 log
> root@fs-hiido-yycluster06-yynn1:/data/logs/hadoop/hdfs# grep blk_-9223372036371044656 hadoop-hdfs-namenode-fs-hiido-yycluster06-yynn1.hiido.host.xxx.com.log
> 2024-07-18 07:20:41,525 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> 2024-07-18 07:20:42,049 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> 2024-07-18 13:20:40,726 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> 2024-07-18 13:20:41,251 WARN BlockStateChange (BlockManager.java:addStoredBlock(3238)) - BLOCK* addStoredBlock: block blk_-9223372036371044656_1688858793 moved to storageType DISK on node 10.12.66.154:1019
> ================
> hdfs fsck -fs hdfs://yycluster06 -blockId blk_-9223372036371044656
> Connecting to namenode via http://fs-hiido-yycluster06-yynn2.hiido.host.xxx.com:50070/fsck?ugi=hdfs&blockId=blk_-9223372036371044656+&path=%2F
> FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 at Thu Jul 18 17:57:06 CST 2024Block Id: blk_-9223372036371044656
> Block belongs to: /hive_warehouse/yydw.db/dwv_event_detail_mob_quality_day/dt=2021-08-10/product_id=171/part-00210-f6fac929-f172-45cf-9fb7-0aa2f68c545e.c000.gz
> No. of Expected Replica: 5
> No. of live Replica: 5
> No. of excess Replica: 0
> No. of stale Replica: 0
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> Block replica on datanode/rack: fs-hiido-dn-12-66-225.hiido.host.xxxyy.com/4F08-02-03 is HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-67-38.hiido.host.xxxyy.com/4F08-02-15 is HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-66-191.hiido.host.xxx.com/4F08-12-06 is HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-67-5.hiido.host.xxxyy.com/4F08-12-09 is HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-66-154.hiido.host.xxxyy.com/4F08-02-13 is HEALTHYhdfs fsck -fs hdfs://yycluster06 /hive_warehouse/yydw.db/dwv_event_detail_mob_quality_day/dt=2021-08-10/product_id=171/part-00210-f6fac929-f172-45cf-9fb7-0aa2f68c545e.c000.gz -files -blocks -locations
> /hive_warehouse/yydw.db/dwv_event_detail_mob_quality_day/dt=2021-08-10/product_id=171/part-00210-f6fac929-f172-45cf-9fb7-0aa2f68c545e.c000.gz 581647843 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s): OK
> 0. BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036371044656_1688858793 len=581647843 Live_repl=5 [
> blk_-9223372036371044656:DatanodeInfoWithStorage[10.12.66.154:1019,DS-4b66fe61-93ca-4f8d-8fe0-e00f2ed09e82,DISK],
> blk_-9223372036371044655:DatanodeInfoWithStorage[10.12.67.5:1019,DS-af799695-8e9b-4884-a741-7a0742db6f79,DISK],
> blk_-9223372036371044654:DatanodeInfoWithStorage[10.12.66.191:1019,DS-cef171fa-8e8e-43bb-9fd1-aab84ea046cd,DISK],
> blk_-9223372036371044653:DatanodeInfoWithStorage[10.12.67.38:1019,DS-d2346a26-14c4-41e9-b349-e198ab5b684e,DISK],
> blk_-9223372036371044652:DatanodeInfoWithStorage[10.12.66.225:1019,DS-13c19c1d-9221-45cc-919f-b54ada9cab15,DISK]]
> ========================================================================================================================================
> root@fs-hiido-dn-12-66-154:/data/logs/hadoop/hdfs# ll /data*/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir*/subdir*/blk_-922337203637104465*
> -rw-r--r-- 1 hdfs hdfs 193986560 Feb 26 19:54 /data12/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044656
> -rw-r--r-- 1 hdfs hdfs 1515527 Feb 26 19:54 /data12/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044656_1688858793.meta
> -rw-r--r-- 1 hdfs hdfs 193986560 Feb 27 21:49 /data3/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652
> -rw-r--r-- 1 hdfs hdfs 1515527 Feb 27 21:49 /data3/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652_1688858793.meta
> root@fs-hiido-dn-12-67-5:/home/liangrui# ll /data*/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir*/subdir*/blk_-922337203637104465*
> -rw-r--r-- 1 hdfs hdfs 193986560 Nov 29 2023 /data12/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044655
> -rw-r--r-- 1 hdfs hdfs 1515527 Nov 29 2023 /data12/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044655_1688858793.meta
> root@fs-hiido-dn-12-66-191:/home/liangrui# ll /data*/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir*/subdir*/blk_-922337203637104465*
> -rw-r--r-- 1 hdfs hdfs 193674723 Aug 30 2023 /data4/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044654
> -rw-r--r-- 1 hdfs hdfs 1513091 Aug 30 2023 /data4/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044654_1688858793.meta
> root@fs-hiido-dn-12-67-38:/home/liangrui# ll /data*/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir*/subdir*/blk_-922337203637104465*
> -rw-r--r-- 1 hdfs hdfs 193986560 Apr 19 18:20 /data1/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044653
> -rw-r--r-- 1 hdfs hdfs 1515527 Apr 19 18:20 /data1/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044653_1688858793.meta
> root@fs-hiido-dn-12-66-225:/home/liangrui# ll /data*/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir*/subdir*/blk_-922337203637104465*
> -rw-r--r-- 1 hdfs hdfs 193986560 Dec 1 2023 /data10/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652
> -rw-r--r-- 1 hdfs hdfs 1515527 Dec 1 2023 /data10/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652_1688858793.meta
> ========
> root@fs-hiido-dn-12-66-154:/data/logs/hadoop/hdfs# md5sum /data3/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652
> b661b6d711d753a82c3bf42bb2ceec51
> /data3/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652
> 10.12.66.225
> md5sum /data10/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652
> b661b6d711d753a82c3bf42bb2ceec51 /data10/hadoop/dfs/data/current/BP-1822992414-10.12.65.48-1660893388633/current/finalized/subdir21/subdir6/blk_-9223372036371044652 {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org