HDFS Datanode Blockreport includes invalid block.

[Drake민영근 <dr...@nexr.com>]

Greetings, All

Recently, I met the worst case of HDFS, the missing block. The
timeline(log) of namenode is below:

initial state: Datanode A(or B or C. not confirm yet) and 192.168.100.90
contains block_1.

(Datanode A, B, C crash w/ hardware fault)
07:33:03: ask replication block_1 from 192.168.100.90 to 192.168.100.210
07:34:14: updatedBlockmap addStoredBlock 192.168.100.210 with block_1
...
(Datanode A, B, C recover)
18:11:36: block_1 of 192.168.100.210 is invalidate
18:11:45: block_1 of 192.168.100.210 is deleted
...
19:43:03: updatedBlockmap addStoredBlock 192.168.100.210 with block_1
19:43:03: block_1 of 192.168.100.90 is invalidate
19:43:04: block_1 of 192.168.100.90 is deleted
...
(Datanode A, B, C crash w/ hardware fault AGAIN)
00:27:21: ask replication block_1 from 192.168.100.210 to 192.168.100.82
07:34:14: Error cause block_1 of 192.168.100.210 is invalid

At 19:43:03, Datanode 192.168.100.210 send the block report to namenode. I
guess 192.168.100.210's block report contains worng, in this case invalid,
block.

Anyone seen this problem ?

Sorry for log format. I cannot get the full logs.

Thanks.

Drake 민영근 Ph.D
kt NexR