HDFS inconsistency issue

[Songting Chen <ke...@yahoo.com>]

We are hitting a weird HDFS issue after a good number of Hadoop nodes 
simultaneously crashed.

The problem is that after all the down servers came back, the HDFS complained 
there was 1 missing block.
But the file that that block belongs to has already been deleted after the 
crash. So it's an orphan block.
Because the block doesn't belong to any file, there is no way to delete that 
block. FSCK also failed with a 

cause:java.io.IOException: Premature EOF exception.

A side effect now is that HDFS won't free up any space even after Trash bin was 
emptied. The space utilization 

just constantly went up.

Any suggestion on how to resolve this issue is highly appreciated!

Thanks,
-Songting