Hadoop All data corrupted

["pranav.puri" <pr...@orkash.com.INVALID>]

Hi

I have a accumulo cluster setup over a hadoop cluster(ver.- 2.9.0). 
Since I had to make changes into the accumulo config files, I had to 
shutdown hadoop multiple times.

After some changes the hadoop started showing this when FSCK command is run:

Total size:    986079103985 B (Total open files size: 372 B)
  Total dirs:    2011
  Total files:    15530
  Total symlinks:        0 (Files currently being written: 4)
  Total blocks (validated):    19663 (avg. block size 50148965 B) (Total 
open file blocks (not validated): 4)
   ********************************
   UNDER MIN REPL'D BLOCKS:    19663 (100.0 %)
   dfs.namenode.replication.min:    1
   CORRUPT FILES:    15310
   MISSING BLOCKS:    19663
   MISSING SIZE:        986079103985 B
   CORRUPT BLOCKS:     19663

All the changes were done in the accumulo config files, while no hadoop 
config files were changed. What are the troubleshooting steps.

Regards
Pranav

Re: Hadoop All data corrupted

[Karthik P <ka...@gmail.com>]

Pranav,

Can you do the following checklist before confirming the missing/corrupt
blocks?


Missing Block: Mark missing if all of the block replicas of that file is
not reported to Namenode.

Corrupt Block: Mark corrupt if all of the block replicas of that file is
corrupted (Or) none of them are reported to Namenode.


1. Check if all datanodes are running in the cluster

2. Check if you see dead datanodes

3. Check if disk failure from multiple datanode

4. Check if disk out of space from multiple datanode

5. Check if block report is rejected by namenode (It can be seen from
namenode log as a warning/error)

6. Check if you changed any config groups

7. Check if block physically exists in local filesystem or removed by users
unknowingly. Ex: "find <dfs.datanode.data.dir> -type f -iname <blkid>*".
Repeat the same step in all datanodes

8. Check if too many blocks hosted in a single datanode

9. Check if block report fails with "exceeding max RPC size", default 64
MB. You can see this warning from namenode log "Protocol message was too
large. May be malicious"

10. Check if mount point is unmounted because of filesystem failure

11. Check if block is written into root volume because of disk auto
unmount. Data might be hidden if you remount the filesystem on top of
existing datanode dir.


Thanks,

Karthik

On Mon, Mar 4, 2019 at 3:27 AM pranav.puri <pr...@orkash.com.invalid>
wrote:

> Hi
>
> I have a accumulo cluster setup over a hadoop cluster(ver.- 2.9.0). Since
> I had to make changes into the accumulo config files, I had to shutdown
> hadoop multiple times.
>
> After some changes the hadoop started showing this when FSCK command is
> run:
>
> Total size:    986079103985 B (Total open files size: 372 B)
>  Total dirs:    2011
>  Total files:    15530
>  Total symlinks:        0 (Files currently being written: 4)
>  Total blocks (validated):    19663 (avg. block size 50148965 B) (Total
> open file blocks (not validated): 4)
>   ********************************
>   UNDER MIN REPL'D BLOCKS:    19663 (100.0 %)
>   dfs.namenode.replication.min:    1
>   CORRUPT FILES:    15310
>   MISSING BLOCKS:    19663
>   MISSING SIZE:        986079103985 B
>   CORRUPT BLOCKS:     19663
>
> All the changes were done in the accumulo config files, while no hadoop
> config files were changed. What are the troubleshooting steps.
>
> Regards
> Pranav
>
>
>
>

-- 
Thank you,
*Karthik Palanisamy*
Bangalore, *India*
Mobile : +91 9940089181
Skype : karthik.p01