Who can tell the meaning from fsck

[jian yi <ey...@gmail.com>]

Hi All,


Run hadoop fsck / will give you summary of current HDFS status
including some useful information :

 Minimally replicated blocks:   51224 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         7 (0.013665469 %)
 Default replication factor:    3
 Average block replication:     3.0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          83
 Number of racks:               6

Who can tell me the meaning of items? Thanks.

Regards
Jian Yi

Re: Who can tell the meaning from fsck

[Suresh Srinivas <su...@yahoo-inc.com>]

On 2/24/10 12:25 AM, "jian yi" <ey...@gmail.com> wrote:

Hi All,


Run hadoop fsck / will give you summary of current HDFS status
including some useful information :

 Minimally replicated blocks:   51224 (100.0 %)
// Number of blocks with number of replicas >= minimum number of replicas configured in "dfs.namenode.replication.min"
 Over-replicated blocks:        0 (0.0 %)
// Number of blocks that have more replicas than the replication factor of the file
 Under-replicated blocks:       0 (0.0 %)
// Number of blocks that have less replicas than the replication factor of the file
 Mis-replicated blocks:         7 (0.013665469 %)
// Blocks with replicas that does not satisfy topology rules. For example instead of 2 replicas in 1 rack and 1 replica in a separate block, all replicas ended up on the same rack.
 Default replication factor:    3
// Default replication factor used by the system
 Average block replication:     3.0
// Average replicas per block
 Missing replicas:              0 (0.0 %)
// Blocks with all the replicas missing
 Number of data-nodes:          83
 Number of racks:               6


Who can tell me the meaning of items? Thanks.

Regards
Jian Yi

Re: Who can tell the meaning from fsck

[Ravi Phulari <rp...@yahoo-inc.com>]

Moving to Common-user@h.a.o
Answers inline .

On 2/24/10 12:25 AM, "jian yi" <ey...@gmail.com> wrote:

Hi All,


Run hadoop fsck / will give you summary of current HDFS status
including some useful information :

 Minimally replicated blocks:   51224 (100.0 %)
These are the minimally replicated block , you can set minimum block replication factor in hdfs-site.xml .  If there are any blocks which are not minimally replicated then NN will try to replicate them.

Over-replicated blocks:        0 (0.0 %)
These are the excessive replicated block. Over-replicated blocks are harmless and could have been created due to running Balancer.

 Under-replicated blocks:       0 (0.0 %)
Blocks which are under-replicated than min

 Mis-replicated blocks:         7 (0.013665469 %)
blocks that do not satisfy block placement policy

 Default replication factor:    3
Default block replication factor set using
<name>dfs.replication</name>
<value>3</value>

 Average block replication:     3.0
Average BR

 Missing replicas:              0 (0.0 %)
If there are any missing block replicas then that count is listed here.

Number of data-nodes:          83
Number of data nodes in your cluster

Number of racks:               6
Number of Racks in your cluster


You can configure your block replications by using following config settings in hdfs-site.xml
<property>
 <name>dfs.replication</name>
 <value>3</value>
 <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>

<property>
 <name>dfs.replication.max</name>
 <value>512</value>
 <description>Maximal block replication.
  </description>
</property>

<property>
 <name>dfs.replication.min</name>
 <value>1</value>
 <description>Minimal block replication.
  </description>
</property>

--
Ravi