You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Dhanasekaran Anbalagan <bu...@gmail.com> on 2013/02/27 11:17:18 UTC
How to get Under-replicated blocks information [Location]
Hi Guys,
I am running three machine cluster, with replication factor 2, I got
problem in replica i changed to 2.
after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
Total size: 105829415143 B
Total dirs: 9704
Total files: 2038873 (Files currently being written: 2)
Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open
file blocks (not validated): 2)
Minimally replicated blocks: 2038567 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 71 (0.0034828386 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9999995
Corrupt blocks: 0
Missing replicas: 71 (0.0017414198 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
Please guide what the block location of the 71. in HDFS file system.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
RE: How to get Under-replicated blocks information [Location]
Posted by Brahma Reddy Battula <br...@huawei.com>.
you get the block locations(hoping that you are asking about which node) by two ways..
i) ./hadoop fsck / -files -blocks -locations.(Commandline)
ii) NameNode UI..(GO to UI and click on browse filesystem and then click on files where you want to check..(Below you can see block locations..))
________________________________
From: Dhanasekaran Anbalagan [bugcy013@gmail.com]
Sent: Wednesday, February 27, 2013 6:17 PM
To: cdh-user; user
Subject: How to get Under-replicated blocks information [Location]
Hi Guys,
I am running three machine cluster, with replication factor 2, I got problem in replica i changed to 2.
after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
Total size: 105829415143 B
Total dirs: 9704
Total files: 2038873 (Files currently being written: 2)
Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 2038567 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 71 (0.0034828386 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9999995
Corrupt blocks: 0
Missing replicas: 71 (0.0017414198 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
Please guide what the block location of the 71. in HDFS file system.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
RE: How to get Under-replicated blocks information [Location]
Posted by Brahma Reddy Battula <br...@huawei.com>.
you get the block locations(hoping that you are asking about which node) by two ways..
i) ./hadoop fsck / -files -blocks -locations.(Commandline)
ii) NameNode UI..(GO to UI and click on browse filesystem and then click on files where you want to check..(Below you can see block locations..))
________________________________
From: Dhanasekaran Anbalagan [bugcy013@gmail.com]
Sent: Wednesday, February 27, 2013 6:17 PM
To: cdh-user; user
Subject: How to get Under-replicated blocks information [Location]
Hi Guys,
I am running three machine cluster, with replication factor 2, I got problem in replica i changed to 2.
after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
Total size: 105829415143 B
Total dirs: 9704
Total files: 2038873 (Files currently being written: 2)
Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 2038567 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 71 (0.0034828386 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9999995
Corrupt blocks: 0
Missing replicas: 71 (0.0017414198 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
Please guide what the block location of the 71. in HDFS file system.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
Re: How to get Under-replicated blocks information [Location]
Posted by Shawn Higgins <sc...@gmail.com>.
When you run FSCK, what options did you run it with? If you're using
'hadoop fsck /' then you will typically see a lot of dots being output and
amongst those dots you should see missing/corrupt or under-replicated
notices. Here's an excerpt containing under replicated blocks, for an
example:
....................................................................................................
.....................................................................................
/user/blank/.staging/job_201302211222_0011/libjars/common.jar: Under
replicated blk_-856998378388135111_3050830. Target Replicas is 10 but found
9 replica(s).
...............
..
/user/blank/.staging/job_201302211222_0733/job.jar: Under replicated
blk_4441280820634984431_3150889. Target Replicas is 10 but found 9
replica(s).
.......
/user/blank/.staging/job_201302211222_0741/job.jar: Under replicated
blk_6336347773734645693_3150957. Target Replicas is 10 but found 9
replica(s).
..........
/user/blank/.staging/job_201302211222_0755/job.jar: Under replicated
blk_-1209937263563068132_3154083. Target Replicas is 10 but found 9
replica(s).
.
/user/blank/.staging/job_201302211222_0756/job.jar: Under replicated
blk_-365467798112255961_3154084. Target Replicas is 10 but found 9
replica(s).
................................................................................
....................................................................................................
....................................................................................................
........................................[...]
The paths you see in the example are the HDFS locations within the cluster.
That should be the information you're looking for.
Does this help?
-Shawn
On Wednesday, February 27, 2013 4:17:18 AM UTC-6, Dhanasekaran Anbalagan
wrote:
>
> Hi Guys,
>
> I am running three machine cluster, with replication factor 2, I got
> problem in replica i changed to 2.
> after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
>
> Total size: 105829415143 B
> Total dirs: 9704
> Total files: 2038873 (Files currently being written: 2)
> Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open
> file blocks (not validated): 2)
> Minimally replicated blocks: 2038567 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 71 (0.0034828386 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9999995
> Corrupt blocks: 0
> Missing replicas: 71 (0.0017414198 %)
> Number of data-nodes: 2
> Number of racks: 1
> FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
>
>
> Please guide what the block location of the 71. in HDFS file system.
>
> -Dhanasekaran.
> Did I learn something today? If not, I wasted it.
>
RE: How to get Under-replicated blocks information [Location]
Posted by Brahma Reddy Battula <br...@huawei.com>.
you get the block locations(hoping that you are asking about which node) by two ways..
i) ./hadoop fsck / -files -blocks -locations.(Commandline)
ii) NameNode UI..(GO to UI and click on browse filesystem and then click on files where you want to check..(Below you can see block locations..))
________________________________
From: Dhanasekaran Anbalagan [bugcy013@gmail.com]
Sent: Wednesday, February 27, 2013 6:17 PM
To: cdh-user; user
Subject: How to get Under-replicated blocks information [Location]
Hi Guys,
I am running three machine cluster, with replication factor 2, I got problem in replica i changed to 2.
after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
Total size: 105829415143 B
Total dirs: 9704
Total files: 2038873 (Files currently being written: 2)
Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 2038567 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 71 (0.0034828386 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9999995
Corrupt blocks: 0
Missing replicas: 71 (0.0017414198 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
Please guide what the block location of the 71. in HDFS file system.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
RE: How to get Under-replicated blocks information [Location]
Posted by Brahma Reddy Battula <br...@huawei.com>.
you get the block locations(hoping that you are asking about which node) by two ways..
i) ./hadoop fsck / -files -blocks -locations.(Commandline)
ii) NameNode UI..(GO to UI and click on browse filesystem and then click on files where you want to check..(Below you can see block locations..))
________________________________
From: Dhanasekaran Anbalagan [bugcy013@gmail.com]
Sent: Wednesday, February 27, 2013 6:17 PM
To: cdh-user; user
Subject: How to get Under-replicated blocks information [Location]
Hi Guys,
I am running three machine cluster, with replication factor 2, I got problem in replica i changed to 2.
after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
Total size: 105829415143 B
Total dirs: 9704
Total files: 2038873 (Files currently being written: 2)
Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open file blocks (not validated): 2)
Minimally replicated blocks: 2038567 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 71 (0.0034828386 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.9999995
Corrupt blocks: 0
Missing replicas: 71 (0.0017414198 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
Please guide what the block location of the 71. in HDFS file system.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
Re: How to get Under-replicated blocks information [Location]
Posted by Shawn Higgins <sc...@gmail.com>.
When you run FSCK, what options did you run it with? If you're using
'hadoop fsck /' then you will typically see a lot of dots being output and
amongst those dots you should see missing/corrupt or under-replicated
notices. Here's an excerpt containing under replicated blocks, for an
example:
....................................................................................................
.....................................................................................
/user/blank/.staging/job_201302211222_0011/libjars/common.jar: Under
replicated blk_-856998378388135111_3050830. Target Replicas is 10 but found
9 replica(s).
...............
..
/user/blank/.staging/job_201302211222_0733/job.jar: Under replicated
blk_4441280820634984431_3150889. Target Replicas is 10 but found 9
replica(s).
.......
/user/blank/.staging/job_201302211222_0741/job.jar: Under replicated
blk_6336347773734645693_3150957. Target Replicas is 10 but found 9
replica(s).
..........
/user/blank/.staging/job_201302211222_0755/job.jar: Under replicated
blk_-1209937263563068132_3154083. Target Replicas is 10 but found 9
replica(s).
.
/user/blank/.staging/job_201302211222_0756/job.jar: Under replicated
blk_-365467798112255961_3154084. Target Replicas is 10 but found 9
replica(s).
................................................................................
....................................................................................................
....................................................................................................
........................................[...]
The paths you see in the example are the HDFS locations within the cluster.
That should be the information you're looking for.
Does this help?
-Shawn
On Wednesday, February 27, 2013 4:17:18 AM UTC-6, Dhanasekaran Anbalagan
wrote:
>
> Hi Guys,
>
> I am running three machine cluster, with replication factor 2, I got
> problem in replica i changed to 2.
> after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
>
> Total size: 105829415143 B
> Total dirs: 9704
> Total files: 2038873 (Files currently being written: 2)
> Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open
> file blocks (not validated): 2)
> Minimally replicated blocks: 2038567 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 71 (0.0034828386 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9999995
> Corrupt blocks: 0
> Missing replicas: 71 (0.0017414198 %)
> Number of data-nodes: 2
> Number of racks: 1
> FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
>
>
> Please guide what the block location of the 71. in HDFS file system.
>
> -Dhanasekaran.
> Did I learn something today? If not, I wasted it.
>
Re: How to get Under-replicated blocks information [Location]
Posted by Shawn Higgins <sc...@gmail.com>.
When you run FSCK, what options did you run it with? If you're using
'hadoop fsck /' then you will typically see a lot of dots being output and
amongst those dots you should see missing/corrupt or under-replicated
notices. Here's an excerpt containing under replicated blocks, for an
example:
....................................................................................................
.....................................................................................
/user/blank/.staging/job_201302211222_0011/libjars/common.jar: Under
replicated blk_-856998378388135111_3050830. Target Replicas is 10 but found
9 replica(s).
...............
..
/user/blank/.staging/job_201302211222_0733/job.jar: Under replicated
blk_4441280820634984431_3150889. Target Replicas is 10 but found 9
replica(s).
.......
/user/blank/.staging/job_201302211222_0741/job.jar: Under replicated
blk_6336347773734645693_3150957. Target Replicas is 10 but found 9
replica(s).
..........
/user/blank/.staging/job_201302211222_0755/job.jar: Under replicated
blk_-1209937263563068132_3154083. Target Replicas is 10 but found 9
replica(s).
.
/user/blank/.staging/job_201302211222_0756/job.jar: Under replicated
blk_-365467798112255961_3154084. Target Replicas is 10 but found 9
replica(s).
................................................................................
....................................................................................................
....................................................................................................
........................................[...]
The paths you see in the example are the HDFS locations within the cluster.
That should be the information you're looking for.
Does this help?
-Shawn
On Wednesday, February 27, 2013 4:17:18 AM UTC-6, Dhanasekaran Anbalagan
wrote:
>
> Hi Guys,
>
> I am running three machine cluster, with replication factor 2, I got
> problem in replica i changed to 2.
> after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
>
> Total size: 105829415143 B
> Total dirs: 9704
> Total files: 2038873 (Files currently being written: 2)
> Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open
> file blocks (not validated): 2)
> Minimally replicated blocks: 2038567 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 71 (0.0034828386 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9999995
> Corrupt blocks: 0
> Missing replicas: 71 (0.0017414198 %)
> Number of data-nodes: 2
> Number of racks: 1
> FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
>
>
> Please guide what the block location of the 71. in HDFS file system.
>
> -Dhanasekaran.
> Did I learn something today? If not, I wasted it.
>
Re: How to get Under-replicated blocks information [Location]
Posted by Shawn Higgins <sc...@gmail.com>.
When you run FSCK, what options did you run it with? If you're using
'hadoop fsck /' then you will typically see a lot of dots being output and
amongst those dots you should see missing/corrupt or under-replicated
notices. Here's an excerpt containing under replicated blocks, for an
example:
....................................................................................................
.....................................................................................
/user/blank/.staging/job_201302211222_0011/libjars/common.jar: Under
replicated blk_-856998378388135111_3050830. Target Replicas is 10 but found
9 replica(s).
...............
..
/user/blank/.staging/job_201302211222_0733/job.jar: Under replicated
blk_4441280820634984431_3150889. Target Replicas is 10 but found 9
replica(s).
.......
/user/blank/.staging/job_201302211222_0741/job.jar: Under replicated
blk_6336347773734645693_3150957. Target Replicas is 10 but found 9
replica(s).
..........
/user/blank/.staging/job_201302211222_0755/job.jar: Under replicated
blk_-1209937263563068132_3154083. Target Replicas is 10 but found 9
replica(s).
.
/user/blank/.staging/job_201302211222_0756/job.jar: Under replicated
blk_-365467798112255961_3154084. Target Replicas is 10 but found 9
replica(s).
................................................................................
....................................................................................................
....................................................................................................
........................................[...]
The paths you see in the example are the HDFS locations within the cluster.
That should be the information you're looking for.
Does this help?
-Shawn
On Wednesday, February 27, 2013 4:17:18 AM UTC-6, Dhanasekaran Anbalagan
wrote:
>
> Hi Guys,
>
> I am running three machine cluster, with replication factor 2, I got
> problem in replica i changed to 2.
> after I ran fsck i got Under-replicated blocks: 71 (0.0034828386 %)
>
> Total size: 105829415143 B
> Total dirs: 9704
> Total files: 2038873 (Files currently being written: 2)
> Total blocks (validated): 2038567 (avg. block size 51913 B) (Total open
> file blocks (not validated): 2)
> Minimally replicated blocks: 2038567 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 71 (0.0034828386 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 2
> Average block replication: 1.9999995
> Corrupt blocks: 0
> Missing replicas: 71 (0.0017414198 %)
> Number of data-nodes: 2
> Number of racks: 1
> FSCK ended at Wed Feb 27 05:12:31 EST 2013 in 32647 milliseconds
>
>
> Please guide what the block location of the 71. in HDFS file system.
>
> -Dhanasekaran.
> Did I learn something today? If not, I wasted it.
>