You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Marcus Herou <ma...@tailsweep.com> on 2009/04/07 09:45:20 UTC
Very assymetric data allocation
Hi.
We are running Hadoop 0.18.3 and noticed a strange issue when one of our
machines went out of disk yesterday.
If you can see the table below it would display that the server
"mapredcoord" is 66.91% allocated and the others are almost empty.
How can that be ?
Any information about this would be very helpful.
mapredcoord is as well our jobtracker.
//Marcus
Node Last Contact Admin State Size (GB) Used (%) Used (%) Remaining (GB) Blocks
mapredcoord<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service416.6966.91
90.9419806 mapreduce2<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service416.696.71
303.54456 mapreduce3<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service416.690.44
351.693975 mapreduce4<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service416.690.25
355.821549 mapreduce5<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service416.690.42
347.683995 mapreduce6<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service416.690.43
352.73982 mapreduce7<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service416.690.5
351.914079 mapreduce8<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service416.690.48
350.154169
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/
Re: Very assymetric data allocation
Posted by Marcus Herou <ma...@tailsweep.com>.
Great thanks for the info!
Right after I finished my last question I started to think about how Hadoop
measures data allocation. Are the figures presented actually the size of
HDFS on each machine or the amount of disk allocated and measured by issuing
something like "df".
The reason why I am asking is that df -h is quite close to the figures
presented in the GUI but it could be a coincidence.
//Marcus
On Tue, Apr 7, 2009 at 4:02 PM, Koji Noguchi <kn...@yahoo-inc.com> wrote:
> Marcus,
>
> One known issue in 0.18.3 is HADOOP-5465.
>
> Copy&Paste from
> https://issues.apache.org/jira/browse/HADOOP-4489?focusedCommentId=12693
> 956&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpa
> nel#action_12693956<https://issues.apache.org/jira/browse/HADOOP-4489?focusedCommentId=12693%0A956&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpa%0Anel#action_12693956>
>
> Hairong said:
> " This bug might be caused by HADOOP-5465. Once a datanode hits
> HADOOP-5465, NameNode sends an empty replication request to the data
> node on every reply to a heartbeat, thus not a single scheduled block
> deletion request can be sent to the data node."
>
> (Also, if you're always writing from one of the nodes, that node is more
> likely to get full.)
>
>
>
> Nigel, not sure if this is the issue, but it would be nice to have
> 0.18.4 out.
>
>
> Koji
>
>
>
> -----Original Message-----
> From: Marcus Herou [mailto:marcus.herou@tailsweep.com]
> Sent: Tuesday, April 07, 2009 12:45 AM
> To: hadoop-user@lucene.apache.org
> Subject: Very assymetric data allocation
>
> Hi.
>
> We are running Hadoop 0.18.3 and noticed a strange issue when one of our
> machines went out of disk yesterday.
> If you can see the table below it would display that the server
> "mapredcoord" is 66.91% allocated and the others are almost empty.
> How can that be ?
>
> Any information about this would be very helpful.
>
> mapredcoord is as well our jobtracker.
>
> //Marcus
>
> Node Last Contact Admin State Size (GB) Used (%) Used (%) Remaining (GB)
> Blocks
> mapredcoord<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPor
> t=50070&dir=%2F<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPor%0At=50070&dir=%2F>
> >2In
> Service416.6966.91
>
> 90.9419806
> mapreduce2<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >2In
> Service416.696.71
>
> 303.54456
> mapreduce3<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >2In
> Service416.690.44
> 351.693975
> mapreduce4<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >0In
> Service416.690.25
> 355.821549
> mapreduce5<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >2In
> Service416.690.42
> 347.683995
> mapreduce6<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >0In
> Service416.690.43
> 352.73982
> mapreduce7<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >0In
> Service416.690.5
> 351.914079
> mapreduce8<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort=
> 50070&dir=%2F<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort=%0A50070&dir=%2F>
> >1In
> Service416.690.48
> 350.154169
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
> http://blogg.tailsweep.com/
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/
RE: Very assymetric data allocation
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
Marcus,
One known issue in 0.18.3 is HADOOP-5465.
Copy&Paste from
https://issues.apache.org/jira/browse/HADOOP-4489?focusedCommentId=12693
956&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpa
nel#action_12693956
Hairong said:
" This bug might be caused by HADOOP-5465. Once a datanode hits
HADOOP-5465, NameNode sends an empty replication request to the data
node on every reply to a heartbeat, thus not a single scheduled block
deletion request can be sent to the data node."
(Also, if you're always writing from one of the nodes, that node is more
likely to get full.)
Nigel, not sure if this is the issue, but it would be nice to have
0.18.4 out.
Koji
-----Original Message-----
From: Marcus Herou [mailto:marcus.herou@tailsweep.com]
Sent: Tuesday, April 07, 2009 12:45 AM
To: hadoop-user@lucene.apache.org
Subject: Very assymetric data allocation
Hi.
We are running Hadoop 0.18.3 and noticed a strange issue when one of our
machines went out of disk yesterday.
If you can see the table below it would display that the server
"mapredcoord" is 66.91% allocated and the others are almost empty.
How can that be ?
Any information about this would be very helpful.
mapredcoord is as well our jobtracker.
//Marcus
Node Last Contact Admin State Size (GB) Used (%) Used (%) Remaining (GB)
Blocks
mapredcoord<http://mapredcoord:50076/browseDirectory.jsp?namenodeInfoPor
t=50070&dir=%2F>2In
Service416.6966.91
90.9419806
mapreduce2<http://mapreduce2:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>2In
Service416.696.71
303.54456
mapreduce3<http://mapreduce3:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>2In
Service416.690.44
351.693975
mapreduce4<http://mapreduce4:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>0In
Service416.690.25
355.821549
mapreduce5<http://mapreduce5:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>2In
Service416.690.42
347.683995
mapreduce6<http://mapreduce6:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>0In
Service416.690.43
352.73982
mapreduce7<http://mapreduce7:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>0In
Service416.690.5
351.914079
mapreduce8<http://mapreduce8:50076/browseDirectory.jsp?namenodeInfoPort=
50070&dir=%2F>1In
Service416.690.48
350.154169
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
http://blogg.tailsweep.com/