You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Usman Waheed <us...@opera.com> on 2009/04/26 17:54:08 UTC
One of my data nodes is under used: Datanode 4
Hi,
One of my data nodes is practically under utilized in the cluster of 4
datanodes and one namenode.
I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
files in the HDFS.
All the nodes are balancer except one. Any clues? Shall i run hadoop
balancer after the rep factor is reset.
Thanks,
Usman
-------------------------------------------------
Datanodes available: 4
Name: 1
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 217963111201(202.99 GB)
Used raw bytes: 52500185088 (48.89 GB)
% used: 17.87%
Last contact: Sun Apr 26 15:49:17 UTC 2009
Name: 2
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 227810456463(212.17 GB)
Used raw bytes: 44199026688 (41.16 GB)
% used: 15.04%
Last contact: Sun Apr 26 15:49:19 UTC 2009
Name: 3
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 207735568331(193.47 GB)
Used raw bytes: 63878131712 (59.49 GB)
% used: 21.74%
Last contact: Sun Apr 26 15:49:19 UTC 2009
Name: 4
State : In Service
Total raw bytes: 293778976768 (273.6 GB)
Remaining raw bytes: 270775018072(252.18 GB)
Used raw bytes: 1916928 (1.83 MB)
% used: 0%
Last contact: Sun Apr 26 15:49:17 UTC 2009
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: One of my data nodes is under used: Datanode 4
Posted by Usman Waheed <us...@opera.com>.
Hi,
I did run the -setrep -w 2 -R /mydir and it basically emptied out one of
my nodes.
I just ran the hadoop balancer and now the datanode that was empty is
picking up blocks.
So lets see what happens once it is done and i will write back here.
Thanks,
Usman
> you need to add the -R parameter to setrep to force recursively setting
> the
> replication, other wise it just affects the single file that is the last
> argument.
>
> Is there some issue with the one datanode, where it is unable to accept
> or
> store blocks?
>
> On Sun, Apr 26, 2009 at 9:37 AM, jason hadoop
> <ja...@gmail.com>wrote:
>
>> hadoop dfsadmin -report
>>
>>
>> On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra
>> <mn...@asu.edu>wrote:
>>
>>> How do I list all the datanodes like you ve shown in your example?
>>> Thanks
>>> Mithila
>>>
>>> On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed <us...@opera.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > One of my data nodes is practically under utilized in the cluster of
>>> 4
>>> > datanodes and one namenode.
>>> > I executed the hadoop -setrep -w 2 changing it from rep factor 3 for
>>> all
>>> > files in the HDFS.
>>> >
>>> > All the nodes are balancer except one. Any clues? Shall i run hadoop
>>> > balancer after the rep factor is reset.
>>> >
>>> > Thanks,
>>> > Usman
>>> >
>>> >
>>> > -------------------------------------------------
>>> > Datanodes available: 4
>>> >
>>> > Name: 1
>>> > State : In Service
>>> > Total raw bytes: 293778976768 (273.6 GB)
>>> > Remaining raw bytes: 217963111201(202.99 GB)
>>> > Used raw bytes: 52500185088 (48.89 GB)
>>> > % used: 17.87%
>>> > Last contact: Sun Apr 26 15:49:17 UTC 2009
>>> >
>>> >
>>> > Name: 2
>>> > State : In Service
>>> > Total raw bytes: 293778976768 (273.6 GB)
>>> > Remaining raw bytes: 227810456463(212.17 GB)
>>> > Used raw bytes: 44199026688 (41.16 GB)
>>> > % used: 15.04%
>>> > Last contact: Sun Apr 26 15:49:19 UTC 2009
>>> >
>>> >
>>> > Name: 3
>>> > State : In Service
>>> > Total raw bytes: 293778976768 (273.6 GB)
>>> > Remaining raw bytes: 207735568331(193.47 GB)
>>> > Used raw bytes: 63878131712 (59.49 GB)
>>> > % used: 21.74%
>>> > Last contact: Sun Apr 26 15:49:19 UTC 2009
>>> >
>>> >
>>> > Name: 4
>>> > State : In Service
>>> > Total raw bytes: 293778976768 (273.6 GB)
>>> > Remaining raw bytes: 270775018072(252.18 GB)
>>> > Used raw bytes: 1916928 (1.83 MB)
>>> > % used: 0%
>>> > Last contact: Sun Apr 26 15:49:17 UTC 2009
>>> >
>>> >
>>> > --
>>> > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>> >
>>>
>>
>>
>>
>> --
>> Alpha Chapters of my book on Hadoop are available
>> http://www.apress.com/book/view/9781430219422
>>
>
>
>
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Re: One of my data nodes is under used: Datanode 4
Posted by jason hadoop <ja...@gmail.com>.
you need to add the -R parameter to setrep to force recursively setting the
replication, other wise it just affects the single file that is the last
argument.
Is there some issue with the one datanode, where it is unable to accept or
store blocks?
On Sun, Apr 26, 2009 at 9:37 AM, jason hadoop <ja...@gmail.com>wrote:
> hadoop dfsadmin -report
>
>
> On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra <mn...@asu.edu>wrote:
>
>> How do I list all the datanodes like you ve shown in your example?
>> Thanks
>> Mithila
>>
>> On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed <us...@opera.com> wrote:
>>
>> > Hi,
>> >
>> > One of my data nodes is practically under utilized in the cluster of 4
>> > datanodes and one namenode.
>> > I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
>> > files in the HDFS.
>> >
>> > All the nodes are balancer except one. Any clues? Shall i run hadoop
>> > balancer after the rep factor is reset.
>> >
>> > Thanks,
>> > Usman
>> >
>> >
>> > -------------------------------------------------
>> > Datanodes available: 4
>> >
>> > Name: 1
>> > State : In Service
>> > Total raw bytes: 293778976768 (273.6 GB)
>> > Remaining raw bytes: 217963111201(202.99 GB)
>> > Used raw bytes: 52500185088 (48.89 GB)
>> > % used: 17.87%
>> > Last contact: Sun Apr 26 15:49:17 UTC 2009
>> >
>> >
>> > Name: 2
>> > State : In Service
>> > Total raw bytes: 293778976768 (273.6 GB)
>> > Remaining raw bytes: 227810456463(212.17 GB)
>> > Used raw bytes: 44199026688 (41.16 GB)
>> > % used: 15.04%
>> > Last contact: Sun Apr 26 15:49:19 UTC 2009
>> >
>> >
>> > Name: 3
>> > State : In Service
>> > Total raw bytes: 293778976768 (273.6 GB)
>> > Remaining raw bytes: 207735568331(193.47 GB)
>> > Used raw bytes: 63878131712 (59.49 GB)
>> > % used: 21.74%
>> > Last contact: Sun Apr 26 15:49:19 UTC 2009
>> >
>> >
>> > Name: 4
>> > State : In Service
>> > Total raw bytes: 293778976768 (273.6 GB)
>> > Remaining raw bytes: 270775018072(252.18 GB)
>> > Used raw bytes: 1916928 (1.83 MB)
>> > % used: 0%
>> > Last contact: Sun Apr 26 15:49:17 UTC 2009
>> >
>> >
>> > --
>> > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>> >
>>
>
>
>
> --
> Alpha Chapters of my book on Hadoop are available
> http://www.apress.com/book/view/9781430219422
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
Re: One of my data nodes is under used: Datanode 4
Posted by jason hadoop <ja...@gmail.com>.
hadoop dfsadmin -report
On Sun, Apr 26, 2009 at 9:18 AM, Mithila Nagendra <mn...@asu.edu> wrote:
> How do I list all the datanodes like you ve shown in your example?
> Thanks
> Mithila
>
> On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed <us...@opera.com> wrote:
>
> > Hi,
> >
> > One of my data nodes is practically under utilized in the cluster of 4
> > datanodes and one namenode.
> > I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
> > files in the HDFS.
> >
> > All the nodes are balancer except one. Any clues? Shall i run hadoop
> > balancer after the rep factor is reset.
> >
> > Thanks,
> > Usman
> >
> >
> > -------------------------------------------------
> > Datanodes available: 4
> >
> > Name: 1
> > State : In Service
> > Total raw bytes: 293778976768 (273.6 GB)
> > Remaining raw bytes: 217963111201(202.99 GB)
> > Used raw bytes: 52500185088 (48.89 GB)
> > % used: 17.87%
> > Last contact: Sun Apr 26 15:49:17 UTC 2009
> >
> >
> > Name: 2
> > State : In Service
> > Total raw bytes: 293778976768 (273.6 GB)
> > Remaining raw bytes: 227810456463(212.17 GB)
> > Used raw bytes: 44199026688 (41.16 GB)
> > % used: 15.04%
> > Last contact: Sun Apr 26 15:49:19 UTC 2009
> >
> >
> > Name: 3
> > State : In Service
> > Total raw bytes: 293778976768 (273.6 GB)
> > Remaining raw bytes: 207735568331(193.47 GB)
> > Used raw bytes: 63878131712 (59.49 GB)
> > % used: 21.74%
> > Last contact: Sun Apr 26 15:49:19 UTC 2009
> >
> >
> > Name: 4
> > State : In Service
> > Total raw bytes: 293778976768 (273.6 GB)
> > Remaining raw bytes: 270775018072(252.18 GB)
> > Used raw bytes: 1916928 (1.83 MB)
> > % used: 0%
> > Last contact: Sun Apr 26 15:49:17 UTC 2009
> >
> >
> > --
> > Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
> >
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
Re: One of my data nodes is under used: Datanode 4
Posted by Mithila Nagendra <mn...@asu.edu>.
How do I list all the datanodes like you ve shown in your example?
Thanks
Mithila
On Sun, Apr 26, 2009 at 6:54 PM, Usman Waheed <us...@opera.com> wrote:
> Hi,
>
> One of my data nodes is practically under utilized in the cluster of 4
> datanodes and one namenode.
> I executed the hadoop -setrep -w 2 changing it from rep factor 3 for all
> files in the HDFS.
>
> All the nodes are balancer except one. Any clues? Shall i run hadoop
> balancer after the rep factor is reset.
>
> Thanks,
> Usman
>
>
> -------------------------------------------------
> Datanodes available: 4
>
> Name: 1
> State : In Service
> Total raw bytes: 293778976768 (273.6 GB)
> Remaining raw bytes: 217963111201(202.99 GB)
> Used raw bytes: 52500185088 (48.89 GB)
> % used: 17.87%
> Last contact: Sun Apr 26 15:49:17 UTC 2009
>
>
> Name: 2
> State : In Service
> Total raw bytes: 293778976768 (273.6 GB)
> Remaining raw bytes: 227810456463(212.17 GB)
> Used raw bytes: 44199026688 (41.16 GB)
> % used: 15.04%
> Last contact: Sun Apr 26 15:49:19 UTC 2009
>
>
> Name: 3
> State : In Service
> Total raw bytes: 293778976768 (273.6 GB)
> Remaining raw bytes: 207735568331(193.47 GB)
> Used raw bytes: 63878131712 (59.49 GB)
> % used: 21.74%
> Last contact: Sun Apr 26 15:49:19 UTC 2009
>
>
> Name: 4
> State : In Service
> Total raw bytes: 293778976768 (273.6 GB)
> Remaining raw bytes: 270775018072(252.18 GB)
> Used raw bytes: 1916928 (1.83 MB)
> % used: 0%
> Last contact: Sun Apr 26 15:49:17 UTC 2009
>
>
> --
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>