You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Raimon Bosch <ra...@gmail.com> on 2011/10/11 14:18:03 UTC
How to get number of live nodes in hadoop
Hi,
Following this instructions at
http://wiki.apache.org/hadoop/HowManyMapsAndReduces I've read that the best
amount of reducers for one process is 0.95 or 1.75 * (nodes *
mapred.tasktracker.tasks.maximum) so I would like to call
to conf.setNumReduceTasks(int num) according to how many nodes I have
working.
So how can I get the number of live nodes from my hadoop code?
Thanks in advance,
Raimon Bosch.
Re: How to get number of live nodes in hadoop
Posted by John Bond <jo...@gmail.com>.
for the nmap fans give the hadoop-jobtracker-info script from the
below link ago ;)
https://github.com/b4ldr/nse-scripts
disclaimer: this email is about shameless, self promotion
On 11 October 2011 19:52, patrick sang <si...@gmail.com> wrote:
> for cli fans,
>
> In DFS side - ./bin/hadoop dfsadmin -report
> In MR side - ./bin/hadoop job -list-active-trackers
>
> hth,
> P
>
> On Tue, Oct 11, 2011 at 5:27 AM, Uma Maheswara Rao G 72686 <
> maheswara@huawei.com> wrote:
>
>> Hello Raimon,
>>
>> In DFS to know the DN status you can use getDataNodeStats API from
>> Distributed fileSystem.
>>
>> In MR, to know the number of active trackers, you can use getClusterStatus
>> from jobclient. It will give other stats as well.
>>
>> Hope this will help.
>>
>> Regards,
>> Uma
>>
>>
>> ----- Original Message -----
>> From: Raimon Bosch <ra...@gmail.com>
>> Date: Tuesday, October 11, 2011 5:48 pm
>> Subject: How to get number of live nodes in hadoop
>> To: common-user@hadoop.apache.org
>>
>> > Hi,
>> >
>> > Following this instructions at
>> > http://wiki.apache.org/hadoop/HowManyMapsAndReduces I've read that
>> > the best
>> > amount of reducers for one process is 0.95 or 1.75 * (nodes *
>> > mapred.tasktracker.tasks.maximum) so I would like to call
>> > to conf.setNumReduceTasks(int num) according to how many nodes I have
>> > working.
>> >
>> > So how can I get the number of live nodes from my hadoop code?
>> >
>> > Thanks in advance,
>> > Raimon Bosch.
>> >
>>
>
Re: How to get number of live nodes in hadoop
Posted by patrick sang <si...@gmail.com>.
for cli fans,
In DFS side - ./bin/hadoop dfsadmin -report
In MR side - ./bin/hadoop job -list-active-trackers
hth,
P
On Tue, Oct 11, 2011 at 5:27 AM, Uma Maheswara Rao G 72686 <
maheswara@huawei.com> wrote:
> Hello Raimon,
>
> In DFS to know the DN status you can use getDataNodeStats API from
> Distributed fileSystem.
>
> In MR, to know the number of active trackers, you can use getClusterStatus
> from jobclient. It will give other stats as well.
>
> Hope this will help.
>
> Regards,
> Uma
>
>
> ----- Original Message -----
> From: Raimon Bosch <ra...@gmail.com>
> Date: Tuesday, October 11, 2011 5:48 pm
> Subject: How to get number of live nodes in hadoop
> To: common-user@hadoop.apache.org
>
> > Hi,
> >
> > Following this instructions at
> > http://wiki.apache.org/hadoop/HowManyMapsAndReduces I've read that
> > the best
> > amount of reducers for one process is 0.95 or 1.75 * (nodes *
> > mapred.tasktracker.tasks.maximum) so I would like to call
> > to conf.setNumReduceTasks(int num) according to how many nodes I have
> > working.
> >
> > So how can I get the number of live nodes from my hadoop code?
> >
> > Thanks in advance,
> > Raimon Bosch.
> >
>
Re: How to get number of live nodes in hadoop
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
Hello Raimon,
In DFS to know the DN status you can use getDataNodeStats API from Distributed fileSystem.
In MR, to know the number of active trackers, you can use getClusterStatus from jobclient. It will give other stats as well.
Hope this will help.
Regards,
Uma
----- Original Message -----
From: Raimon Bosch <ra...@gmail.com>
Date: Tuesday, October 11, 2011 5:48 pm
Subject: How to get number of live nodes in hadoop
To: common-user@hadoop.apache.org
> Hi,
>
> Following this instructions at
> http://wiki.apache.org/hadoop/HowManyMapsAndReduces I've read that
> the best
> amount of reducers for one process is 0.95 or 1.75 * (nodes *
> mapred.tasktracker.tasks.maximum) so I would like to call
> to conf.setNumReduceTasks(int num) according to how many nodes I have
> working.
>
> So how can I get the number of live nodes from my hadoop code?
>
> Thanks in advance,
> Raimon Bosch.
>