Resetting dead datanodes list

[Ashwin Shankar <as...@gmail.com>]

Hi,
Hadoop-1's name node UI displays dead datanodes
even if those instances are terminated and are not part of the cluster
anymore.
Is there a way to reset the dead datenode list without bouncing namenode ?

This would help me in my script(which would run nightly) which parses the
html page,terminates
dead datanodes and resize the cluster.
-- 
Thanks,
Ashwin

Re: Resetting dead datanodes list

[divye sheth <di...@gmail.com>]

If by resetting the list of dead datanodes you mean the web-console or the
report command not showing the datanode that you removed, in this case you
will have to do the following:

1. Remove the entry from the slaves file corresponding to the dead datanode.
2. Remove entry from exclude file
3. Run hadoop dfsadmin -refreshNodes

Performing these steps would no longer show the dead datanode in either the
UI or report command. But if you only add that host to your exclude file (I
am assuming you already have configured dfs.hosts.exclude property in NN),
the namenode will still report this server as dead though no corresponding
value i.e. DFS used, Non-DFS Used etc. would be shown.

Thanks
Divye Sheth


On Mon, Apr 14, 2014 at 12:39 PM, Sandeep Nemuri <nh...@gmail.com>wrote:

> You can add the hostname/IP in exclude file and run this command
>
> hadoop dfsadmin -refreshNodes.
>
>
> On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:
>
>> I believe there's some command to show list of datanodes from CLI, using
>> parsing HTML is not a good idea. HTML page is intended to be read by human.
>> I also don't know how to refresh node list;
>>
>>
>>
>> Regards,
>> *Stanley Shi,*
>>
>>
>>
>> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi,
>>> Hadoop-1's name node UI displays dead datanodes
>>> even if those instances are terminated and are not part of the cluster
>>> anymore.
>>> Is there a way to reset the dead datenode list without bouncing namenode
>>> ?
>>>
>>> This would help me in my script(which would run nightly) which parses
>>> the html page,terminates
>>> dead datanodes and resize the cluster.
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>
>
> --
> --Regards
>   Sandeep Nemuri
>

Re: Resetting dead datanodes list

[divye sheth <di...@gmail.com>]

If by resetting the list of dead datanodes you mean the web-console or the
report command not showing the datanode that you removed, in this case you
will have to do the following:

1. Remove the entry from the slaves file corresponding to the dead datanode.
2. Remove entry from exclude file
3. Run hadoop dfsadmin -refreshNodes

Performing these steps would no longer show the dead datanode in either the
UI or report command. But if you only add that host to your exclude file (I
am assuming you already have configured dfs.hosts.exclude property in NN),
the namenode will still report this server as dead though no corresponding
value i.e. DFS used, Non-DFS Used etc. would be shown.

Thanks
Divye Sheth


On Mon, Apr 14, 2014 at 12:39 PM, Sandeep Nemuri <nh...@gmail.com>wrote:

> You can add the hostname/IP in exclude file and run this command
>
> hadoop dfsadmin -refreshNodes.
>
>
> On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:
>
>> I believe there's some command to show list of datanodes from CLI, using
>> parsing HTML is not a good idea. HTML page is intended to be read by human.
>> I also don't know how to refresh node list;
>>
>>
>>
>> Regards,
>> *Stanley Shi,*
>>
>>
>>
>> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi,
>>> Hadoop-1's name node UI displays dead datanodes
>>> even if those instances are terminated and are not part of the cluster
>>> anymore.
>>> Is there a way to reset the dead datenode list without bouncing namenode
>>> ?
>>>
>>> This would help me in my script(which would run nightly) which parses
>>> the html page,terminates
>>> dead datanodes and resize the cluster.
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>
>
> --
> --Regards
>   Sandeep Nemuri
>

Re: Resetting dead datanodes list

[divye sheth <di...@gmail.com>]

If by resetting the list of dead datanodes you mean the web-console or the
report command not showing the datanode that you removed, in this case you
will have to do the following:

1. Remove the entry from the slaves file corresponding to the dead datanode.
2. Remove entry from exclude file
3. Run hadoop dfsadmin -refreshNodes

Performing these steps would no longer show the dead datanode in either the
UI or report command. But if you only add that host to your exclude file (I
am assuming you already have configured dfs.hosts.exclude property in NN),
the namenode will still report this server as dead though no corresponding
value i.e. DFS used, Non-DFS Used etc. would be shown.

Thanks
Divye Sheth


On Mon, Apr 14, 2014 at 12:39 PM, Sandeep Nemuri <nh...@gmail.com>wrote:

> You can add the hostname/IP in exclude file and run this command
>
> hadoop dfsadmin -refreshNodes.
>
>
> On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:
>
>> I believe there's some command to show list of datanodes from CLI, using
>> parsing HTML is not a good idea. HTML page is intended to be read by human.
>> I also don't know how to refresh node list;
>>
>>
>>
>> Regards,
>> *Stanley Shi,*
>>
>>
>>
>> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi,
>>> Hadoop-1's name node UI displays dead datanodes
>>> even if those instances are terminated and are not part of the cluster
>>> anymore.
>>> Is there a way to reset the dead datenode list without bouncing namenode
>>> ?
>>>
>>> This would help me in my script(which would run nightly) which parses
>>> the html page,terminates
>>> dead datanodes and resize the cluster.
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>
>
> --
> --Regards
>   Sandeep Nemuri
>

Re: Resetting dead datanodes list

[divye sheth <di...@gmail.com>]

If by resetting the list of dead datanodes you mean the web-console or the
report command not showing the datanode that you removed, in this case you
will have to do the following:

1. Remove the entry from the slaves file corresponding to the dead datanode.
2. Remove entry from exclude file
3. Run hadoop dfsadmin -refreshNodes

Performing these steps would no longer show the dead datanode in either the
UI or report command. But if you only add that host to your exclude file (I
am assuming you already have configured dfs.hosts.exclude property in NN),
the namenode will still report this server as dead though no corresponding
value i.e. DFS used, Non-DFS Used etc. would be shown.

Thanks
Divye Sheth


On Mon, Apr 14, 2014 at 12:39 PM, Sandeep Nemuri <nh...@gmail.com>wrote:

> You can add the hostname/IP in exclude file and run this command
>
> hadoop dfsadmin -refreshNodes.
>
>
> On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:
>
>> I believe there's some command to show list of datanodes from CLI, using
>> parsing HTML is not a good idea. HTML page is intended to be read by human.
>> I also don't know how to refresh node list;
>>
>>
>>
>> Regards,
>> *Stanley Shi,*
>>
>>
>>
>> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
>> ashwinshankar77@gmail.com> wrote:
>>
>>> Hi,
>>> Hadoop-1's name node UI displays dead datanodes
>>> even if those instances are terminated and are not part of the cluster
>>> anymore.
>>> Is there a way to reset the dead datenode list without bouncing namenode
>>> ?
>>>
>>> This would help me in my script(which would run nightly) which parses
>>> the html page,terminates
>>> dead datanodes and resize the cluster.
>>> --
>>> Thanks,
>>> Ashwin
>>>
>>>
>>>
>>
>
>
> --
> --Regards
>   Sandeep Nemuri
>

Re: Resetting dead datanodes list

[Sandeep Nemuri <nh...@gmail.com>]

You can add the hostname/IP in exclude file and run this command

hadoop dfsadmin -refreshNodes.


On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:

> I believe there's some command to show list of datanodes from CLI, using
> parsing HTML is not a good idea. HTML page is intended to be read by human.
> I also don't know how to refresh node list;
>
>
>
> Regards,
> *Stanley Shi,*
>
>
>
> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi,
>> Hadoop-1's name node UI displays dead datanodes
>> even if those instances are terminated and are not part of the cluster
>> anymore.
>> Is there a way to reset the dead datenode list without bouncing namenode ?
>>
>> This would help me in my script(which would run nightly) which parses the
>> html page,terminates
>> dead datanodes and resize the cluster.
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>


-- 
--Regards
  Sandeep Nemuri

Re: Resetting dead datanodes list

[Sandeep Nemuri <nh...@gmail.com>]

You can add the hostname/IP in exclude file and run this command

hadoop dfsadmin -refreshNodes.


On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:

> I believe there's some command to show list of datanodes from CLI, using
> parsing HTML is not a good idea. HTML page is intended to be read by human.
> I also don't know how to refresh node list;
>
>
>
> Regards,
> *Stanley Shi,*
>
>
>
> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi,
>> Hadoop-1's name node UI displays dead datanodes
>> even if those instances are terminated and are not part of the cluster
>> anymore.
>> Is there a way to reset the dead datenode list without bouncing namenode ?
>>
>> This would help me in my script(which would run nightly) which parses the
>> html page,terminates
>> dead datanodes and resize the cluster.
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>


-- 
--Regards
  Sandeep Nemuri

Re: Resetting dead datanodes list

[Sandeep Nemuri <nh...@gmail.com>]

You can add the hostname/IP in exclude file and run this command

hadoop dfsadmin -refreshNodes.


On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:

> I believe there's some command to show list of datanodes from CLI, using
> parsing HTML is not a good idea. HTML page is intended to be read by human.
> I also don't know how to refresh node list;
>
>
>
> Regards,
> *Stanley Shi,*
>
>
>
> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi,
>> Hadoop-1's name node UI displays dead datanodes
>> even if those instances are terminated and are not part of the cluster
>> anymore.
>> Is there a way to reset the dead datenode list without bouncing namenode ?
>>
>> This would help me in my script(which would run nightly) which parses the
>> html page,terminates
>> dead datanodes and resize the cluster.
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>


-- 
--Regards
  Sandeep Nemuri

Re: Resetting dead datanodes list

[Sandeep Nemuri <nh...@gmail.com>]

You can add the hostname/IP in exclude file and run this command

hadoop dfsadmin -refreshNodes.


On Mon, Apr 14, 2014 at 11:34 AM, Stanley Shi <ss...@gopivotal.com> wrote:

> I believe there's some command to show list of datanodes from CLI, using
> parsing HTML is not a good idea. HTML page is intended to be read by human.
> I also don't know how to refresh node list;
>
>
>
> Regards,
> *Stanley Shi,*
>
>
>
> On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar <
> ashwinshankar77@gmail.com> wrote:
>
>> Hi,
>> Hadoop-1's name node UI displays dead datanodes
>> even if those instances are terminated and are not part of the cluster
>> anymore.
>> Is there a way to reset the dead datenode list without bouncing namenode ?
>>
>> This would help me in my script(which would run nightly) which parses the
>> html page,terminates
>> dead datanodes and resize the cluster.
>> --
>> Thanks,
>> Ashwin
>>
>>
>>
>


-- 
--Regards
  Sandeep Nemuri

Re: Resetting dead datanodes list

[Stanley Shi <ss...@gopivotal.com>]

I believe there's some command to show list of datanodes from CLI, using
parsing HTML is not a good idea. HTML page is intended to be read by human.
I also don't know how to refresh node list;



Regards,
*Stanley Shi,*



On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar
<as...@gmail.com>wrote:

> Hi,
> Hadoop-1's name node UI displays dead datanodes
> even if those instances are terminated and are not part of the cluster
> anymore.
> Is there a way to reset the dead datenode list without bouncing namenode ?
>
> This would help me in my script(which would run nightly) which parses the
> html page,terminates
> dead datanodes and resize the cluster.
> --
> Thanks,
> Ashwin
>
>
>

Re: Resetting dead datanodes list

[Stanley Shi <ss...@gopivotal.com>]

I believe there's some command to show list of datanodes from CLI, using
parsing HTML is not a good idea. HTML page is intended to be read by human.
I also don't know how to refresh node list;



Regards,
*Stanley Shi,*



On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar
<as...@gmail.com>wrote:

> Hi,
> Hadoop-1's name node UI displays dead datanodes
> even if those instances are terminated and are not part of the cluster
> anymore.
> Is there a way to reset the dead datenode list without bouncing namenode ?
>
> This would help me in my script(which would run nightly) which parses the
> html page,terminates
> dead datanodes and resize the cluster.
> --
> Thanks,
> Ashwin
>
>
>

Re: Resetting dead datanodes list

[Stanley Shi <ss...@gopivotal.com>]

I believe there's some command to show list of datanodes from CLI, using
parsing HTML is not a good idea. HTML page is intended to be read by human.
I also don't know how to refresh node list;



Regards,
*Stanley Shi,*



On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar
<as...@gmail.com>wrote:

> Hi,
> Hadoop-1's name node UI displays dead datanodes
> even if those instances are terminated and are not part of the cluster
> anymore.
> Is there a way to reset the dead datenode list without bouncing namenode ?
>
> This would help me in my script(which would run nightly) which parses the
> html page,terminates
> dead datanodes and resize the cluster.
> --
> Thanks,
> Ashwin
>
>
>

Re: Resetting dead datanodes list

[Stanley Shi <ss...@gopivotal.com>]

I believe there's some command to show list of datanodes from CLI, using
parsing HTML is not a good idea. HTML page is intended to be read by human.
I also don't know how to refresh node list;



Regards,
*Stanley Shi,*



On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar
<as...@gmail.com>wrote:

> Hi,
> Hadoop-1's name node UI displays dead datanodes
> even if those instances are terminated and are not part of the cluster
> anymore.
> Is there a way to reset the dead datenode list without bouncing namenode ?
>
> This would help me in my script(which would run nightly) which parses the
> html page,terminates
> dead datanodes and resize the cluster.
> --
> Thanks,
> Ashwin
>
>
>