You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by John Meza <j_...@hotmail.com> on 2013/03/22 22:50:56 UTC
Cluster lost IP addresses
I have a 18 node cluster that had to be physically moved.Unfortunately all the ip addresses were lost (recreated).
This must have happened to someone before.Nothing else on the machines has been changed. Most importantly the data in HDFS is still sitting there.
Is there a way to recover this cluster to a useable state?thanksJohn
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
Hi Chris,
Where exactly are you seeing issues with change of NN/DN IPs? I've
never encountered trouble on IP changes (I keep moving across networks
everyday and the HDFS plus MR I run both stand tall without requiring
a restart). We do not store (generally) nor rely on IP addresses. An
exclusion may apply to files under construction I think, but a
properly shutdown cluster pre-move would not have that and such files
wouldn't matter too much in such a scenario anyway.
Obviously, a hostname change could cause issues.
FWIW, you can easily take any person's fsimage from across the world
and start your NN on top of that and add in new DNs with the block
data under them and setup the HDFS cluster. This is rather painless
and well-built and goes to show that its not really IP dependent in
any way. Please do elaborate.
On Sat, Mar 23, 2013 at 10:52 AM, Chris Embree <ce...@gmail.com> wrote:
> Hey John,
>
> Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
> are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
>
> BTW, NN devs. Seriously? You rely on IP addr instead of hostname? Someone
> should probably be shot or at least be responsible for fixing this
> abomination. Sad that this code was released GA.
>
> Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
> kind of silliness.
>
> In my Not So Humble Opinion.
> Chris
>
>
> On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> NameNode does not persist block locations; so this is still
>> recoverable if the configs are changed to use the new set of hostnames
>> to bind to/look up.
>>
>> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > it has issues, namenode save blockid->nodes, using ip addr if your
>> > slaves
>> > config file using ip addr instead of hostname.
>> >
>> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>> > <li...@balajin.net> wrote:
>> >>
>> >> Assuming you are using hostnAmes and not ip address in your config
>> >> files
>> >> What happens when you start the cluster? If you are using IP address in
>> >> your
>> >> configs just update them and start. It should work with no issues.
>> >>
>> >> On Friday, March 22, 2013, John Meza wrote:
>> >>>
>> >>> I have a 18 node cluster that had to be physically moved.
>> >>> Unfortunately all the ip addresses were lost (recreated).
>> >>>
>> >>> This must have happened to someone before.
>> >>> Nothing else on the machines has been changed. Most importantly the
>> >>> data
>> >>> in HDFS is still sitting there.
>> >>>
>> >>> Is there a way to recover this cluster to a useable state?
>> >>> thanks
>> >>> John
>> >>
>> >>
>> >>
>> >> --
>> >> http://balajin.net/blog
>> >> http://flic.kr/balajijegan
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
Hi Chris,
Where exactly are you seeing issues with change of NN/DN IPs? I've
never encountered trouble on IP changes (I keep moving across networks
everyday and the HDFS plus MR I run both stand tall without requiring
a restart). We do not store (generally) nor rely on IP addresses. An
exclusion may apply to files under construction I think, but a
properly shutdown cluster pre-move would not have that and such files
wouldn't matter too much in such a scenario anyway.
Obviously, a hostname change could cause issues.
FWIW, you can easily take any person's fsimage from across the world
and start your NN on top of that and add in new DNs with the block
data under them and setup the HDFS cluster. This is rather painless
and well-built and goes to show that its not really IP dependent in
any way. Please do elaborate.
On Sat, Mar 23, 2013 at 10:52 AM, Chris Embree <ce...@gmail.com> wrote:
> Hey John,
>
> Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
> are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
>
> BTW, NN devs. Seriously? You rely on IP addr instead of hostname? Someone
> should probably be shot or at least be responsible for fixing this
> abomination. Sad that this code was released GA.
>
> Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
> kind of silliness.
>
> In my Not So Humble Opinion.
> Chris
>
>
> On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> NameNode does not persist block locations; so this is still
>> recoverable if the configs are changed to use the new set of hostnames
>> to bind to/look up.
>>
>> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > it has issues, namenode save blockid->nodes, using ip addr if your
>> > slaves
>> > config file using ip addr instead of hostname.
>> >
>> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>> > <li...@balajin.net> wrote:
>> >>
>> >> Assuming you are using hostnAmes and not ip address in your config
>> >> files
>> >> What happens when you start the cluster? If you are using IP address in
>> >> your
>> >> configs just update them and start. It should work with no issues.
>> >>
>> >> On Friday, March 22, 2013, John Meza wrote:
>> >>>
>> >>> I have a 18 node cluster that had to be physically moved.
>> >>> Unfortunately all the ip addresses were lost (recreated).
>> >>>
>> >>> This must have happened to someone before.
>> >>> Nothing else on the machines has been changed. Most importantly the
>> >>> data
>> >>> in HDFS is still sitting there.
>> >>>
>> >>> Is there a way to recover this cluster to a useable state?
>> >>> thanks
>> >>> John
>> >>
>> >>
>> >>
>> >> --
>> >> http://balajin.net/blog
>> >> http://flic.kr/balajijegan
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
Hi Chris,
Where exactly are you seeing issues with change of NN/DN IPs? I've
never encountered trouble on IP changes (I keep moving across networks
everyday and the HDFS plus MR I run both stand tall without requiring
a restart). We do not store (generally) nor rely on IP addresses. An
exclusion may apply to files under construction I think, but a
properly shutdown cluster pre-move would not have that and such files
wouldn't matter too much in such a scenario anyway.
Obviously, a hostname change could cause issues.
FWIW, you can easily take any person's fsimage from across the world
and start your NN on top of that and add in new DNs with the block
data under them and setup the HDFS cluster. This is rather painless
and well-built and goes to show that its not really IP dependent in
any way. Please do elaborate.
On Sat, Mar 23, 2013 at 10:52 AM, Chris Embree <ce...@gmail.com> wrote:
> Hey John,
>
> Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
> are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
>
> BTW, NN devs. Seriously? You rely on IP addr instead of hostname? Someone
> should probably be shot or at least be responsible for fixing this
> abomination. Sad that this code was released GA.
>
> Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
> kind of silliness.
>
> In my Not So Humble Opinion.
> Chris
>
>
> On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> NameNode does not persist block locations; so this is still
>> recoverable if the configs are changed to use the new set of hostnames
>> to bind to/look up.
>>
>> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > it has issues, namenode save blockid->nodes, using ip addr if your
>> > slaves
>> > config file using ip addr instead of hostname.
>> >
>> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>> > <li...@balajin.net> wrote:
>> >>
>> >> Assuming you are using hostnAmes and not ip address in your config
>> >> files
>> >> What happens when you start the cluster? If you are using IP address in
>> >> your
>> >> configs just update them and start. It should work with no issues.
>> >>
>> >> On Friday, March 22, 2013, John Meza wrote:
>> >>>
>> >>> I have a 18 node cluster that had to be physically moved.
>> >>> Unfortunately all the ip addresses were lost (recreated).
>> >>>
>> >>> This must have happened to someone before.
>> >>> Nothing else on the machines has been changed. Most importantly the
>> >>> data
>> >>> in HDFS is still sitting there.
>> >>>
>> >>> Is there a way to recover this cluster to a useable state?
>> >>> thanks
>> >>> John
>> >>
>> >>
>> >>
>> >> --
>> >> http://balajin.net/blog
>> >> http://flic.kr/balajijegan
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
Hi Chris,
Where exactly are you seeing issues with change of NN/DN IPs? I've
never encountered trouble on IP changes (I keep moving across networks
everyday and the HDFS plus MR I run both stand tall without requiring
a restart). We do not store (generally) nor rely on IP addresses. An
exclusion may apply to files under construction I think, but a
properly shutdown cluster pre-move would not have that and such files
wouldn't matter too much in such a scenario anyway.
Obviously, a hostname change could cause issues.
FWIW, you can easily take any person's fsimage from across the world
and start your NN on top of that and add in new DNs with the block
data under them and setup the HDFS cluster. This is rather painless
and well-built and goes to show that its not really IP dependent in
any way. Please do elaborate.
On Sat, Mar 23, 2013 at 10:52 AM, Chris Embree <ce...@gmail.com> wrote:
> Hey John,
>
> Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
> are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
>
> BTW, NN devs. Seriously? You rely on IP addr instead of hostname? Someone
> should probably be shot or at least be responsible for fixing this
> abomination. Sad that this code was released GA.
>
> Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
> kind of silliness.
>
> In my Not So Humble Opinion.
> Chris
>
>
> On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> NameNode does not persist block locations; so this is still
>> recoverable if the configs are changed to use the new set of hostnames
>> to bind to/look up.
>>
>> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > it has issues, namenode save blockid->nodes, using ip addr if your
>> > slaves
>> > config file using ip addr instead of hostname.
>> >
>> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
>> > <li...@balajin.net> wrote:
>> >>
>> >> Assuming you are using hostnAmes and not ip address in your config
>> >> files
>> >> What happens when you start the cluster? If you are using IP address in
>> >> your
>> >> configs just update them and start. It should work with no issues.
>> >>
>> >> On Friday, March 22, 2013, John Meza wrote:
>> >>>
>> >>> I have a 18 node cluster that had to be physically moved.
>> >>> Unfortunately all the ip addresses were lost (recreated).
>> >>>
>> >>> This must have happened to someone before.
>> >>> Nothing else on the machines has been changed. Most importantly the
>> >>> data
>> >>> in HDFS is still sitting there.
>> >>>
>> >>> Is there a way to recover this cluster to a useable state?
>> >>> thanks
>> >>> John
>> >>
>> >>
>> >>
>> >> --
>> >> http://balajin.net/blog
>> >> http://flic.kr/balajijegan
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Cluster lost IP addresses
Posted by Chris Embree <ce...@gmail.com>.
Hey John,
Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
BTW, NN devs. Seriously? You rely on IP addr instead of hostname?
Someone should probably be shot or at least be responsible for fixing
this abomination. Sad that this code was released GA.
Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
kind of silliness.
In my Not So Humble Opinion.
Chris
On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Chris Embree <ce...@gmail.com>.
Hey John,
Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
BTW, NN devs. Seriously? You rely on IP addr instead of hostname?
Someone should probably be shot or at least be responsible for fixing
this abomination. Sad that this code was released GA.
Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
kind of silliness.
In my Not So Humble Opinion.
Chris
On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
RE: Cluster lost IP addresses
Posted by John Meza <j_...@hotmail.com>.
After 2 hard restarts of all nodes, the cluster is back. But 7 of 18 nodes are dead. the NN is aware of all nodes, data (of non-dead) nodes is available.
We're pulling off data now, will give hardware back to the HW team next week.
*the emails in response to my original are very helpful in learning more about Hadoop concepts.many thanksJohn
Date: Sat, 23 Mar 2013 13:45:18 +0800
Subject: Re: Cluster lost IP addresses
From: azuryyyu@gmail.com
To: user@hadoop.apache.org
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
RE: Cluster lost IP addresses
Posted by John Meza <j_...@hotmail.com>.
After 2 hard restarts of all nodes, the cluster is back. But 7 of 18 nodes are dead. the NN is aware of all nodes, data (of non-dead) nodes is available.
We're pulling off data now, will give hardware back to the HW team next week.
*the emails in response to my original are very helpful in learning more about Hadoop concepts.many thanksJohn
Date: Sat, 23 Mar 2013 13:45:18 +0800
Subject: Re: Cluster lost IP addresses
From: azuryyyu@gmail.com
To: user@hadoop.apache.org
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
RE: Cluster lost IP addresses
Posted by John Meza <j_...@hotmail.com>.
After 2 hard restarts of all nodes, the cluster is back. But 7 of 18 nodes are dead. the NN is aware of all nodes, data (of non-dead) nodes is available.
We're pulling off data now, will give hardware back to the HW team next week.
*the emails in response to my original are very helpful in learning more about Hadoop concepts.many thanksJohn
Date: Sat, 23 Mar 2013 13:45:18 +0800
Subject: Re: Cluster lost IP addresses
From: azuryyyu@gmail.com
To: user@hadoop.apache.org
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
RE: Cluster lost IP addresses
Posted by John Meza <j_...@hotmail.com>.
After 2 hard restarts of all nodes, the cluster is back. But 7 of 18 nodes are dead. the NN is aware of all nodes, data (of non-dead) nodes is available.
We're pulling off data now, will give hardware back to the HW team next week.
*the emails in response to my original are very helpful in learning more about Hadoop concepts.many thanksJohn
Date: Sat, 23 Mar 2013 13:45:18 +0800
Subject: Re: Cluster lost IP addresses
From: azuryyyu@gmail.com
To: user@hadoop.apache.org
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Chris Embree <ce...@gmail.com>.
Hey John,
Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
BTW, NN devs. Seriously? You rely on IP addr instead of hostname?
Someone should probably be shot or at least be responsible for fixing
this abomination. Sad that this code was released GA.
Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
kind of silliness.
In my Not So Humble Opinion.
Chris
On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Chris Embree <ce...@gmail.com>.
Hey John,
Make sure your /etc/hosts ( or DNS) is up to date and any topology scripts
are updated. Unfortunately, NN is pretty dumb about IP's vs. Hostnames.
BTW, NN devs. Seriously? You rely on IP addr instead of hostname?
Someone should probably be shot or at least be responsible for fixing
this abomination. Sad that this code was released GA.
Sorry folks. HDFS/Mapred is really cool tech, I'm just jaded about this
kind of silliness.
In my Not So Humble Opinion.
Chris
On Sat, Mar 23, 2013 at 1:12 AM, Harsh J <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
oh,yes,it's not persist, only in memory. so there is no issue.
On Mar 23, 2013 1:13 PM, "Harsh J" <ha...@cloudera.com> wrote:
> NameNode does not persist block locations; so this is still
> recoverable if the configs are changed to use the new set of hostnames
> to bind to/look up.
>
> On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> > it has issues, namenode save blockid->nodes, using ip addr if your slaves
> > config file using ip addr instead of hostname.
> >
> > On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> > <li...@balajin.net> wrote:
> >>
> >> Assuming you are using hostnAmes and not ip address in your config files
> >> What happens when you start the cluster? If you are using IP address in
> your
> >> configs just update them and start. It should work with no issues.
> >>
> >> On Friday, March 22, 2013, John Meza wrote:
> >>>
> >>> I have a 18 node cluster that had to be physically moved.
> >>> Unfortunately all the ip addresses were lost (recreated).
> >>>
> >>> This must have happened to someone before.
> >>> Nothing else on the machines has been changed. Most importantly the
> data
> >>> in HDFS is still sitting there.
> >>>
> >>> Is there a way to recover this cluster to a useable state?
> >>> thanks
> >>> John
> >>
> >>
> >>
> >> --
> >> http://balajin.net/blog
> >> http://flic.kr/balajijegan
>
>
>
> --
> Harsh J
>
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
Re: Cluster lost IP addresses
Posted by Harsh J <ha...@cloudera.com>.
NameNode does not persist block locations; so this is still
recoverable if the configs are changed to use the new set of hostnames
to bind to/look up.
On Sat, Mar 23, 2013 at 9:01 AM, Azuryy Yu <az...@gmail.com> wrote:
> it has issues, namenode save blockid->nodes, using ip addr if your slaves
> config file using ip addr instead of hostname.
>
> On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)"
> <li...@balajin.net> wrote:
>>
>> Assuming you are using hostnAmes and not ip address in your config files
>> What happens when you start the cluster? If you are using IP address in your
>> configs just update them and start. It should work with no issues.
>>
>> On Friday, March 22, 2013, John Meza wrote:
>>>
>>> I have a 18 node cluster that had to be physically moved.
>>> Unfortunately all the ip addresses were lost (recreated).
>>>
>>> This must have happened to someone before.
>>> Nothing else on the machines has been changed. Most importantly the data
>>> in HDFS is still sitting there.
>>>
>>> Is there a way to recover this cluster to a useable state?
>>> thanks
>>> John
>>
>>
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
--
Harsh J
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
it has issues, namenode save blockid->nodes, using ip addr if your slaves
config file using ip addr instead of hostname.
On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)" <
lists@balajin.net> wrote:
> Assuming you are using hostnAmes and not ip address in your config
> files What happens when you start the cluster? If you are using IP address
> in your configs just update them and start. It should work with no issues.
>
> On Friday, March 22, 2013, John Meza wrote:
>
>> I have a 18 node cluster that had to be physically moved.
>> Unfortunately all the ip addresses were lost (recreated).
>>
>> This must have happened to someone before.
>> Nothing else on the machines has been changed. Most importantly the data
>> in HDFS is still sitting there.
>>
>> Is there a way to recover this cluster to a useable state?
>> thanks
>> John
>>
>
>
> --
> http://balajin.net/blog
> http://flic.kr/balajijegan
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
it has issues, namenode save blockid->nodes, using ip addr if your slaves
config file using ip addr instead of hostname.
On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)" <
lists@balajin.net> wrote:
> Assuming you are using hostnAmes and not ip address in your config
> files What happens when you start the cluster? If you are using IP address
> in your configs just update them and start. It should work with no issues.
>
> On Friday, March 22, 2013, John Meza wrote:
>
>> I have a 18 node cluster that had to be physically moved.
>> Unfortunately all the ip addresses were lost (recreated).
>>
>> This must have happened to someone before.
>> Nothing else on the machines has been changed. Most importantly the data
>> in HDFS is still sitting there.
>>
>> Is there a way to recover this cluster to a useable state?
>> thanks
>> John
>>
>
>
> --
> http://balajin.net/blog
> http://flic.kr/balajijegan
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
it has issues, namenode save blockid->nodes, using ip addr if your slaves
config file using ip addr instead of hostname.
On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)" <
lists@balajin.net> wrote:
> Assuming you are using hostnAmes and not ip address in your config
> files What happens when you start the cluster? If you are using IP address
> in your configs just update them and start. It should work with no issues.
>
> On Friday, March 22, 2013, John Meza wrote:
>
>> I have a 18 node cluster that had to be physically moved.
>> Unfortunately all the ip addresses were lost (recreated).
>>
>> This must have happened to someone before.
>> Nothing else on the machines has been changed. Most importantly the data
>> in HDFS is still sitting there.
>>
>> Is there a way to recover this cluster to a useable state?
>> thanks
>> John
>>
>
>
> --
> http://balajin.net/blog
> http://flic.kr/balajijegan
>
Re: Cluster lost IP addresses
Posted by Azuryy Yu <az...@gmail.com>.
it has issues, namenode save blockid->nodes, using ip addr if your slaves
config file using ip addr instead of hostname.
On Mar 23, 2013 10:14 AM, "Balaji Narayanan (பாலாஜி நாராயணன்)" <
lists@balajin.net> wrote:
> Assuming you are using hostnAmes and not ip address in your config
> files What happens when you start the cluster? If you are using IP address
> in your configs just update them and start. It should work with no issues.
>
> On Friday, March 22, 2013, John Meza wrote:
>
>> I have a 18 node cluster that had to be physically moved.
>> Unfortunately all the ip addresses were lost (recreated).
>>
>> This must have happened to someone before.
>> Nothing else on the machines has been changed. Most importantly the data
>> in HDFS is still sitting there.
>>
>> Is there a way to recover this cluster to a useable state?
>> thanks
>> John
>>
>
>
> --
> http://balajin.net/blog
> http://flic.kr/balajijegan
>
Re: Cluster lost IP addresses
Posted by "Balaji Narayanan (பாலாஜி நாராயணன்)" <li...@balajin.net>.
Assuming you are using hostnAmes and not ip address in your config
files What happens when you start the cluster? If you are using IP address
in your configs just update them and start. It should work with no issues.
On Friday, March 22, 2013, John Meza wrote:
> I have a 18 node cluster that had to be physically moved.
> Unfortunately all the ip addresses were lost (recreated).
>
> This must have happened to someone before.
> Nothing else on the machines has been changed. Most importantly the data
> in HDFS is still sitting there.
>
> Is there a way to recover this cluster to a useable state?
> thanks
> John
>
--
http://balajin.net/blog
http://flic.kr/balajijegan
Re: Cluster lost IP addresses
Posted by "Balaji Narayanan (பாலாஜி நாராயணன்)" <li...@balajin.net>.
Assuming you are using hostnAmes and not ip address in your config
files What happens when you start the cluster? If you are using IP address
in your configs just update them and start. It should work with no issues.
On Friday, March 22, 2013, John Meza wrote:
> I have a 18 node cluster that had to be physically moved.
> Unfortunately all the ip addresses were lost (recreated).
>
> This must have happened to someone before.
> Nothing else on the machines has been changed. Most importantly the data
> in HDFS is still sitting there.
>
> Is there a way to recover this cluster to a useable state?
> thanks
> John
>
--
http://balajin.net/blog
http://flic.kr/balajijegan
Re: Cluster lost IP addresses
Posted by "Balaji Narayanan (பாலாஜி நாராயணன்)" <li...@balajin.net>.
Assuming you are using hostnAmes and not ip address in your config
files What happens when you start the cluster? If you are using IP address
in your configs just update them and start. It should work with no issues.
On Friday, March 22, 2013, John Meza wrote:
> I have a 18 node cluster that had to be physically moved.
> Unfortunately all the ip addresses were lost (recreated).
>
> This must have happened to someone before.
> Nothing else on the machines has been changed. Most importantly the data
> in HDFS is still sitting there.
>
> Is there a way to recover this cluster to a useable state?
> thanks
> John
>
--
http://balajin.net/blog
http://flic.kr/balajijegan
Re: Cluster lost IP addresses
Posted by "Balaji Narayanan (பாலாஜி நாராயணன்)" <li...@balajin.net>.
Assuming you are using hostnAmes and not ip address in your config
files What happens when you start the cluster? If you are using IP address
in your configs just update them and start. It should work with no issues.
On Friday, March 22, 2013, John Meza wrote:
> I have a 18 node cluster that had to be physically moved.
> Unfortunately all the ip addresses were lost (recreated).
>
> This must have happened to someone before.
> Nothing else on the machines has been changed. Most importantly the data
> in HDFS is still sitting there.
>
> Is there a way to recover this cluster to a useable state?
> thanks
> John
>
--
http://balajin.net/blog
http://flic.kr/balajijegan