Changing hostnames of tasktracker/datanode nodes - any problems?

[Erik Forsberg <fo...@opera.com>]

Hi!

Due to network reconfigurations, I need to change the hostnames of some
of my worker nodes, i.e. the nodes running tasktracker and datanode. I
need to do this to make my hostname naming schema actually concur with
the network setup.

Are there any problems doing this? It seems HDFS identifies things
using some kind of hostname-independent UID, so I guess it should
recover from one machine going down and then appear again under another
hostname?

Thanks,
\EF
-- 
Erik Forsberg <fo...@opera.com>
Developer, Opera Software - http://www.opera.com/

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Bill Graham <bi...@gmail.com>]

Ahh yes of course, distcp. Thanks!

On Tue, Aug 10, 2010 at 11:01 AM, Allen Wittenauer <awittenauer@linkedin.com
> wrote:

>
> On Aug 10, 2010, at 10:54 AM, Bill Graham wrote:
> > Is is correct to say that that would work fine? We have a replication
> factor
> > of 2, so we'd be copying twice as much data as we'd need to so I'm sure
> > there's a more efficient approach.
>
> It should work fine.  But yes, highly inefficient.
>
> > What about adding the new nodes in the new colo to the existing cluster,
> > rebalancing and then decommissioning the old cluster nodes before finally
> > migrating the NN/SNN? I know Hadoop isn't intended to run cross-colo, but
> > would this be a more efficient approach than the one above?
>
> If you can keep both grids up at the same time, use distcp to do the copy.
>  This will make sure the blocks get copied once, will keep permissions with
> -p, keep the replication factor, redistribute data (free balancing!), etc,
> etc, etc.
>
>
>
>
>

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Allen Wittenauer <aw...@linkedin.com>]

On Aug 10, 2010, at 10:54 AM, Bill Graham wrote:
> Is is correct to say that that would work fine? We have a replication factor
> of 2, so we'd be copying twice as much data as we'd need to so I'm sure
> there's a more efficient approach.

It should work fine.  But yes, highly inefficient.

> What about adding the new nodes in the new colo to the existing cluster,
> rebalancing and then decommissioning the old cluster nodes before finally
> migrating the NN/SNN? I know Hadoop isn't intended to run cross-colo, but
> would this be a more efficient approach than the one above?

If you can keep both grids up at the same time, use distcp to do the copy.  This will make sure the blocks get copied once, will keep permissions with -p, keep the replication factor, redistribute data (free balancing!), etc, etc, etc.

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Bill Graham <bi...@gmail.com>]

Sorry to hijack the thread but I have a similar use case.

In a few months we're going to be moving colos. The new cluster will be the
same size as the current cluster and some downtime is acceptable. The
hostnames will be different. From what I've read in this thread it seems
like it would be safe to do the following:

1. Build out the new cluster without starting it.
2. Shut down the entire old cluster (NN, SNN, DNs)
3. scp the relevant data and name dirs for each host to the new hardware.
4. Start the new cluster

Is is correct to say that that would work fine? We have a replication factor
of 2, so we'd be copying twice as much data as we'd need to so I'm sure
there's a more efficient approach.

What about adding the new nodes in the new colo to the existing cluster,
rebalancing and then decommissioning the old cluster nodes before finally
migrating the NN/SNN? I know Hadoop isn't intended to run cross-colo, but
would this be a more efficient approach than the one above?



On Tue, Aug 10, 2010 at 8:59 AM, Allen Wittenauer
<aw...@linkedin.com>wrote:

>
> On Aug 10, 2010, at 7:07 AM, Brian Bockelman wrote:
>
> > Hi Erik,
> >
> > You can also do this one-by-one (aka, a rolling reboot).  Shut it down,
> wait for it to be recognized as dead, then bring it back up with a new
> hostname.  It will take a much longer time, but you won't have any decrease
> in availability, just some minor decrease in capacity.
>
> ... and potentially problems with dfs.hosts.
>
>
>

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Allen Wittenauer <aw...@linkedin.com>]

On Aug 10, 2010, at 7:07 AM, Brian Bockelman wrote:

> Hi Erik,
> 
> You can also do this one-by-one (aka, a rolling reboot).  Shut it down, wait for it to be recognized as dead, then bring it back up with a new hostname.  It will take a much longer time, but you won't have any decrease in availability, just some minor decrease in capacity.

... and potentially problems with dfs.hosts.

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Brian Bockelman <bb...@cse.unl.edu>]

Hi Erik,

You can also do this one-by-one (aka, a rolling reboot).  Shut it down, wait for it to be recognized as dead, then bring it back up with a new hostname.  It will take a much longer time, but you won't have any decrease in availability, just some minor decrease in capacity.

This is useful in sites like ours where we have 24/7 usage and try to avoid any unnecessary downtime.

Brian

On Aug 10, 2010, at 8:42 AM, Allen Wittenauer wrote:

> 
> On Aug 10, 2010, at 3:51 AM, Erik Forsberg wrote:
> 
>> Hi!
>> 
>> Due to network reconfigurations, I need to change the hostnames of some
>> of my worker nodes, i.e. the nodes running tasktracker and datanode. I
>> need to do this to make my hostname naming schema actually concur with
>> the network setup.
>> 
>> Are there any problems doing this? It seems HDFS identifies things
>> using some kind of hostname-independent UID, so I guess it should
>> recover from one machine going down and then appear again under another
>> hostname?
> 
> 
> The name node metadata only matches blocks #s to files.  The data node to block part is determined at run time.  So it is perfectly safe and legal to:
> 
> a) bring grid down
> b) rename everything
> c) fix config files
> d) bring grid up
> 
> Don't forget to change your network topology script, slaves, and dfs.hosts as necessary.  If fsck complains about topology violations, use setrep to increase then setrep to decrease the violated files.

Re: Changing hostnames of tasktracker/datanode nodes - any problems?

[Allen Wittenauer <aw...@linkedin.com>]

On Aug 10, 2010, at 3:51 AM, Erik Forsberg wrote:

> Hi!
> 
> Due to network reconfigurations, I need to change the hostnames of some
> of my worker nodes, i.e. the nodes running tasktracker and datanode. I
> need to do this to make my hostname naming schema actually concur with
> the network setup.
> 
> Are there any problems doing this? It seems HDFS identifies things
> using some kind of hostname-independent UID, so I guess it should
> recover from one machine going down and then appear again under another
> hostname?


The name node metadata only matches blocks #s to files.  The data node to block part is determined at run time.  So it is perfectly safe and legal to:

a) bring grid down
b) rename everything
c) fix config files
d) bring grid up

Don't forget to change your network topology script, slaves, and dfs.hosts as necessary.  If fsck complains about topology violations, use setrep to increase then setrep to decrease the violated files.