Persistent storage on EC2

[Malcolm Matalka <mm...@millennialmedia.com>]

I'm using EBS volumes to have a persistent HDFS on EC2.  Do I need to keep the master updated on how to map the internal IPs, which change as I understand, to a known set of host names so it knows where the blocks are located each time I bring a cluster up?  If so, is keeping a mapping up to date in /etc/hosts sufficient?

Thanks

Re: Persistent storage on EC2

[Kevin Peterson <kp...@biz360.com>]

On Tue, May 26, 2009 at 7:50 PM, Malcolm Matalka <
mmatalka@millennialmedia.com> wrote:

> I'm using EBS volumes to have a persistent HDFS on EC2.  Do I need to keep
> the master updated on how to map the internal IPs, which change as I
> understand, to a known set of host names so it knows where the blocks are
> located each time I bring a cluster up?  If so, is keeping a mapping up to
> date in /etc/hosts sufficient?
>

I can't answer your first question of whether it's necessary. The namenode
might be able to figure it out when the DNs report their blocks.

Our staging cluster uses the setup you describe, with /etc/hosts pushed out
to all the machines, and the EBS volumes always mounted on the same
hostname. This works great.