Typical Hardware for Name Node?

[Tom Wheeler <to...@gmail.com>]

I gave a presentation on Hadoop last night to a technical group
(http://lambdalounge.org/) here in St. Louis.  I pointed out that the
name node is currently a single point of failure and it's one place
where spending money for extremely reliable hardware is important.
Someone asked what type of machine people use for the name node, but I
could only describe what I've personally seen.

If anyone is willing to talk about their name node hardware, I'd love
to follow up at our next meeting with a more thorough answer than I
could give last night.

P.S. I've linked my presentation from the Wiki in case anyone's
interested in reading it:

    http://wiki.apache.org/hadoop/HadoopPresentation

-- 
Tom Wheeler
http://www.tomwheeler.com/

Re: Typical Hardware for Name Node?

[Taeho Kang <tk...@gmail.com>]

Also, throw in lots of RAM for the Namenode machine. The size of file system
(e.g. # of files) is dependent on the amount of RAM the Namenode can use.

On Sat, Oct 3, 2009 at 3:38 AM, Allen Wittenauer
<aw...@linkedin.com>wrote:

> On 10/2/09 10:53 AM, "Tom Wheeler" <to...@gmail.com> wrote:
> > If anyone is willing to talk about their name node hardware, I'd love
> > to follow up at our next meeting with a more thorough answer than I
> > could give last night.
>
> I suspect most people (and this includes both locations I've worked with
> Hadoop at), basically buy the same gear but just throw more memory in it,
> RAID, etc.
>
> We're exploring using IPMP on Solaris to protect us from network failures.
> The problem is that Hadoop really really hates multiple nics due to an
> inability to bind outbound traffic to a specific addr and/or uses IP
> address
> as ident information.
>
>

Re: Typical Hardware for Name Node?

[Allen Wittenauer <aw...@linkedin.com>]

On 10/2/09 10:53 AM, "Tom Wheeler" <to...@gmail.com> wrote:
> If anyone is willing to talk about their name node hardware, I'd love
> to follow up at our next meeting with a more thorough answer than I
> could give last night.

I suspect most people (and this includes both locations I've worked with
Hadoop at), basically buy the same gear but just throw more memory in it,
RAID, etc.

We're exploring using IPMP on Solaris to protect us from network failures.
The problem is that Hadoop really really hates multiple nics due to an
inability to bind outbound traffic to a specific addr and/or uses IP address
as ident information.