Re: Adding hard-disks to an existing HDFS cluster

[Steve Loughran <st...@apache.org>]

Eli Collins wrote:
>> I presume it makes no sense to try to spread the NameNode across multiple
>> disks?
> 
> Not quite sure what you mean here, but dfs.name.dir (where the NN
> stores its metadata) should have multiple directories on different
> disks to guard against the failure of any single disk. Many people
> also use RAIDed disks and include an NFS mount in dfs.name.dir to have
> additional, reliable copies of this data.

Best of all: a secondary namenode to get the streamed event log, as that 
will mean your cluster restarts faster. You do not want to lose your NN 
data.


We've discussed making it easier to hot-swap a disk drive on a live 
datanode, that could be simplified by having the DN take all data off 
that disks onto its other disks (if there is room), and then copying it 
back in later. Nobody has done any work on this yet:
http://issues.apache.org/jira/browse/HDFS-664

Re: Adding hard-disks to an existing HDFS cluster

[Marc Farnum Rendino <mv...@gmail.com>]

On Mon, Mar 1, 2010 at 2:02 PM, Eli Collins <el...@cloudera.com> wrote:

> Yes, it is important that the NN metadata (the fsimage and edits log)
> are stored reliably.  There's a great chapter on HDFS administration
> in the book Hadoop: The Definitive Guide.


Ah; I'd discounted it, because the hadoop ecosystem seems to be evolving so
fast, it seemed better to focus on online and recently-updated info.

Maybe it's not quite that divergent yet. :)

Thanks,

Marc

Re: Adding hard-disks to an existing HDFS cluster

[Eli Collins <el...@cloudera.com>]

On Mon, Mar 1, 2010 at 8:21 AM, Marc Farnum Rendino <mv...@gmail.com> wrote:
> On Mon, Mar 1, 2010 at 5:48 AM, Steve Loughran <st...@apache.org> wrote:
>
>> Best of all: a secondary namenode to get the streamed event log, as that
>> will mean your cluster restarts faster. You do not want to lose your NN
>> data.
>>
>
> If the NN data is lost, all the HDFS data is functionally lost, right?
>

Yes, it is important that the NN metadata (the fsimage and edits log)
are stored reliably.  There's a great chapter on HDFS administration
in the book Hadoop: The Definitive Guide.

Thanks,
Eli

Re: Adding hard-disks to an existing HDFS cluster

[Marc Farnum Rendino <mv...@gmail.com>]

On Mon, Mar 1, 2010 at 5:48 AM, Steve Loughran <st...@apache.org> wrote:

> Best of all: a secondary namenode to get the streamed event log, as that
> will mean your cluster restarts faster. You do not want to lose your NN
> data.
>

If the NN data is lost, all the HDFS data is functionally lost, right?

- Marc