How does decommissioning work

[Bryan Beaudreault <bb...@hubspot.com>]

Hello,

I'm using cdh3u2, if it matters.  I'm using the dfs.exclude.hosts to
decommission a good percentage of my cluster as I scale it down for a
period of time.  I'm just trying to understand how hdfs goes about this,
because I haven't found anything more than a "how to use" documentation for
the feature.

When I look at the name node UI, I see under replicated blocks count go up
when I decommission.  Also, when I look at the dfsnodelist with
whatNodes=decommissioning, there are stats there like Blocks with no live
replicas, etc.

When I decommission a node does it immediately make that node unavailable,
thus these stats?  Or does it move them off safely and these counts are
just to say what would happen if the node was shut down without
decommission?  Something else?

Thanks for any insight.

Re: How does decommissioning work

[Suresh Srinivas <su...@hortonworks.com>]

Bryan,

I am going to assume that you know about replication factor per file,
block, replicas etc.

Nodes are marked for decommissioned by adding it to excludes file (btw you
mean dfs.hosts.exclude right?).
HDFS marks these datanodes as decommissioning. HDFS no longer counts those
replicas towards the replication factor. This results in increased number
of under replicated blocks. HDFS starts replicating this under replicated
blocks, preferring decommissioning node as the source as much as possible.

At this time, the decommissioning nodes are used reads only.
Decommissioning completes when replication completes and replicas from
those nodes are no more needed. The node is then marked decommissioned.

Not sure I answered your questions.

Regards,
Suresh

On Tue, Sep 18, 2012 at 4:01 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> wrote:

> Hello,
>
> I'm using cdh3u2, if it matters.  I'm using the dfs.exclude.hosts to
> decommission a good percentage of my cluster as I scale it down for a
> period of time.  I'm just trying to understand how hdfs goes about this,
> because I haven't found anything more than a "how to use" documentation for
> the feature.
>
> When I look at the name node UI, I see under replicated blocks count go up
> when I decommission.  Also, when I look at the dfsnodelist with
> whatNodes=decommissioning, there are stats there like Blocks with no live
> replicas, etc.
>
> When I decommission a node does it immediately make that node unavailable,
> thus these stats?  Or does it move them off safely and these counts are
> just to say what would happen if the node was shut down without
> decommission?  Something else?
>
> Thanks for any insight.
>



-- 
http://hortonworks.com/download/