You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Some Body <so...@squareplanet.de> on 2010/07/06 16:31:58 UTC

decomission a node

Hi,

Is it possible move all the data blocks off a cluster node and then decommision the node?

I'm asking because,  now that my MR job is working, I'd like see how things scale. I.e.,
 less processing nodes, amount of data (number & size of files, etc.). I currently have 8 nodes, 
and am processing 5GB spread across 2000 files. 

Alan

Re: decomission a node

Posted by Ken Goodhope <ke...@gmail.com>.
Inside the hdfs conf,

   <property>
      <name>dfs.hosts.exclude</name>
      <value></value>
      <description>Names a file that contains a list of hosts that are
      not permitted to connect to the namenode.  The full pathname of the
      file must be specified.  If the value is empty, no hosts are
      excluded.</description>
   </property>

Point this property at a file containing a list of nodes you want to
decommision.  From there, use the command line "hadoop dfsadmin
-refreshNodes".


On Tue, Jul 6, 2010 at 7:31 AM, Some Body <so...@squareplanet.de> wrote:

> Hi,
>
> Is it possible move all the data blocks off a cluster node and then
> decommision the node?
>
> I'm asking because,  now that my MR job is working, I'd like see how things
> scale. I.e.,
>  less processing nodes, amount of data (number & size of files, etc.). I
> currently have 8 nodes,
> and am processing 5GB spread across 2000 files.
>
> Alan
>

RE: decomission a node

Posted by Michael Segel <mi...@hotmail.com>.

This is pretty predictable.
Determine the average time it takes to process a m/r task.
If you can process 100 m/r tasks simultaneously, and then cut that to 50 m/r tasks you can handle simultaneously, your job will take twice as long to run.

Granted this will give you a rough estimate of how long it will take your job to run.

HTH

-Mike

> Date: Wed, 7 Jul 2010 09:00:47 +0200
> From: somebody@squareplanet.de
> To: common-user@hadoop.apache.org
> Subject: Re: decomission a node
> 
> Yes the effect of "scaling down" was the first thing I wanted to look at.
> To process X GB it currently takes Y seconds with Z nodes.
> If I process X GB with Z/2 nodes, does it take Y/2 seconds?
> How about Z-1,Z-2,Z-3,.... nodes?
> 
> Right now my MR job process alot of small files (2000 files, @2.5MB each)
> individually, so the next test would involve changing my MR job to combine
> the small files into bigger pieces (closer to hdfs block size) and see 
> if that
> is more effective.
> 
> Each line of my small files has a timestamp column and 55 columns with
> numerical data and my reducer needs to calc the column averages for
> certain time periods (last day, last hour,etc.) based on the timestamp.
> 
> Alan
> 
> On 07/06/2010 08:06 PM, Allen Wittenauer wrote:
> > On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:
> >    
> >> I'm also not sure how dropping a node will test the scalability. You would be testing resilience.
> >>      
> > He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).
> >
> > In other words, "if I drop a node, how much of a performance hit is my job going to take?"
> >
> > Also, for this type of timing/testing, I'd probably make sure speculative execution is off.   It will likely throw some curve balls into the time.
> 
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

Re: decomission a node

Posted by Alan Miller <so...@squareplanet.de>.
Yes the effect of "scaling down" was the first thing I wanted to look at.
To process X GB it currently takes Y seconds with Z nodes.
If I process X GB with Z/2 nodes, does it take Y/2 seconds?
How about Z-1,Z-2,Z-3,.... nodes?

Right now my MR job process alot of small files (2000 files, @2.5MB each)
individually, so the next test would involve changing my MR job to combine
the small files into bigger pieces (closer to hdfs block size) and see 
if that
is more effective.

Each line of my small files has a timestamp column and 55 columns with
numerical data and my reducer needs to calc the column averages for
certain time periods (last day, last hour,etc.) based on the timestamp.

Alan

On 07/06/2010 08:06 PM, Allen Wittenauer wrote:
> On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:
>    
>> I'm also not sure how dropping a node will test the scalability. You would be testing resilience.
>>      
> He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).
>
> In other words, "if I drop a node, how much of a performance hit is my job going to take?"
>
> Also, for this type of timing/testing, I'd probably make sure speculative execution is off.   It will likely throw some curve balls into the time.


Re: decomission a node

Posted by Allen Wittenauer <aw...@linkedin.com>.
On Jul 6, 2010, at 8:35 AM, Michael Segel wrote:
> I'm also not sure how dropping a node will test the scalability. You would be testing resilience.

He's testing scale down, not scale up (which is the way we normally think of things... I was confused by the wording too).

In other words, "if I drop a node, how much of a performance hit is my job going to take?"

Also, for this type of timing/testing, I'd probably make sure speculative execution is off.   It will likely throw some curve balls into the time.

RE: decomission a node

Posted by Michael Segel <mi...@hotmail.com>.
Alan,

You shouldn't need to worry about moving the data blocks off your node, assuming you're replicating your blocks 3x times. (I think its the default...)

You can bring down your node, and within 15 mins, Hadoop will recognize that node as down...

I think if you do a $> hadoop fsck / your system will recognize that those blocks are under replicated and will replicate the blocks on another machine. 

If you're decommissioning ?sp? a node, then you're taking the machine out of the cluster permanently.

I'm also not sure how dropping a node will test the scalability. You would be testing resilience.

HTH

-Mike
 

> Date: Tue, 6 Jul 2010 16:31:58 +0200
> To: common-user@hadoop.apache.org
> From: somebody@squareplanet.de
> Subject: decomission a node
> 
> Hi,
> 
> Is it possible move all the data blocks off a cluster node and then decommision the node?
> 
> I'm asking because,  now that my MR job is working, I'd like see how things scale. I.e.,
>  less processing nodes, amount of data (number & size of files, etc.). I currently have 8 nodes, 
> and am processing 5GB spread across 2000 files. 
> 
> Alan
 		 	   		  
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2