You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jagadeesh <ja...@x-minds.org> on 2006/11/01 17:03:29 UTC
Some queries on Master node
Hi,
I have a cluster of more than 25 servers which basically serves the purpose
for storage. However I was wondering what will happen if the master node
gets exhausted of disk space. If I add more nodes to the cluster,
1. Will Hadoop move the blocks in master node to the newly added nodes?
2. Is there any parameter where I can specify not to write file chunks /
blocks in the master node and rather always use other nodes in the cluster?
Since there is always a possibility for single point failure, I would like
to keep the master node as secure as possible.
I am planning to go live with this setup by end of this week and I really
appreciate if you can send me a reply for the above queries.
Thanks
Jugs
Re: Some queries on Master node
Posted by Vetle Roeim <ve...@opera.com>.
On Thu, 02 Nov 2006 20:15:34 +0100, Konstantin Shvachko
<sh...@yahoo-inc.com> wrote:
[...]
>>> A simpler way, with no interruption of service, is to merely turn up
>>> the
>>> replication of files, wait for transfers to stabilize, then turn the
>>> replication back down.
>>
>>
>> Can this be done without having to restart DFS?
>
> bin/hadoop -dfs setrep 7 -R /directory
Thanks.
--
Vetle Roeim
Team Manager, Information Systems
Opera Software ASA <URL: http://www.opera.com/ >
RE: Some queries on Master node
Posted by Yoram Arnon <ya...@yahoo-inc.com>.
Very cool, thanks.
Yoram
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv@yahoo-inc.com]
> Sent: Thursday, November 02, 2006 5:30 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Some queries on Master node
>
> It is not getting empty, because a replica to be removed for an
> over-replicated block is chosen randomly.
> So when the data-node is restarted only a part of its blocks will
> removed, not all of them.
> In practice I see 10 - 20% reduction of blocks.
> I actually used this procedure because some nodes were so full I had
> problems starting them.
>
> Yoram Arnon wrote:
>
> >Right, it's good as an emergency measure, but the
> disadvantage is that the
> >full node now becomes empty, so you're not really
> rebalancing, just cleaning
> >up a node. If you only want to remove say 20% of a node's
> contents, you need
> >more subtle measures.
> >
> >
> >
> >
> >>-----Original Message-----
> >>From: Konstantin Shvachko [mailto:shv@yahoo-inc.com]
> >>Sent: Thursday, November 02, 2006 11:16 AM
> >>To: hadoop-user@lucene.apache.org
> >>Subject: Re: Some queries on Master node
> >>
> >>Yet another way to re-balance blocks is to turn off the
> node which is
> >>full, wait
> >>until its blocks are replicated, and then bring it back again. The
> >>over-replicated
> >>blocks will be randomly removed from different nodes, so you
> >>really get them
> >>rebalanced not just removed from the current node.
> >>
> >>Vetle Roeim wrote:
> >>
> >>
> >>
> >>>On Thu, 02 Nov 2006 00:55:24 +0100, Bryan A. P. Pendleton
> >>><bp...@geekdom.net> wrote:
> >>>
> >>>
> >>>
> >>>>>>1. Will Hadoop move the blocks in master node to the
> >>>>>>
> >>>>>>
> >>newly added
> >>
> >>
> >>>>>nodes?
> >>>>>
> >>>>>No, HDFS will not move blocks to new nodes automatically.
> >>>>>
> >>>>>
> >>New files
> >>
> >>
> >>>>>that
> >>>>>are added will likely have their blocks placed on the new
> >>>>>
> >>>>>
> >>nodes. Also,
> >>
> >>
> >>>>>removing old files will remove some blocks that are on
> your older
> >>>>>nodes.
> >>>>>
> >>>>>One way to re-balance your cluster would be to:
> >>>>>- Select a subset of files that take up a good percentage
> >>>>>
> >>>>>
> >>of your disk
> >>
> >>
> >>>>>space
> >>>>>- Copy them to new locations in your HDFS
> >>>>>- Remove the *old* copies of the files
> >>>>>- Rename the new copies to their previous names
> >>>>>
> >>>>>
> >>>>A simpler way, with no interruption of service, is to
> >>>>
> >>>>
> >>merely turn up the
> >>
> >>
> >>>>replication of files, wait for transfers to stabilize,
> >>>>
> >>>>
> >>then turn the
> >>
> >>
> >>>>replication back down.
> >>>>
> >>>>
> >>>Can this be done without having to restart DFS?
> >>>
> >>>
> >>bin/hadoop -dfs setrep 7 -R /directory
> >>
> >>--Konstantin
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> >
>
>
Re: Some queries on Master node
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
It is not getting empty, because a replica to be removed for an
over-replicated block is chosen randomly.
So when the data-node is restarted only a part of its blocks will
removed, not all of them.
In practice I see 10 - 20% reduction of blocks.
I actually used this procedure because some nodes were so full I had
problems starting them.
Yoram Arnon wrote:
>Right, it's good as an emergency measure, but the disadvantage is that the
>full node now becomes empty, so you're not really rebalancing, just cleaning
>up a node. If you only want to remove say 20% of a node's contents, you need
>more subtle measures.
>
>
>
>
>>-----Original Message-----
>>From: Konstantin Shvachko [mailto:shv@yahoo-inc.com]
>>Sent: Thursday, November 02, 2006 11:16 AM
>>To: hadoop-user@lucene.apache.org
>>Subject: Re: Some queries on Master node
>>
>>Yet another way to re-balance blocks is to turn off the node which is
>>full, wait
>>until its blocks are replicated, and then bring it back again. The
>>over-replicated
>>blocks will be randomly removed from different nodes, so you
>>really get them
>>rebalanced not just removed from the current node.
>>
>>Vetle Roeim wrote:
>>
>>
>>
>>>On Thu, 02 Nov 2006 00:55:24 +0100, Bryan A. P. Pendleton
>>><bp...@geekdom.net> wrote:
>>>
>>>
>>>
>>>>>>1. Will Hadoop move the blocks in master node to the
>>>>>>
>>>>>>
>>newly added
>>
>>
>>>>>nodes?
>>>>>
>>>>>No, HDFS will not move blocks to new nodes automatically.
>>>>>
>>>>>
>>New files
>>
>>
>>>>>that
>>>>>are added will likely have their blocks placed on the new
>>>>>
>>>>>
>>nodes. Also,
>>
>>
>>>>>removing old files will remove some blocks that are on your older
>>>>>nodes.
>>>>>
>>>>>One way to re-balance your cluster would be to:
>>>>>- Select a subset of files that take up a good percentage
>>>>>
>>>>>
>>of your disk
>>
>>
>>>>>space
>>>>>- Copy them to new locations in your HDFS
>>>>>- Remove the *old* copies of the files
>>>>>- Rename the new copies to their previous names
>>>>>
>>>>>
>>>>A simpler way, with no interruption of service, is to
>>>>
>>>>
>>merely turn up the
>>
>>
>>>>replication of files, wait for transfers to stabilize,
>>>>
>>>>
>>then turn the
>>
>>
>>>>replication back down.
>>>>
>>>>
>>>Can this be done without having to restart DFS?
>>>
>>>
>>bin/hadoop -dfs setrep 7 -R /directory
>>
>>--Konstantin
>>
>>
>>
>>
>
>
>
>
>
RE: Some queries on Master node
Posted by Yoram Arnon <ya...@yahoo-inc.com>.
Right, it's good as an emergency measure, but the disadvantage is that the
full node now becomes empty, so you're not really rebalancing, just cleaning
up a node. If you only want to remove say 20% of a node's contents, you need
more subtle measures.
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv@yahoo-inc.com]
> Sent: Thursday, November 02, 2006 11:16 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Some queries on Master node
>
> Yet another way to re-balance blocks is to turn off the node which is
> full, wait
> until its blocks are replicated, and then bring it back again. The
> over-replicated
> blocks will be randomly removed from different nodes, so you
> really get them
> rebalanced not just removed from the current node.
>
> Vetle Roeim wrote:
>
> > On Thu, 02 Nov 2006 00:55:24 +0100, Bryan A. P. Pendleton
> > <bp...@geekdom.net> wrote:
> >
> >>>
> >>> > 1. Will Hadoop move the blocks in master node to the
> newly added
> >>> nodes?
> >>>
> >>> No, HDFS will not move blocks to new nodes automatically.
> New files
> >>> that
> >>> are added will likely have their blocks placed on the new
> nodes. Also,
> >>> removing old files will remove some blocks that are on your older
> >>> nodes.
> >>>
> >>> One way to re-balance your cluster would be to:
> >>> - Select a subset of files that take up a good percentage
> of your disk
> >>> space
> >>> - Copy them to new locations in your HDFS
> >>> - Remove the *old* copies of the files
> >>> - Rename the new copies to their previous names
> >>
> >>
> >> A simpler way, with no interruption of service, is to
> merely turn up the
> >> replication of files, wait for transfers to stabilize,
> then turn the
> >> replication back down.
> >
> >
> > Can this be done without having to restart DFS?
>
> bin/hadoop -dfs setrep 7 -R /directory
>
> --Konstantin
>
>
Re: Some queries on Master node
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
Yet another way to re-balance blocks is to turn off the node which is
full, wait
until its blocks are replicated, and then bring it back again. The
over-replicated
blocks will be randomly removed from different nodes, so you really get them
rebalanced not just removed from the current node.
Vetle Roeim wrote:
> On Thu, 02 Nov 2006 00:55:24 +0100, Bryan A. P. Pendleton
> <bp...@geekdom.net> wrote:
>
>>>
>>> > 1. Will Hadoop move the blocks in master node to the newly added
>>> nodes?
>>>
>>> No, HDFS will not move blocks to new nodes automatically. New files
>>> that
>>> are added will likely have their blocks placed on the new nodes. Also,
>>> removing old files will remove some blocks that are on your older
>>> nodes.
>>>
>>> One way to re-balance your cluster would be to:
>>> - Select a subset of files that take up a good percentage of your disk
>>> space
>>> - Copy them to new locations in your HDFS
>>> - Remove the *old* copies of the files
>>> - Rename the new copies to their previous names
>>
>>
>> A simpler way, with no interruption of service, is to merely turn up the
>> replication of files, wait for transfers to stabilize, then turn the
>> replication back down.
>
>
> Can this be done without having to restart DFS?
bin/hadoop -dfs setrep 7 -R /directory
--Konstantin
Re: Some queries on Master node
Posted by Vetle Roeim <ve...@opera.com>.
On Thu, 02 Nov 2006 00:55:24 +0100, Bryan A. P. Pendleton <bp...@geekdom.net>
wrote:
>>
>> > 1. Will Hadoop move the blocks in master node to the newly added
>> nodes?
>>
>> No, HDFS will not move blocks to new nodes automatically. New files that
>> are added will likely have their blocks placed on the new nodes. Also,
>> removing old files will remove some blocks that are on your older nodes.
>>
>> One way to re-balance your cluster would be to:
>> - Select a subset of files that take up a good percentage of your disk
>> space
>> - Copy them to new locations in your HDFS
>> - Remove the *old* copies of the files
>> - Rename the new copies to their previous names
>
>
> A simpler way, with no interruption of service, is to merely turn up the
> replication of files, wait for transfers to stabilize, then turn the
> replication back down.
Can this be done without having to restart DFS?
[...]
--
Vetle Roeim
Team Manager, Information Systems
Opera Software ASA <URL: http://www.opera.com/ >
Re: Some queries on Master node
Posted by "Bryan A. P. Pendleton" <bp...@geekdom.net>.
>
> > 1. Will Hadoop move the blocks in master node to the newly added nodes?
>
> No, HDFS will not move blocks to new nodes automatically. New files that
> are added will likely have their blocks placed on the new nodes. Also,
> removing old files will remove some blocks that are on your older nodes.
>
> One way to re-balance your cluster would be to:
> - Select a subset of files that take up a good percentage of your disk
> space
> - Copy them to new locations in your HDFS
> - Remove the *old* copies of the files
> - Rename the new copies to their previous names
A simpler way, with no interruption of service, is to merely turn up the
replication of files, wait for transfers to stabilize, then turn the
replication back down.
This will tend to distribute data from "full" nodes to "empty" nodes. And,
it can be done to any subset of your files, as long as there is some room to
add extra replications of each file you're up-replicating.
Of course, it'd still be lovely to have some background I/O dedicated to
space re-balancing, but this is a simple-enough way to get things
redistributed.
--
Bryan A. P. Pendleton
Ph: (877) geek-1-bp
Re: Some queries on Master node
Posted by Sameer Paranjpye <sa...@yahoo-inc.com>.
Jagadeesh wrote:
> Hi,
>
>
>
> I have a cluster of more than 25 servers which basically serves the purpose
> for storage. However I was wondering what will happen if the master node
> gets exhausted of disk space. If I add more nodes to the cluster,
>
>
>
> 1. Will Hadoop move the blocks in master node to the newly added nodes?
No, HDFS will not move blocks to new nodes automatically. New files that
are added will likely have their blocks placed on the new nodes. Also,
removing old files will remove some blocks that are on your older nodes.
One way to re-balance your cluster would be to:
- Select a subset of files that take up a good percentage of your disk space
- Copy them to new locations in your HDFS
- Remove the *old* copies of the files
- Rename the new copies to their previous names
>
> 2. Is there any parameter where I can specify not to write file chunks /
> blocks in the master node and rather always use other nodes in the cluster?
>
Do you need to keep data on your master node? The simple way to not do
this is to remove the master node from the 'slaves' file.
>
>
> Since there is always a possibility for single point failure, I would like
> to keep the master node as secure as possible.
>
To keep your filesystem metadata safe, you might consider one of the
following options:
- Set up a cron tab to periodically copy the 'image' and 'edits' files
from under the namenode to a different location. If your master node
loses its disk you can bring up the namenode with the copies.
- AFAIK, the upcoming 0.8 release will include a mechanism by which the
namenode writes the image and edits files to multiple locations. You can
then specify multiply directories in the 'dfs.name.dir' config variable.
One or more of these can be NFS mounted volumes on another node. Again,
if you lose the master node, you can bring up the namenode with one of
the remaining copies of the image and edits.
>
>
> I am planning to go live with this setup by end of this week and I really
> appreciate if you can send me a reply for the above queries.
>
Good luck!
>
>
> Thanks
>
> Jugs
>
>