You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark <st...@gmail.com> on 2011/03/16 17:34:23 UTC
Question on Master
I know the master node is responsible for namenode and job tracker, but
other than that is there any data stored on that machine? Basically what
I am asking is should there be an generous amount of free space on that
machine?
So for example I have a large drive I want to swap out of my master and
put into another machine which will be a used as a node. Before doing
this, is there anything I should back up from my master? Where is the
namenode data stored?
Thanks
Re: Question on Master
Posted by Harsh J <qw...@gmail.com>.
Yes, ${dfs.name.dir} is a NameNode used prop, while the other's a
DataNode used prop.
On Wed, Mar 16, 2011 at 11:41 PM, Mark <st...@gmail.com> wrote:
> Ok thanks for the clarification.
>
> Just to be sure though..
>
> - The master will have the ${dfs.name.dir} but not ${dfs.data.dir}
> - The nodes will have ${dfs.data.dir} but not ${dfs.name.dir}
>
> Is that correct?
>
> On 3/16/11 10:43 AM, Harsh J wrote:
>>
>> NameNode and JobTracker do not require a lot of storage space by
>> themselves. The NameNode needs some space to store its edits and
>> fsimage, and both require logging space.
>>
>> However, you may make use of multiple disks for NameNode, in order to
>> have a redundant backup copy of the NN image available in case one of
>> the disks crash. Other solutions to this second/third location include
>> storing to an HA-NFS mount, or an externally attached disk mount.
>>
>> NameNode data is stored in the ${dfs.name.dir} set of directories
>> [Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
>> (including permissions) if you're planning to switch disks for your
>> NN.
>>
>> On Wed, Mar 16, 2011 at 10:04 PM, Mark<st...@gmail.com> wrote:
>>>
>>> I know the master node is responsible for namenode and job tracker, but
>>> other than that is there any data stored on that machine? Basically what
>>> I
>>> am asking is should there be an generous amount of free space on that
>>> machine?
>>>
>>> So for example I have a large drive I want to swap out of my master and
>>> put
>>> into another machine which will be a used as a node. Before doing this,
>>> is
>>> there anything I should back up from my master? Where is the namenode
>>> data
>>> stored?
>>>
>>> Thanks
>>>
>>
>>
>
--
Harsh J
http://harshj.com
Re: Question on Master
Posted by Mark <st...@gmail.com>.
Ok thanks for the clarification.
Just to be sure though..
- The master will have the ${dfs.name.dir} but not ${dfs.data.dir}
- The nodes will have ${dfs.data.dir} but not ${dfs.name.dir}
Is that correct?
On 3/16/11 10:43 AM, Harsh J wrote:
> NameNode and JobTracker do not require a lot of storage space by
> themselves. The NameNode needs some space to store its edits and
> fsimage, and both require logging space.
>
> However, you may make use of multiple disks for NameNode, in order to
> have a redundant backup copy of the NN image available in case one of
> the disks crash. Other solutions to this second/third location include
> storing to an HA-NFS mount, or an externally attached disk mount.
>
> NameNode data is stored in the ${dfs.name.dir} set of directories
> [Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
> (including permissions) if you're planning to switch disks for your
> NN.
>
> On Wed, Mar 16, 2011 at 10:04 PM, Mark<st...@gmail.com> wrote:
>> I know the master node is responsible for namenode and job tracker, but
>> other than that is there any data stored on that machine? Basically what I
>> am asking is should there be an generous amount of free space on that
>> machine?
>>
>> So for example I have a large drive I want to swap out of my master and put
>> into another machine which will be a used as a node. Before doing this, is
>> there anything I should back up from my master? Where is the namenode data
>> stored?
>>
>> Thanks
>>
>
>
Re: Question on Master
Posted by Harsh J <qw...@gmail.com>.
NameNode and JobTracker do not require a lot of storage space by
themselves. The NameNode needs some space to store its edits and
fsimage, and both require logging space.
However, you may make use of multiple disks for NameNode, in order to
have a redundant backup copy of the NN image available in case one of
the disks crash. Other solutions to this second/third location include
storing to an HA-NFS mount, or an externally attached disk mount.
NameNode data is stored in the ${dfs.name.dir} set of directories
[Defined in hdfs-site.xml]. Ensure its content is preserved perfectly
(including permissions) if you're planning to switch disks for your
NN.
On Wed, Mar 16, 2011 at 10:04 PM, Mark <st...@gmail.com> wrote:
> I know the master node is responsible for namenode and job tracker, but
> other than that is there any data stored on that machine? Basically what I
> am asking is should there be an generous amount of free space on that
> machine?
>
> So for example I have a large drive I want to swap out of my master and put
> into another machine which will be a used as a node. Before doing this, is
> there anything I should back up from my master? Where is the namenode data
> stored?
>
> Thanks
>
--
Harsh J
http://harshj.com