You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Phantom <gh...@gmail.com> on 2007/06/15 07:39:43 UTC
Formatting the namenode
Hi
When I format my namenode it does format the directory specified under
dfs.name.dir. However there is a folder under /tmp called hadoop-alakshman.
What is this for ? Will all blocks be stored under /var/hadoop somewhere.
Here is a copy of hadoop-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/var/hadoop</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop001.sf2p.com:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop001.sf2p.com:50029</value>
</property>
<property>
<name>mapred.job.tracker.info.port</name>
<value>50030</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024M</value>
</property>
<property>
<name>mapred.min.split.size</name>
<value>65536</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
Re: Formatting the namenode
Posted by Michael Bieniosek <mi...@powerset.com>.
Go through hadoop-default.xml and look at all the paths that install into
/tmp/hadoop-${user.name}. You should probably override all of those values
in hadoop-site.xml.
-Michael
On 6/15/07 12:23 PM, "Avinash Lakshman" <al...@facebook.com> wrote:
> Dfs and mapred are the directories created inside the
> /tmp/hadoop-<user-name> directory.
>
> Thanks
> A
>
> -----Original Message-----
> From: Michael Bieniosek [mailto:michael@powerset.com]
> Sent: Friday, June 15, 2007 11:31 AM
> To: hadoop-user@lucene.apache.org; Phantom
> Subject: Re: Formatting the namenode
>
> In hadoop-default.xml you should find:
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp/hadoop-${user.name}</value>
> <description>A base for other temporary directories.</description>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>${hadoop.tmp.dir}/dfs/name</value>
> <description>Determines where on the local filesystem the DFS name
> node
> should store the name table. If this is a comma-delimited list
> of directories then the name table is replicated in all of the
> directories, for redundancy. </description>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>${hadoop.tmp.dir}/dfs/data</value>
> <description>Determines where on the local filesystem an DFS data node
> should store its blocks. If this is a comma-delimited
> list of directories, then data will be stored in all named
> directories, typically on different devices.
> Directories that do not exist are ignored.
> </description>
> </property>
>
> and some other properties that use the hadoop.tmp.dir prefix. What
> subdirectories did hadoop create inside /tmp/hadoop-${user.name}?
>
> -Michael
>
> On 6/14/07 10:39 PM, "Phantom" <gh...@gmail.com> wrote:
>
>> Hi
>>
>> When I format my namenode it does format the directory specified under
>> dfs.name.dir. However there is a folder under /tmp called
> hadoop-alakshman.
>> What is this for ? Will all blocks be stored under /var/hadoop
> somewhere.
>> Here is a copy of hadoop-site.xml
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/var/hadoop</value>
>> </property>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://hadoop001.sf2p.com:9000</value>
>> </property>
>>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>hadoop001.sf2p.com:50029</value>
>> </property>
>>
>> <property>
>> <name>mapred.job.tracker.info.port</name>
>> <value>50030</value>
>> </property>
>>
>> <property>
>> <name>mapred.child.java.opts</name>
>> <value>-Xmx1024M</value>
>> </property>
>>
>> <property>
>> <name>mapred.min.split.size</name>
>> <value>65536</value>
>> </property>
>>
>> <property>
>> <name>dfs.replication</name>
>> <value>3</value>
>> </property>
>>
>> </configuration>
>
RE: Formatting the namenode
Posted by Avinash Lakshman <al...@facebook.com>.
Dfs and mapred are the directories created inside the
/tmp/hadoop-<user-name> directory.
Thanks
A
-----Original Message-----
From: Michael Bieniosek [mailto:michael@powerset.com]
Sent: Friday, June 15, 2007 11:31 AM
To: hadoop-user@lucene.apache.org; Phantom
Subject: Re: Formatting the namenode
In hadoop-default.xml you should find:
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>${hadoop.tmp.dir}/dfs/name</value>
<description>Determines where on the local filesystem the DFS name
node
should store the name table. If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>${hadoop.tmp.dir}/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
and some other properties that use the hadoop.tmp.dir prefix. What
subdirectories did hadoop create inside /tmp/hadoop-${user.name}?
-Michael
On 6/14/07 10:39 PM, "Phantom" <gh...@gmail.com> wrote:
> Hi
>
> When I format my namenode it does format the directory specified under
> dfs.name.dir. However there is a folder under /tmp called
hadoop-alakshman.
> What is this for ? Will all blocks be stored under /var/hadoop
somewhere.
> Here is a copy of hadoop-site.xml
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>dfs.name.dir</name>
> <value>/var/hadoop</value>
> </property>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://hadoop001.sf2p.com:9000</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>hadoop001.sf2p.com:50029</value>
> </property>
>
> <property>
> <name>mapred.job.tracker.info.port</name>
> <value>50030</value>
> </property>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1024M</value>
> </property>
>
> <property>
> <name>mapred.min.split.size</name>
> <value>65536</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>3</value>
> </property>
>
> </configuration>
Re: Formatting the namenode
Posted by Michael Bieniosek <mi...@powerset.com>.
In hadoop-default.xml you should find:
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>${hadoop.tmp.dir}/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node
should store the name table. If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>${hadoop.tmp.dir}/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
and some other properties that use the hadoop.tmp.dir prefix. What
subdirectories did hadoop create inside /tmp/hadoop-${user.name}?
-Michael
On 6/14/07 10:39 PM, "Phantom" <gh...@gmail.com> wrote:
> Hi
>
> When I format my namenode it does format the directory specified under
> dfs.name.dir. However there is a folder under /tmp called hadoop-alakshman.
> What is this for ? Will all blocks be stored under /var/hadoop somewhere.
> Here is a copy of hadoop-site.xml
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>dfs.name.dir</name>
> <value>/var/hadoop</value>
> </property>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://hadoop001.sf2p.com:9000</value>
> </property>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>hadoop001.sf2p.com:50029</value>
> </property>
>
> <property>
> <name>mapred.job.tracker.info.port</name>
> <value>50030</value>
> </property>
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1024M</value>
> </property>
>
> <property>
> <name>mapred.min.split.size</name>
> <value>65536</value>
> </property>
>
> <property>
> <name>dfs.replication</name>
> <value>3</value>
> </property>
>
> </configuration>