You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by cliff palmer <pa...@gmail.com> on 2010/08/23 15:15:35 UTC
Hadoop startup problem - directory name required
The namenode log for a Hadoop-0.20 installation contains this error message:
"/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
This directory does not exist and I would like to understand why this
particular directory name is required (not what the directory is used for,
but why this particular directory name). The *-site.xml files are below (IP
addresses have been masked).
Thanks in advance for your help.
Cliff
core-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://xxx.xxx.xxx.xxx:8020</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop-0.20/cache/hadoop</value>
</property>
</configuration>
-------- end of core-site.xml -----------
hdfs-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
<property>
</property>
<name>dfs.data.dir</name>
<value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>20000000000</value>
</property>
<property>
<name>dfs.namenode,plugins</name>
<value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
</property>
<property>
<name>dfs.thrift.address</name>
<value>0.0.0.0:9090</value>
</property>
</configuration>
-------- end of core-site.xml -----------
mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>xxx.xxx.xxx.xxx:8020</value>
</property>
<property>
<name>jobtracker.thrift.address</name>
<value>0.0.0:9290</value>
</property>
<property>
<name>mapred.jobtracker.plugins</name>
<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
</property>
</configuration>
-------- end of core-site.xml -----------
Re: Hadoop startup problem - directory name required
Posted by cliff palmer <pa...@gmail.com>.
No complaints from hadoop, other than the error for the mangled directory
name.
On Wed, Aug 25, 2010 at 2:04 PM, Hemanth Yamijala <yh...@gmail.com>wrote:
> Hmm. Without the / in the property tag, isn't the file malformed XML ?
> I am pretty sure Hadoop complains in such cases ?
>
> On Wed, Aug 25, 2010 at 4:44 AM, cliff palmer <pa...@gmail.com>
> wrote:
> > Thanks Allen - that has resolved the problem. Good catch!
> > Cliff
> >
> > On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
> > <aw...@linkedin.com>wrote:
> >
> >>
> >> On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:
> >>
> >> > Thanks Harsh, but I am still not sure I understand what is going on.
> >> > The directory specified in the dfs.name.dir property,
> >> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that
> directory
> >> have
> >> > been granted to the OS user that is running the Hadoop startup script.
> >> > The directory mentioned in the error message is
> >> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> >> > I can create this directory and that would (I assume) remove the
> error,
> >> but
> >> > I want to understand how the name is derived.
> >>
> >> From here:
> >>
> >> > <property>
> >> > <name>hadoop.tmp.dir</name>
> >> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> >> > </property>
> >> > </configuration>
> >>
> >> because:
> >>
> >> >
> >> > <property>
> >> > <name>dfs.name.dir</name>
> >> >
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> >> > <property>
> >>
> >> is missing a / in the property line.
> >>
> >>
> >>
> >>
> >>
> >
>
Re: Hadoop startup problem - directory name required
Posted by Hemanth Yamijala <yh...@gmail.com>.
Hmm. Without the / in the property tag, isn't the file malformed XML ?
I am pretty sure Hadoop complains in such cases ?
On Wed, Aug 25, 2010 at 4:44 AM, cliff palmer <pa...@gmail.com> wrote:
> Thanks Allen - that has resolved the problem. Good catch!
> Cliff
>
> On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
> <aw...@linkedin.com>wrote:
>
>>
>> On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:
>>
>> > Thanks Harsh, but I am still not sure I understand what is going on.
>> > The directory specified in the dfs.name.dir property,
>> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory
>> have
>> > been granted to the OS user that is running the Hadoop startup script.
>> > The directory mentioned in the error message is
>> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
>> > I can create this directory and that would (I assume) remove the error,
>> but
>> > I want to understand how the name is derived.
>>
>> From here:
>>
>> > <property>
>> > <name>hadoop.tmp.dir</name>
>> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
>> > </property>
>> > </configuration>
>>
>> because:
>>
>> >
>> > <property>
>> > <name>dfs.name.dir</name>
>> > <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> > <property>
>>
>> is missing a / in the property line.
>>
>>
>>
>>
>>
>
Re: Hadoop startup problem - directory name required
Posted by cliff palmer <pa...@gmail.com>.
Thanks Allen - that has resolved the problem. Good catch!
Cliff
On Tue, Aug 24, 2010 at 3:05 PM, Allen Wittenauer
<aw...@linkedin.com>wrote:
>
> On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:
>
> > Thanks Harsh, but I am still not sure I understand what is going on.
> > The directory specified in the dfs.name.dir property,
> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory
> have
> > been granted to the OS user that is running the Hadoop startup script.
> > The directory mentioned in the error message is
> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> > I can create this directory and that would (I assume) remove the error,
> but
> > I want to understand how the name is derived.
>
> From here:
>
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> > </property>
> > </configuration>
>
> because:
>
> >
> > <property>
> > <name>dfs.name.dir</name>
> > <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> > <property>
>
> is missing a / in the property line.
>
>
>
>
>
Re: Hadoop startup problem - directory name required
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Aug 23, 2010, at 6:49 AM, cliff palmer wrote:
> Thanks Harsh, but I am still not sure I understand what is going on.
> The directory specified in the dfs.name.dir property,
> "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
> been granted to the OS user that is running the Hadoop startup script.
> The directory mentioned in the error message is
> "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> I can create this directory and that would (I assume) remove the error, but
> I want to understand how the name is derived.
>From here:
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> </property>
> </configuration>
because:
>
> <property>
> <name>dfs.name.dir</name>
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> <property>
is missing a / in the property line.
Re: Hadoop startup problem - directory name required
Posted by cliff palmer <pa...@gmail.com>.
Yes, the hadoop user has execute permissions on the parent directories.
Thanks!
Cliff
On Mon, Aug 23, 2010 at 12:12 PM, Edward Capriolo <ed...@gmail.com>wrote:
> On Mon, Aug 23, 2010 at 12:09 PM, cliff palmer <pa...@gmail.com>
> wrote:
> > The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory. I've
> > confirmed that these are the files being used.
> > Thanks again.
> > Cliff
> >
> > On Mon, Aug 23, 2010 at 10:26 AM, Harsh J <qw...@gmail.com>
> wrote:
> >
> >> Can you confirm that this is the right configuration your NN is starting
> >> with?
> >>
> >> On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <pa...@gmail.com>
> >> wrote:
> >> > Thanks Harsh, but I am still not sure I understand what is going on.
> >> > The directory specified in the dfs.name.dir property,
> >> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that
> directory
> >> have
> >> > been granted to the OS user that is running the Hadoop startup script.
> >> > The directory mentioned in the error message is
> >> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> >> > I can create this directory and that would (I assume) remove the
> error,
> >> but
> >> > I want to understand how the name is derived. It's not a child of the
> >> > directory name specified in the dfs.name.dir property.
> >> > Thanks again!
> >> > Cliff
> >> >
> >> >
> >> > On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qw...@gmail.com>
> wrote:
> >> >
> >> >> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
> >> >> has it in its list of dirs to write a copy to:
> >> >>
> >> >> <property>
> >> >> <name>dfs.name.dir</name>
> >> >>
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> >> >> <property>
> >> >>
> >> >> Remove it from this property if you don't need it. If you need
> >> >> multiple dirs, you should create the path and allocate proper
> >> >> permissions to it so that it may write to it. Its good to use a
> second
> >> >> path for backup purposes (most have this on the NFS). The path is so
> >> >> because your distribution of hadoop perhaps placed its files around
> >> >> that place and its up in the conf files as some form of default :)
> >> >>
> >> >> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <palmercliff@gmail.com
> >
> >> >> wrote:
> >> >> > The namenode log for a Hadoop-0.20 installation contains this error
> >> >> message:
> >> >> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
> >> state".
> >> >> > This directory does not exist and I would like to understand why
> this
> >> >> > particular directory name is required (not what the directory is
> used
> >> >> for,
> >> >> > but why this particular directory name). The *-site.xml files are
> >> below
> >> >> (IP
> >> >> > addresses have been masked).
> >> >> > Thanks in advance for your help.
> >> >> > Cliff
> >> >> >
> >> >> > core-site.xml:
> >> >> >
> >> >> > <?xml version="1.0"?>
> >> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >> >
> >> >> > <!-- Put site-specific property overrides in this file. -->
> >> >> >
> >> >> > <configuration>
> >> >> > <property>
> >> >> > <name>fs.default.name</name>
> >> >> > <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
> >> >> > <final>true</final>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>hadoop.tmp.dir</name>
> >> >> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> >> >> > </property>
> >> >> > </configuration>
> >> >> > -------- end of core-site.xml -----------
> >> >> >
> >> >> > hdfs-site.xml:
> >> >> >
> >> >> > <?xml version="1.0"?>
> >> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >> >
> >> >> > <!-- Put site-specific property overrides in this file. -->
> >> >> >
> >> >> > <configuration>
> >> >> > <property>
> >> >> > <name>dfs.replication</name>
> >> >> > <value>3</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.name.dir</name>
> >> >> >
> >> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> >> >> > <property>
> >> >> > </property>
> >> >> > <name>dfs.data.dir</name>
> >> >> >
> >> <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.permissions</name>
> >> >> > <value>false</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.datanode.du.reserved</name>
> >> >> > <value>20000000000</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.namenode,plugins</name>
> >> >> >
> >> <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.datanode.plugins</name>
> >> >> >
> >> <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>dfs.thrift.address</name>
> >> >> > <value>0.0.0.0:9090</value>
> >> >> > </property>
> >> >> > </configuration>
> >> >> > -------- end of core-site.xml -----------
> >> >> >
> >> >> > mapred-site.xml:
> >> >> >
> >> >> > <?xml version="1.0"?>
> >> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >> >
> >> >> > <!-- Put site-specific property overrides in this file. -->
> >> >> >
> >> >> > <configuration>
> >> >> > <property>
> >> >> > <name>mapred.job.tracker</name>
> >> >> > <value>xxx.xxx.xxx.xxx:8020</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>jobtracker.thrift.address</name>
> >> >> > <value>0.0.0:9290</value>
> >> >> > </property>
> >> >> > <property>
> >> >> > <name>mapred.jobtracker.plugins</name>
> >> >> >
> >> >> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
> >> >> > </property>
> >> >> > </configuration>
> >> >> > -------- end of core-site.xml -----------
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >> www.harshj.com
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
>
> Check the permissions of the paths above this one. Make sure parent
> directories have execute(search) permission for the user you run
> hadoop as.
>
Re: Hadoop startup problem - directory name required
Posted by Edward Capriolo <ed...@gmail.com>.
On Mon, Aug 23, 2010 at 12:09 PM, cliff palmer <pa...@gmail.com> wrote:
> The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory. I've
> confirmed that these are the files being used.
> Thanks again.
> Cliff
>
> On Mon, Aug 23, 2010 at 10:26 AM, Harsh J <qw...@gmail.com> wrote:
>
>> Can you confirm that this is the right configuration your NN is starting
>> with?
>>
>> On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <pa...@gmail.com>
>> wrote:
>> > Thanks Harsh, but I am still not sure I understand what is going on.
>> > The directory specified in the dfs.name.dir property,
>> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory
>> have
>> > been granted to the OS user that is running the Hadoop startup script.
>> > The directory mentioned in the error message is
>> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
>> > I can create this directory and that would (I assume) remove the error,
>> but
>> > I want to understand how the name is derived. It's not a child of the
>> > directory name specified in the dfs.name.dir property.
>> > Thanks again!
>> > Cliff
>> >
>> >
>> > On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qw...@gmail.com> wrote:
>> >
>> >> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
>> >> has it in its list of dirs to write a copy to:
>> >>
>> >> <property>
>> >> <name>dfs.name.dir</name>
>> >> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> >> <property>
>> >>
>> >> Remove it from this property if you don't need it. If you need
>> >> multiple dirs, you should create the path and allocate proper
>> >> permissions to it so that it may write to it. Its good to use a second
>> >> path for backup purposes (most have this on the NFS). The path is so
>> >> because your distribution of hadoop perhaps placed its files around
>> >> that place and its up in the conf files as some form of default :)
>> >>
>> >> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <pa...@gmail.com>
>> >> wrote:
>> >> > The namenode log for a Hadoop-0.20 installation contains this error
>> >> message:
>> >> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
>> state".
>> >> > This directory does not exist and I would like to understand why this
>> >> > particular directory name is required (not what the directory is used
>> >> for,
>> >> > but why this particular directory name). The *-site.xml files are
>> below
>> >> (IP
>> >> > addresses have been masked).
>> >> > Thanks in advance for your help.
>> >> > Cliff
>> >> >
>> >> > core-site.xml:
>> >> >
>> >> > <?xml version="1.0"?>
>> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >> >
>> >> > <!-- Put site-specific property overrides in this file. -->
>> >> >
>> >> > <configuration>
>> >> > <property>
>> >> > <name>fs.default.name</name>
>> >> > <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
>> >> > <final>true</final>
>> >> > </property>
>> >> > <property>
>> >> > <name>hadoop.tmp.dir</name>
>> >> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
>> >> > </property>
>> >> > </configuration>
>> >> > -------- end of core-site.xml -----------
>> >> >
>> >> > hdfs-site.xml:
>> >> >
>> >> > <?xml version="1.0"?>
>> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >> >
>> >> > <!-- Put site-specific property overrides in this file. -->
>> >> >
>> >> > <configuration>
>> >> > <property>
>> >> > <name>dfs.replication</name>
>> >> > <value>3</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.name.dir</name>
>> >> >
>> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> >> > <property>
>> >> > </property>
>> >> > <name>dfs.data.dir</name>
>> >> >
>> <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.permissions</name>
>> >> > <value>false</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.datanode.du.reserved</name>
>> >> > <value>20000000000</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.namenode,plugins</name>
>> >> >
>> <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.datanode.plugins</name>
>> >> >
>> <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>dfs.thrift.address</name>
>> >> > <value>0.0.0.0:9090</value>
>> >> > </property>
>> >> > </configuration>
>> >> > -------- end of core-site.xml -----------
>> >> >
>> >> > mapred-site.xml:
>> >> >
>> >> > <?xml version="1.0"?>
>> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >> >
>> >> > <!-- Put site-specific property overrides in this file. -->
>> >> >
>> >> > <configuration>
>> >> > <property>
>> >> > <name>mapred.job.tracker</name>
>> >> > <value>xxx.xxx.xxx.xxx:8020</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>jobtracker.thrift.address</name>
>> >> > <value>0.0.0:9290</value>
>> >> > </property>
>> >> > <property>
>> >> > <name>mapred.jobtracker.plugins</name>
>> >> >
>> >> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
>> >> > </property>
>> >> > </configuration>
>> >> > -------- end of core-site.xml -----------
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >> www.harshj.com
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
Check the permissions of the paths above this one. Make sure parent
directories have execute(search) permission for the user you run
hadoop as.
Re: Hadoop startup problem - directory name required
Posted by cliff palmer <pa...@gmail.com>.
The 3 *-site.xml files are in the /etc/hadoop-0.20/conf directory. I've
confirmed that these are the files being used.
Thanks again.
Cliff
On Mon, Aug 23, 2010 at 10:26 AM, Harsh J <qw...@gmail.com> wrote:
> Can you confirm that this is the right configuration your NN is starting
> with?
>
> On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <pa...@gmail.com>
> wrote:
> > Thanks Harsh, but I am still not sure I understand what is going on.
> > The directory specified in the dfs.name.dir property,
> > "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory
> have
> > been granted to the OS user that is running the Hadoop startup script.
> > The directory mentioned in the error message is
> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> > I can create this directory and that would (I assume) remove the error,
> but
> > I want to understand how the name is derived. It's not a child of the
> > directory name specified in the dfs.name.dir property.
> > Thanks again!
> > Cliff
> >
> >
> > On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qw...@gmail.com> wrote:
> >
> >> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
> >> has it in its list of dirs to write a copy to:
> >>
> >> <property>
> >> <name>dfs.name.dir</name>
> >> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> >> <property>
> >>
> >> Remove it from this property if you don't need it. If you need
> >> multiple dirs, you should create the path and allocate proper
> >> permissions to it so that it may write to it. Its good to use a second
> >> path for backup purposes (most have this on the NFS). The path is so
> >> because your distribution of hadoop perhaps placed its files around
> >> that place and its up in the conf files as some form of default :)
> >>
> >> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <pa...@gmail.com>
> >> wrote:
> >> > The namenode log for a Hadoop-0.20 installation contains this error
> >> message:
> >> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent
> state".
> >> > This directory does not exist and I would like to understand why this
> >> > particular directory name is required (not what the directory is used
> >> for,
> >> > but why this particular directory name). The *-site.xml files are
> below
> >> (IP
> >> > addresses have been masked).
> >> > Thanks in advance for your help.
> >> > Cliff
> >> >
> >> > core-site.xml:
> >> >
> >> > <?xml version="1.0"?>
> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >
> >> > <!-- Put site-specific property overrides in this file. -->
> >> >
> >> > <configuration>
> >> > <property>
> >> > <name>fs.default.name</name>
> >> > <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
> >> > <final>true</final>
> >> > </property>
> >> > <property>
> >> > <name>hadoop.tmp.dir</name>
> >> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> >> > </property>
> >> > </configuration>
> >> > -------- end of core-site.xml -----------
> >> >
> >> > hdfs-site.xml:
> >> >
> >> > <?xml version="1.0"?>
> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >
> >> > <!-- Put site-specific property overrides in this file. -->
> >> >
> >> > <configuration>
> >> > <property>
> >> > <name>dfs.replication</name>
> >> > <value>3</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.name.dir</name>
> >> >
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> >> > <property>
> >> > </property>
> >> > <name>dfs.data.dir</name>
> >> >
> <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.permissions</name>
> >> > <value>false</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.datanode.du.reserved</name>
> >> > <value>20000000000</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.namenode,plugins</name>
> >> >
> <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.datanode.plugins</name>
> >> >
> <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
> >> > </property>
> >> > <property>
> >> > <name>dfs.thrift.address</name>
> >> > <value>0.0.0.0:9090</value>
> >> > </property>
> >> > </configuration>
> >> > -------- end of core-site.xml -----------
> >> >
> >> > mapred-site.xml:
> >> >
> >> > <?xml version="1.0"?>
> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >> >
> >> > <!-- Put site-specific property overrides in this file. -->
> >> >
> >> > <configuration>
> >> > <property>
> >> > <name>mapred.job.tracker</name>
> >> > <value>xxx.xxx.xxx.xxx:8020</value>
> >> > </property>
> >> > <property>
> >> > <name>jobtracker.thrift.address</name>
> >> > <value>0.0.0:9290</value>
> >> > </property>
> >> > <property>
> >> > <name>mapred.jobtracker.plugins</name>
> >> >
> >> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
> >> > </property>
> >> > </configuration>
> >> > -------- end of core-site.xml -----------
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>
Re: Hadoop startup problem - directory name required
Posted by Harsh J <qw...@gmail.com>.
Can you confirm that this is the right configuration your NN is starting with?
On Mon, Aug 23, 2010 at 7:19 PM, cliff palmer <pa...@gmail.com> wrote:
> Thanks Harsh, but I am still not sure I understand what is going on.
> The directory specified in the dfs.name.dir property,
> "/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
> been granted to the OS user that is running the Hadoop startup script.
> The directory mentioned in the error message is
> "/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
> I can create this directory and that would (I assume) remove the error, but
> I want to understand how the name is derived. It's not a child of the
> directory name specified in the dfs.name.dir property.
> Thanks again!
> Cliff
>
>
> On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qw...@gmail.com> wrote:
>
>> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
>> has it in its list of dirs to write a copy to:
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> <property>
>>
>> Remove it from this property if you don't need it. If you need
>> multiple dirs, you should create the path and allocate proper
>> permissions to it so that it may write to it. Its good to use a second
>> path for backup purposes (most have this on the NFS). The path is so
>> because your distribution of hadoop perhaps placed its files around
>> that place and its up in the conf files as some form of default :)
>>
>> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <pa...@gmail.com>
>> wrote:
>> > The namenode log for a Hadoop-0.20 installation contains this error
>> message:
>> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
>> > This directory does not exist and I would like to understand why this
>> > particular directory name is required (not what the directory is used
>> for,
>> > but why this particular directory name). The *-site.xml files are below
>> (IP
>> > addresses have been masked).
>> > Thanks in advance for your help.
>> > Cliff
>> >
>> > core-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> > <property>
>> > <name>fs.default.name</name>
>> > <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
>> > <final>true</final>
>> > </property>
>> > <property>
>> > <name>hadoop.tmp.dir</name>
>> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
>> > </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>> > hdfs-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> > <property>
>> > <name>dfs.replication</name>
>> > <value>3</value>
>> > </property>
>> > <property>
>> > <name>dfs.name.dir</name>
>> > <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
>> > <property>
>> > </property>
>> > <name>dfs.data.dir</name>
>> > <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
>> > </property>
>> > <property>
>> > <name>dfs.permissions</name>
>> > <value>false</value>
>> > </property>
>> > <property>
>> > <name>dfs.datanode.du.reserved</name>
>> > <value>20000000000</value>
>> > </property>
>> > <property>
>> > <name>dfs.namenode,plugins</name>
>> > <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
>> > </property>
>> > <property>
>> > <name>dfs.datanode.plugins</name>
>> > <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
>> > </property>
>> > <property>
>> > <name>dfs.thrift.address</name>
>> > <value>0.0.0.0:9090</value>
>> > </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>> > mapred-site.xml:
>> >
>> > <?xml version="1.0"?>
>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >
>> > <!-- Put site-specific property overrides in this file. -->
>> >
>> > <configuration>
>> > <property>
>> > <name>mapred.job.tracker</name>
>> > <value>xxx.xxx.xxx.xxx:8020</value>
>> > </property>
>> > <property>
>> > <name>jobtracker.thrift.address</name>
>> > <value>0.0.0:9290</value>
>> > </property>
>> > <property>
>> > <name>mapred.jobtracker.plugins</name>
>> >
>> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
>> > </property>
>> > </configuration>
>> > -------- end of core-site.xml -----------
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
--
Harsh J
www.harshj.com
Re: Hadoop startup problem - directory name required
Posted by cliff palmer <pa...@gmail.com>.
Thanks Harsh, but I am still not sure I understand what is going on.
The directory specified in the dfs.name.dir property,
"/var/lib/hadoop-0.20/dfsname", does exist and rights to that directory have
been granted to the OS user that is running the Hadoop startup script.
The directory mentioned in the error message is
"/var/lib/hadoop-0.20/cache/hadoop/dfs/name".
I can create this directory and that would (I assume) remove the error, but
I want to understand how the name is derived. It's not a child of the
directory name specified in the dfs.name.dir property.
Thanks again!
Cliff
On Mon, Aug 23, 2010 at 9:21 AM, Harsh J <qw...@gmail.com> wrote:
> Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
> has it in its list of dirs to write a copy to:
>
> <property>
> <name>dfs.name.dir</name>
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> <property>
>
> Remove it from this property if you don't need it. If you need
> multiple dirs, you should create the path and allocate proper
> permissions to it so that it may write to it. Its good to use a second
> path for backup purposes (most have this on the NFS). The path is so
> because your distribution of hadoop perhaps placed its files around
> that place and its up in the conf files as some form of default :)
>
> On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <pa...@gmail.com>
> wrote:
> > The namenode log for a Hadoop-0.20 installation contains this error
> message:
> > "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
> > This directory does not exist and I would like to understand why this
> > particular directory name is required (not what the directory is used
> for,
> > but why this particular directory name). The *-site.xml files are below
> (IP
> > addresses have been masked).
> > Thanks in advance for your help.
> > Cliff
> >
> > core-site.xml:
> >
> > <?xml version="1.0"?>
> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >
> > <!-- Put site-specific property overrides in this file. -->
> >
> > <configuration>
> > <property>
> > <name>fs.default.name</name>
> > <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
> > <final>true</final>
> > </property>
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> > </property>
> > </configuration>
> > -------- end of core-site.xml -----------
> >
> > hdfs-site.xml:
> >
> > <?xml version="1.0"?>
> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >
> > <!-- Put site-specific property overrides in this file. -->
> >
> > <configuration>
> > <property>
> > <name>dfs.replication</name>
> > <value>3</value>
> > </property>
> > <property>
> > <name>dfs.name.dir</name>
> > <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> > <property>
> > </property>
> > <name>dfs.data.dir</name>
> > <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
> > </property>
> > <property>
> > <name>dfs.permissions</name>
> > <value>false</value>
> > </property>
> > <property>
> > <name>dfs.datanode.du.reserved</name>
> > <value>20000000000</value>
> > </property>
> > <property>
> > <name>dfs.namenode,plugins</name>
> > <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
> > </property>
> > <property>
> > <name>dfs.datanode.plugins</name>
> > <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
> > </property>
> > <property>
> > <name>dfs.thrift.address</name>
> > <value>0.0.0.0:9090</value>
> > </property>
> > </configuration>
> > -------- end of core-site.xml -----------
> >
> > mapred-site.xml:
> >
> > <?xml version="1.0"?>
> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >
> > <!-- Put site-specific property overrides in this file. -->
> >
> > <configuration>
> > <property>
> > <name>mapred.job.tracker</name>
> > <value>xxx.xxx.xxx.xxx:8020</value>
> > </property>
> > <property>
> > <name>jobtracker.thrift.address</name>
> > <value>0.0.0:9290</value>
> > </property>
> > <property>
> > <name>mapred.jobtracker.plugins</name>
> >
> > <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
> > </property>
> > </configuration>
> > -------- end of core-site.xml -----------
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>
Re: Hadoop startup problem - directory name required
Posted by Harsh J <qw...@gmail.com>.
Its checking this directory cause your dfs.name.dir (hdfs-site.xml)
has it in its list of dirs to write a copy to:
<property>
<name>dfs.name.dir</name>
<value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
<property>
Remove it from this property if you don't need it. If you need
multiple dirs, you should create the path and allocate proper
permissions to it so that it may write to it. Its good to use a second
path for backup purposes (most have this on the NFS). The path is so
because your distribution of hadoop perhaps placed its files around
that place and its up in the conf files as some form of default :)
On Mon, Aug 23, 2010 at 6:45 PM, cliff palmer <pa...@gmail.com> wrote:
> The namenode log for a Hadoop-0.20 installation contains this error message:
> "/var/lib/hadoop-0.20/cache/hadoop/dfs/name in in an inconsistent state".
> This directory does not exist and I would like to understand why this
> particular directory name is required (not what the directory is used for,
> but why this particular directory name). The *-site.xml files are below (IP
> addresses have been masked).
> Thanks in advance for your help.
> Cliff
>
> core-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://xxx.xxx.xxx.xxx:8020</value>
> <final>true</final>
> </property>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/var/lib/hadoop-0.20/cache/hadoop</value>
> </property>
> </configuration>
> -------- end of core-site.xml -----------
>
> hdfs-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>3</value>
> </property>
> <property>
> <name>dfs.name.dir</name>
> <value>/DFS/dfsname,/var/lib/hadoop-0.20/dfsname</value>
> <property>
> </property>
> <name>dfs.data.dir</name>
> <value>/DFS1/dfsdata,/DFS2/dfsdata,/DFS3/dfsdata</value>
> </property>
> <property>
> <name>dfs.permissions</name>
> <value>false</value>
> </property>
> <property>
> <name>dfs.datanode.du.reserved</name>
> <value>20000000000</value>
> </property>
> <property>
> <name>dfs.namenode,plugins</name>
> <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
> </property>
> <property>
> <name>dfs.datanode.plugins</name>
> <value>org.apache.hadoop.thriftfs.DatanodePlugins</value>
> </property>
> <property>
> <name>dfs.thrift.address</name>
> <value>0.0.0.0:9090</value>
> </property>
> </configuration>
> -------- end of core-site.xml -----------
>
> mapred-site.xml:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>xxx.xxx.xxx.xxx:8020</value>
> </property>
> <property>
> <name>jobtracker.thrift.address</name>
> <value>0.0.0:9290</value>
> </property>
> <property>
> <name>mapred.jobtracker.plugins</name>
>
> <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
> </property>
> </configuration>
> -------- end of core-site.xml -----------
>
--
Harsh J
www.harshj.com