You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark question <ma...@gmail.com> on 2011/05/25 04:22:42 UTC

I can't see this email ... So to clarify ..

Hi guys,

I'm using an NFS cluster consisting of 30 machines, but only specified 3 of
the nodes to be my hadoop cluster. So my problem is this. Datanode won't
start in one of the nodes because of the following error:

org.apache.hadoop.hdfs.server.
common.Storage: Cannot lock storage /cs/student/mark/tmp/hodhod/dfs/data.
The directory is already locked

I think it's because of the NFS property which allows one node to lock it
then the second node can't lock it. So I had to change the following
configuration:
       dfs.data.dir to be "/tmp/hadoop-user/dfs/data"

But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data where my
hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from above.

Where is this configuration over-written ? I thought my core-site.xml has
the final configuration values.
Thanks,
Mark

Re: I can't see this email ... So to clarify ..

Posted by Luca Pireddu <pi...@crs4.it>.
Is  /cs/student/mark/ on the *shared* NFS volume you mentioned in your 
original post?  In that case all nodes would be trying to use the exact same  
directory.

Luca 

On May 25, 2011 08:22:50 Mark question wrote:
> I do ...
> 
>  $ ls -l /cs/student/mark/tmp/hodhod
> total 4
> drwxr-xr-x 3 mark grad 4096 May 24 21:10 dfs
> 
> and ..
> 
> $ ls -l /tmp/hadoop-mark
> total 4
> drwxr-xr-x 3 mark grad 4096 May 24 21:10 dfs
> 
> $ ls -l /tmp/hadoop-maha/dfs/name/       <<<< only name is created here no
> data
> 
> Thanks agian,
> Mark
> 
> On Tue, May 24, 2011 at 9:26 PM, Mapred Learn <ma...@gmail.com>wrote:
> > Do u Hv right permissions on the new dirs ?
> > Try stopping n starting cluster...
> > 
> > -JJ
> > 
> > On May 24, 2011, at 9:13 PM, Mark question <ma...@gmail.com> wrote:
> > > Well, you're right  ... moving it to hdfs-site.xml had an effect at
> > 
> > least.
> > 
> > > But now I'm in the NameSpace incompatable error:
> > > 
> > > WARN org.apache.hadoop.hdfs.server.common.Util: Path
> > > /tmp/hadoop-mark/dfs/data should be specified as a URI in configuration
> > > files. Please update hdfs configuration.
> > > java.io.IOException: Incompatible namespaceIDs in
> > 
> > /tmp/hadoop-mark/dfs/data
> > 
> > > My configuration for this part in hdfs-site.xml:
> > > <configuration>
> > > <property>
> > > 
> > >    <name>dfs.data.dir</name>
> > >    <value>/tmp/hadoop-mark/dfs/data</value>
> > > 
> > > </property>
> > > <property>
> > > 
> > >    <name>dfs.name.dir</name>
> > >    <value>/tmp/hadoop-mark/dfs/name</value>
> > > 
> > > </property>
> > > <property>
> > > 
> > >    <name>hadoop.tmp.dir</name>
> > >    <value>/cs/student/mark/tmp/hodhod</value>
> > > 
> > > </property>
> > > </configuration>
> > > 
> > > The reason why I want to change hadoop.tmp.dir is because the student
> > 
> > quota
> > 
> > > under /tmp is small so I wanted to mount on /cs/student instead for
> > > hadoop.tmp.dir.
> > > 
> > > Thanks,
> > > Mark
> > > 
> > > On Tue, May 24, 2011 at 7:25 PM, Joey Echeverria <jo...@cloudera.com>
> > 
> > wrote:
> > >> Try moving the the configuration to hdfs-site.xml.
> > >> 
> > >> One word of warning, if you use /tmp to store your HDFS data, you risk
> > >> data loss. On many operating systems, files and directories in /tmp
> > >> are automatically deleted.
> > >> 
> > >> -Joey
> > >> 
> > >> On Tue, May 24, 2011 at 10:22 PM, Mark question <ma...@gmail.com>
> > >> 
> > >> wrote:
> > >>> Hi guys,
> > >>> 
> > >>> I'm using an NFS cluster consisting of 30 machines, but only
> > >>> specified
> > 
> > 3
> > 
> > >> of
> > >> 
> > >>> the nodes to be my hadoop cluster. So my problem is this. Datanode
> > 
> > won't
> > 
> > >>> start in one of the nodes because of the following error:
> > >>> 
> > >>> org.apache.hadoop.hdfs.server.
> > >>> common.Storage: Cannot lock storage
> > 
> > /cs/student/mark/tmp/hodhod/dfs/data.
> > 
> > >>> The directory is already locked
> > >>> 
> > >>> I think it's because of the NFS property which allows one node to
> > >>> lock
> > 
> > it
> > 
> > >>> then the second node can't lock it. So I had to change the following
> > >>> 
> > >>> configuration:
> > >>>      dfs.data.dir to be "/tmp/hadoop-user/dfs/data"
> > >>> 
> > >>> But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data
> > 
> > where
> > 
> > >> my
> > >> 
> > >>> hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from
> > >>> above.
> > >>> 
> > >>> Where is this configuration over-written ? I thought my core-site.xml
> > 
> > has
> > 
> > >>> the final configuration values.
> > >>> Thanks,
> > >>> Mark
> > >> 
> > >> --
> > >> Joseph Echeverria
> > >> Cloudera, Inc.
> > >> 443.305.9434

-- 
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
Pula 09010 (CA), Italy
Tel:  +39 0709250452

Re: I can't see this email ... So to clarify ..

Posted by Mark question <ma...@gmail.com>.
I do ...

 $ ls -l /cs/student/mark/tmp/hodhod
total 4
drwxr-xr-x 3 mark grad 4096 May 24 21:10 dfs

and ..

$ ls -l /tmp/hadoop-mark
total 4
drwxr-xr-x 3 mark grad 4096 May 24 21:10 dfs

$ ls -l /tmp/hadoop-maha/dfs/name/       <<<< only name is created here no
data

Thanks agian,
Mark

On Tue, May 24, 2011 at 9:26 PM, Mapred Learn <ma...@gmail.com>wrote:

> Do u Hv right permissions on the new dirs ?
> Try stopping n starting cluster...
>
> -JJ
>
> On May 24, 2011, at 9:13 PM, Mark question <ma...@gmail.com> wrote:
>
> > Well, you're right  ... moving it to hdfs-site.xml had an effect at
> least.
> > But now I'm in the NameSpace incompatable error:
> >
> > WARN org.apache.hadoop.hdfs.server.common.Util: Path
> > /tmp/hadoop-mark/dfs/data should be specified as a URI in configuration
> > files. Please update hdfs configuration.
> > java.io.IOException: Incompatible namespaceIDs in
> /tmp/hadoop-mark/dfs/data
> >
> > My configuration for this part in hdfs-site.xml:
> > <configuration>
> > <property>
> >    <name>dfs.data.dir</name>
> >    <value>/tmp/hadoop-mark/dfs/data</value>
> > </property>
> > <property>
> >    <name>dfs.name.dir</name>
> >    <value>/tmp/hadoop-mark/dfs/name</value>
> > </property>
> > <property>
> >    <name>hadoop.tmp.dir</name>
> >    <value>/cs/student/mark/tmp/hodhod</value>
> > </property>
> > </configuration>
> >
> > The reason why I want to change hadoop.tmp.dir is because the student
> quota
> > under /tmp is small so I wanted to mount on /cs/student instead for
> > hadoop.tmp.dir.
> >
> > Thanks,
> > Mark
> >
> > On Tue, May 24, 2011 at 7:25 PM, Joey Echeverria <jo...@cloudera.com>
> wrote:
> >
> >> Try moving the the configuration to hdfs-site.xml.
> >>
> >> One word of warning, if you use /tmp to store your HDFS data, you risk
> >> data loss. On many operating systems, files and directories in /tmp
> >> are automatically deleted.
> >>
> >> -Joey
> >>
> >> On Tue, May 24, 2011 at 10:22 PM, Mark question <ma...@gmail.com>
> >> wrote:
> >>> Hi guys,
> >>>
> >>> I'm using an NFS cluster consisting of 30 machines, but only specified
> 3
> >> of
> >>> the nodes to be my hadoop cluster. So my problem is this. Datanode
> won't
> >>> start in one of the nodes because of the following error:
> >>>
> >>> org.apache.hadoop.hdfs.server.
> >>> common.Storage: Cannot lock storage
> /cs/student/mark/tmp/hodhod/dfs/data.
> >>> The directory is already locked
> >>>
> >>> I think it's because of the NFS property which allows one node to lock
> it
> >>> then the second node can't lock it. So I had to change the following
> >>> configuration:
> >>>      dfs.data.dir to be "/tmp/hadoop-user/dfs/data"
> >>>
> >>> But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data
> where
> >> my
> >>> hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from above.
> >>>
> >>> Where is this configuration over-written ? I thought my core-site.xml
> has
> >>> the final configuration values.
> >>> Thanks,
> >>> Mark
> >>>
> >>
> >>
> >>
> >> --
> >> Joseph Echeverria
> >> Cloudera, Inc.
> >> 443.305.9434
> >>
>

Re: I can't see this email ... So to clarify ..

Posted by Mapred Learn <ma...@gmail.com>.
Do u Hv right permissions on the new dirs ?
Try stopping n starting cluster...

-JJ

On May 24, 2011, at 9:13 PM, Mark question <ma...@gmail.com> wrote:

> Well, you're right  ... moving it to hdfs-site.xml had an effect at least.
> But now I'm in the NameSpace incompatable error:
> 
> WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /tmp/hadoop-mark/dfs/data should be specified as a URI in configuration
> files. Please update hdfs configuration.
> java.io.IOException: Incompatible namespaceIDs in /tmp/hadoop-maha/dfs/data
> 
> My configuration for this part in hdfs-site.xml:
> <configuration>
> <property>
>    <name>dfs.data.dir</name>
>    <value>/tmp/hadoop-mark/dfs/data</value>
> </property>
> <property>
>    <name>dfs.name.dir</name>
>    <value>/tmp/hadoop-mark/dfs/name</value>
> </property>
> <property>
>    <name>hadoop.tmp.dir</name>
>    <value>/cs/student/mark/tmp/hodhod</value>
> </property>
> </configuration>
> 
> The reason why I want to change hadoop.tmp.dir is because the student quota
> under /tmp is small so I wanted to mount on /cs/student instead for
> hadoop.tmp.dir.
> 
> Thanks,
> Mark
> 
> On Tue, May 24, 2011 at 7:25 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> 
>> Try moving the the configuration to hdfs-site.xml.
>> 
>> One word of warning, if you use /tmp to store your HDFS data, you risk
>> data loss. On many operating systems, files and directories in /tmp
>> are automatically deleted.
>> 
>> -Joey
>> 
>> On Tue, May 24, 2011 at 10:22 PM, Mark question <ma...@gmail.com>
>> wrote:
>>> Hi guys,
>>> 
>>> I'm using an NFS cluster consisting of 30 machines, but only specified 3
>> of
>>> the nodes to be my hadoop cluster. So my problem is this. Datanode won't
>>> start in one of the nodes because of the following error:
>>> 
>>> org.apache.hadoop.hdfs.server.
>>> common.Storage: Cannot lock storage /cs/student/mark/tmp/hodhod/dfs/data.
>>> The directory is already locked
>>> 
>>> I think it's because of the NFS property which allows one node to lock it
>>> then the second node can't lock it. So I had to change the following
>>> configuration:
>>>      dfs.data.dir to be "/tmp/hadoop-user/dfs/data"
>>> 
>>> But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data where
>> my
>>> hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from above.
>>> 
>>> Where is this configuration over-written ? I thought my core-site.xml has
>>> the final configuration values.
>>> Thanks,
>>> Mark
>>> 
>> 
>> 
>> 
>> --
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
>> 

Re: I can't see this email ... So to clarify ..

Posted by Mark question <ma...@gmail.com>.
Well, you're right  ... moving it to hdfs-site.xml had an effect at least.
But now I'm in the NameSpace incompatable error:

WARN org.apache.hadoop.hdfs.server.common.Util: Path
/tmp/hadoop-mark/dfs/data should be specified as a URI in configuration
files. Please update hdfs configuration.
java.io.IOException: Incompatible namespaceIDs in /tmp/hadoop-maha/dfs/data

My configuration for this part in hdfs-site.xml:
<configuration>
 <property>
    <name>dfs.data.dir</name>
    <value>/tmp/hadoop-mark/dfs/data</value>
 </property>
 <property>
    <name>dfs.name.dir</name>
    <value>/tmp/hadoop-mark/dfs/name</value>
 </property>
 <property>
    <name>hadoop.tmp.dir</name>
    <value>/cs/student/mark/tmp/hodhod</value>
 </property>
</configuration>

The reason why I want to change hadoop.tmp.dir is because the student quota
under /tmp is small so I wanted to mount on /cs/student instead for
hadoop.tmp.dir.

Thanks,
Mark

On Tue, May 24, 2011 at 7:25 PM, Joey Echeverria <jo...@cloudera.com> wrote:

> Try moving the the configuration to hdfs-site.xml.
>
> One word of warning, if you use /tmp to store your HDFS data, you risk
> data loss. On many operating systems, files and directories in /tmp
> are automatically deleted.
>
> -Joey
>
> On Tue, May 24, 2011 at 10:22 PM, Mark question <ma...@gmail.com>
> wrote:
> > Hi guys,
> >
> > I'm using an NFS cluster consisting of 30 machines, but only specified 3
> of
> > the nodes to be my hadoop cluster. So my problem is this. Datanode won't
> > start in one of the nodes because of the following error:
> >
> > org.apache.hadoop.hdfs.server.
> > common.Storage: Cannot lock storage /cs/student/mark/tmp/hodhod/dfs/data.
> > The directory is already locked
> >
> > I think it's because of the NFS property which allows one node to lock it
> > then the second node can't lock it. So I had to change the following
> > configuration:
> >       dfs.data.dir to be "/tmp/hadoop-user/dfs/data"
> >
> > But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data where
> my
> > hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from above.
> >
> > Where is this configuration over-written ? I thought my core-site.xml has
> > the final configuration values.
> > Thanks,
> > Mark
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>

Re: I can't see this email ... So to clarify ..

Posted by Joey Echeverria <jo...@cloudera.com>.
Try moving the the configuration to hdfs-site.xml.

One word of warning, if you use /tmp to store your HDFS data, you risk
data loss. On many operating systems, files and directories in /tmp
are automatically deleted.

-Joey

On Tue, May 24, 2011 at 10:22 PM, Mark question <ma...@gmail.com> wrote:
> Hi guys,
>
> I'm using an NFS cluster consisting of 30 machines, but only specified 3 of
> the nodes to be my hadoop cluster. So my problem is this. Datanode won't
> start in one of the nodes because of the following error:
>
> org.apache.hadoop.hdfs.server.
> common.Storage: Cannot lock storage /cs/student/mark/tmp/hodhod/dfs/data.
> The directory is already locked
>
> I think it's because of the NFS property which allows one node to lock it
> then the second node can't lock it. So I had to change the following
> configuration:
>       dfs.data.dir to be "/tmp/hadoop-user/dfs/data"
>
> But this configuration is overwritten by ${hadoop.tmp.dir}/dfs/data where my
> hadoop.tmp.dir = " /cs/student/mark/tmp" as you might guess from above.
>
> Where is this configuration over-written ? I thought my core-site.xml has
> the final configuration values.
> Thanks,
> Mark
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434