You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jeffrey Wang <jw...@palantir.com> on 2011/04/13 01:55:01 UTC
namenode format error
Hey all,
I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
at sun.nio.ch.FileChannelImpl.lock0(Native Method)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
11/04/12 16:47:32 ERROR namenode.NameNode: java.io.IOException: Input/output error
at sun.nio.ch.FileChannelImpl.lock0(Native Method)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
I'm running Hadoop 0.20.2 and never ran into this problem in the several times I've formatted previously, but now it shows up every time and I can't format or start the NameNode. Any ideas? Thanks.
-Jeffrey
Re: namenode format error
Posted by Raghu Angadi <ra...@apache.org>.
Your NFS mount is not letting NameNode lock a file.
On Wed, Apr 13, 2011 at 12:38 PM, Jeffrey Wang <jw...@palantir.com> wrote:
> Hi,
>
> It's just in my home directory, which is an NFS mount. I moved it off NFS
> and it seems to work fine. Is there some reason it doesn't work with NFS?
>
> -Jeffrey
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Wednesday, April 13, 2011 3:48 AM
> To: common-user@hadoop.apache.org
> Subject: Re: namenode format error
>
> Hello,
>
> On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <jw...@palantir.com> wrote:
> > Hey all,
> >
> > I'm trying to format my NameNode (I've done it successfully in the past),
> but I'm getting a strange error:
> >
> > 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output
> error
> > at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > at
>
> What's your underlying FS of the ${dfs.name.dir} directories? Are they
> NFS mounts?
>
> --
> Harsh J
>
Re: Hive regex SerDe issue?
Posted by hadoopman <ha...@gmail.com>.
I appreciate the feedback. I'll check in the morning. We're running
64bit Ubuntu 10.04 LTS server and I 'believe' it should all be 64 bit
but I'll verify that.
Thanks !!
On 04/13/2011 08:31 PM, Lance Norskog wrote:
> . Usually this is a positive-integer problem somewhere. If you use a
> 32-bit Java this would be a problem.
>
> On Wed, Apr 13, 2011 at 3:16 PM, hadoopman<ha...@gmail.com> wrote:
>
>> Is there an issue with using the regex SerDe with loading into Hive text
>> files above 2 gigs in size? I've been experiencing out of memory errors
>> with a select group of logs when running a hive job. I have been able to
>> load the data if I use split to cut it in half or thirds. No problem.
>>
>> Googling around and figured I'd ask while I'm digging into this.
>>
>> Thanks!
>>
>>
>
>
>
Re: Hive regex SerDe issue?
Posted by Lance Norskog <go...@gmail.com>.
. Usually this is a positive-integer problem somewhere. If you use a
32-bit Java this would be a problem.
On Wed, Apr 13, 2011 at 3:16 PM, hadoopman <ha...@gmail.com> wrote:
> Is there an issue with using the regex SerDe with loading into Hive text
> files above 2 gigs in size? I've been experiencing out of memory errors
> with a select group of logs when running a hive job. I have been able to
> load the data if I use split to cut it in half or thirds. No problem.
>
> Googling around and figured I'd ask while I'm digging into this.
>
> Thanks!
>
--
Lance Norskog
goksron@gmail.com
Hive regex SerDe issue?
Posted by hadoopman <ha...@gmail.com>.
Is there an issue with using the regex SerDe with loading into Hive text
files above 2 gigs in size? I've been experiencing out of memory errors
with a select group of logs when running a hive job. I have been able
to load the data if I use split to cut it in half or thirds. No problem.
Googling around and figured I'd ask while I'm digging into this.
Thanks!
Re: namenode format error
Posted by Allen Wittenauer <aw...@apache.org>.
On Apr 13, 2011, at 12:38 PM, Jeffrey Wang wrote:
> It's just in my home directory, which is an NFS mount. I moved it off NFS and it seems to work fine. Is there some reason it doesn't work with NFS?
Locking on NFS--regardless of application--is a dice roll, especially when client/server are different OSes and platforms. Heck even when they are the same, it might not work. For example, on some older Linux kernels, flock() on NFS only created a lock locally and didn't tell the NFS server, allowing another client to claim a lock! (IIRC, fcntl did work...)
Best bet is to make sure rpc.lockd (or whatever daemon provides NFS locking) is running/working and to use the highest compatible version of NFS. NFSv4, in particular, has file locking built into the protocol rather than as an 'external' service.
RE: namenode format error
Posted by Jeffrey Wang <jw...@palantir.com>.
Hi,
It's just in my home directory, which is an NFS mount. I moved it off NFS and it seems to work fine. Is there some reason it doesn't work with NFS?
-Jeffrey
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, April 13, 2011 3:48 AM
To: common-user@hadoop.apache.org
Subject: Re: namenode format error
Hello,
On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <jw...@palantir.com> wrote:
> Hey all,
>
> I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
>
> 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> at
What's your underlying FS of the ${dfs.name.dir} directories? Are they
NFS mounts?
--
Harsh J
Re: namenode format error
Posted by Harsh J <ha...@cloudera.com>.
Hello,
On Wed, Apr 13, 2011 at 5:25 AM, Jeffrey Wang <jw...@palantir.com> wrote:
> Hey all,
>
> I'm trying to format my NameNode (I've done it successfully in the past), but I'm getting a strange error:
>
> 11/04/12 16:47:32 INFO common.Storage: java.io.IOException: Input/output error
> at sun.nio.ch.FileChannelImpl.lock0(Native Method)
> at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
> at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> at
What's your underlying FS of the ${dfs.name.dir} directories? Are they
NFS mounts?
--
Harsh J