You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Matt Tanquary <ma...@gmail.com> on 2010/10/27 19:15:30 UTC

Bounced cluster - won't restart

My small cluster was working fine. I bounced it just because I've been
running a lot of MR jobs and wanted to take down a couple nodes, balance,
etc. But when I attempt to start it, I get this in the namenode log. Any
ideas what to look out for?


2010-10-27 10:11:11,560 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-10-27 10:11:11,887 INFO org.apache.hadoop.hdfs.server.common.Storage:
java.io.IOException: Input/output error
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:285)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:293)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:306)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1006)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1015)

2010-10-27 10:11:11,891 ERROR
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
java.io.IOException: Input/output error
        at sun.nio.ch.FileChannelImpl.lock0(Native Method)
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:285)
        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:293)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:306)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1006)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1015)


-- 
Have you thanked a teacher today? ---> http://www.liftateacher.org

Re: Bounced cluster - won't restart

Posted by Matt Tanquary <ma...@gmail.com>.

I had 2 locations set up in dfs.name.dir. I removed one of them and the
namenode started up.

On Wed, Oct 27, 2010 at 10:15 AM, Matt Tanquary <ma...@gmail.com>wrote:

> My small cluster was working fine. I bounced it just because I've been
> running a lot of MR jobs and wanted to take down a couple nodes, balance,
> etc. But when I attempt to start it, I get this in the namenode log. Any
> ideas what to look out for?
>
>
> 2010-10-27 10:11:11,560 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2010-10-27 10:11:11,887 INFO org.apache.hadoop.hdfs.server.common.Storage:
> java.io.IOException: Input/output error
>         at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>         at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>         at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:285)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:293)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:306)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1006)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1015)
>
> 2010-10-27 10:11:11,891 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> java.io.IOException: Input/output error
>         at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>         at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:881)
>         at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>         at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:285)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:293)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:306)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1006)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1015)
>
>
> --
> Have you thanked a teacher today? ---> http://www.liftateacher.org
>



-- 
Have you thanked a teacher today? ---> http://www.liftateacher.org