You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by iain wright <ia...@gmail.com> on 2013/12/07 02:38:55 UTC
Namenode won't start after unclean shutdown of RS
Hi folks,
One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted
about reports failing, and i saw wierd logs in the jobtracker for "pending
shutdown" etc.
Proceeded to stop jobtracker/tt nodes, hbase, and hdfs.
On attempting to turn back on the NN + DN's our namenode is failing to
start, complaining about being unable to move one of our hbase region files
because source file does exist.
Has anyone come across this/have suggestions??
2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage:
Image file of size 149098688 loaded in 14 seconds.
2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR*
FSDirectory.unprotectedRenameTo: failed to rename
/hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
to
/hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
because source does not exist
2013-12-07 01:20:43,348 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
2013-12-07 01:20:43,349 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at
namenode0.hadoop.compe.tv/10.10.11.10
Thank you,
--
iain
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
Re: Namenode won't start after unclean shutdown of RS
Posted by Ted Yu <yu...@gmail.com>.
What version of Hadoop / HBase are you using ?
Cheers
On Sat, Dec 7, 2013 at 9:38 AM, iain wright <ia...@gmail.com> wrote:
> Hi folks,
>
> One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted
> about reports failing, and i saw wierd logs in the jobtracker for "pending
> shutdown" etc.
>
> Proceeded to stop jobtracker/tt nodes, hbase, and hdfs.
>
> On attempting to turn back on the NN + DN's our namenode is failing to
> start, complaining about being unable to move one of our hbase region files
> because source file does exist.
>
> Has anyone come across this/have suggestions??
>
> 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 149098688 loaded in 14 seconds.
> 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo: failed to rename
>
> /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> to
>
> /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> because source does not exist
> 2013-12-07 01:20:43,348 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>
> 2013-12-07 01:20:43,349 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> namenode0.hadoop.compe.tv/10.10.11.10
>
> Thank you,
>
> --
> iain
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>