You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by sam liu <sa...@gmail.com> on 2014/09/01 09:11:17 UTC
HDFS rollingUpgrade failed due to unexpected storage info
Hi Experts,
According to section 'Upgrading Non-Federated Clusters' of
http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html,
I tried to upgrade hadoop 2.2.0 to hadoop 2.4.1. However, I failed on step
2.2 'Start NN2 as standby with the "-rollingUpgrade started" option.'. The
exception is as below. Any comments from you will be appreciated!
14/09/01 11:54:22 INFO namenode.FSImage: Reading
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@cb7c0ebc
expect ing start txid #3353
14/09/01 11:54:22 INFO namenode.FSImage: Start loading edits file
http://host74-public:8480/getJournal?jid=BICluster&s
egmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5,
http://host71-public:8480
/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5
14/09/01 11:54:22 INFO namenode.EditLogInputStream: Fast-forwarding stream '
http://host74-public:8480/getJournal?jid=B
ICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5,
http://host71-public.cn.ib
m.com:8480/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5'
t o transaction ID 3353
14/09/01 11:54:22 INFO namenode.EditLogInputStream: Fast-forwarding stream '
http://host74-public:8480/getJournal?jid=B
ICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5'
to transaction ID 3353
14/09/01 11:54:22 ERROR namenode.EditLogInputStream: caught exception
initializing http://host74-public:8480/getJourna
l?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException:
Fetch of http://host74-public:8480/getJ
ournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5
failed with statu s code 403
Response message:
This node has storage info
'-47:1518542315:0:CID-9513e950-061b-4e96-924b-3faab87520d5' but the
requesting node expected '-56:151854
2315:0:CID-9513e950-061b-4e96-924b-3faab87520d5'
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:472)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:460)
at
java.security.AccessController.doPrivileged(AccessController.java:366)
at javax.security.auth.Subject.doAs(Subject.java:572)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1558)
at
org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
at
org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:459)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:145)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:196)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:254)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424)
14/09/01 11:54:22 ERROR namenode.EditLogInputStream: Got error reading edit
log input stream http://host74-public:8480
/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3ACID-9513e950-061b-4e96-924b-3faab87520d5;
failing ove r to edit log
http://host71-public:8480/getJournal?jid=BICluster&segmentTxId=3353&storageInfo=-56%3A1518542315%3A0%3AC
ID-9513e950-061b-4e96-924b-3faab87520d5
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException:
got premature end-of-file at txid 3352; e xpected file to go up to 3354
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424)
... ...
14/09/01 11:54:32 INFO mortbay.log: Stopped
SelectChannelConnector@host73-public:50070
14/09/01 11:54:32 INFO impl.MetricsSystemImpl: Stopping NameNode metrics
system...
14/09/01 11:54:32 INFO impl.MetricsSystemImpl: NameNode metrics system
stopped.
14/09/01 11:54:32 INFO impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
14/09/01 11:54:32 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error
replaying edit log at offset 0. Expected transaction ID was 33 53
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:193)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424)
Caused by:
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException:
got premature end-of-file at t xid 3352; expected file to go up to 3354
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:180)
... 12 more
14/09/01 11:54:32 INFO util.ExitUtil: Exiting with status 1
14/09/01 11:54:32 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at host73-public/9.11.64.232
************************************************************/