You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Brahma Reddy Battula (JIRA)" <ji...@apache.org> on 2014/11/20 15:54:34 UTC
[jira] [Created] (HDFS-7414) Namenode got shutdown and can't
recover where edit update might be missed
Brahma Reddy Battula created HDFS-7414:
------------------------------------------
Summary: Namenode got shutdown and can't recover where edit update might be missed
Key: HDFS-7414
URL: https://issues.apache.org/jira/browse/HDFS-7414
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.1, 2.4.1
Reporter: Brahma Reddy Battula
Priority: Blocker
Scenario:
========
Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..
*Did not get the exact root cause, but as cpu usage more edit log updation might be missed...Need dig to more...anyone any thoughts.*
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation CloseOp [length=0, inodeId=0, path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025, replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864, blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337], permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=, clientMachine=, opCode=OP_CLOSE, txid=162982] | org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading fsimage | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)