You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2010/09/02 07:36:53 UTC
[jira] Commented: (HDFS-1366) reduce namenode startup time by
optimising checkBlockInfo while loading fsimage
[ https://issues.apache.org/jira/browse/HDFS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905412#action_12905412 ]
dhruba borthakur commented on HDFS-1366:
----------------------------------------
{code}
"main" prio=10 tid=0x0000000040114800 nid=0x3f15 runnable [0x000000004022a000]
java.lang.Thread.State: RUNNABLE
at org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo.getNext(BlocksMap.java:277)
at org.apache.hadoop.hdfs.util.LightWeightGSet.get(LightWeightGSet.java:121)
at org.apache.hadoop.hdfs.server.namenode.BlocksMap.checkBlockInfo(BlocksMap.java:353)
at org.apache.hadoop.hdfs.server.namenode.BlocksMap.addINode(BlocksMap.java:370)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addToParent(FSDirectory.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:956)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:813)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:370)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:141)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:328)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:305)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:254)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:332)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1120)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1129)
{code}
The idea here is that when the fsimage is being loaded, BlocksMap.checkBlockInfo() will never find the block in the blocksMap, i.e. the call to blocks.get() will always return null.
> reduce namenode startup time by optimising checkBlockInfo while loading fsimage
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1366
> URL: https://issues.apache.org/jira/browse/HDFS-1366
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> The namenode spends about 10 minutes reading in a 14 GB fsimage file into memory and creating all the in-memory data structures. A jstack based debugger clearly shows that most of the time during the fsimage load is spent in BlocksMap.checkBlockInfo. There is a easy way to optimize this method especially for this code path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.