You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by heyongqiang <he...@software.ict.ac.cn> on 2008/07/01 07:25:11 UTC
Should there be a way not maintaining the whole namespace structure in memory?
In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this is done when setting up the FSNameSpacs structure set up at namenode startup.
the namenode will analyze the fsimage file and edit log file. And if there are milllions of files or directories how it can be handled?
I have done an exprements by making dirs,before i exprements:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
153648 1193868 1275340 3.7
after i creating 10000 directories, it turns:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
169084 1193868 1275340 4.0
I m trying to improve the fsimage file,so that namenode can locate and load the needed information at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading the whole namespace structure at startup.
Best regards,
Yongqiang He
2008-07-01
Email: heyongqiang@software.ict.ac.cn
Tel: 86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology,
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China
Fw: Re: Should there be a way not maintaining the whole namespace structure in memory?
Posted by heyongqiang <he...@software.ict.ac.cn>.
发件人： heyongqiang
发送时间： 2008-07-01 13:28:33
收件人： core-user
抄送：
主题： Re: Should there be a way not maintaining the whole namespace structure in memory?
at now,i have not find a good way for maintainning the fsimage format. so, any suggestions?
thanks!
Best regards,
Yongqiang He
2008-07-01
Email: heyongqiang@software.ict.ac.cn
Tel: 86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology,
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China
发件人： heyongqiang
发送时间： 2008-07-01 13:25:11
收件人： core-user
抄送：
主题： Should there be a way not maintaining the whole namespace structure in memory?
In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this is done when setting up the FSNameSpacs structure set up at namenode startup.
the namenode will analyze the fsimage file and edit log file. And if there are milllions of files or directories how it can be handled?
I have done an exprements by making dirs,before i exprements:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
153648 1193868 1275340 3.7
after i creating 10000 directories, it turns:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
169084 1193868 1275340 4.0
I m trying to improve the fsimage file,so that namenode can locate and load the needed information at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading the whole namespace structure at startup.
Best regards,
Yongqiang He
2008-07-01
Email: heyongqiang@software.ict.ac.cn
Tel: 86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology,
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China
Re: Should there be a way not maintaining the whole namespace structure in memory?
Posted by heyongqiang <he...@software.ict.ac.cn>.
at now,i have not find a good way for maintainning the fsimage format. so, any suggestions?
thanks!
Best regards,
Yongqiang He
2008-07-01
Email: heyongqiang@software.ict.ac.cn
Tel: 86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology,
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China
发件人： heyongqiang
发送时间： 2008-07-01 13:25:11
收件人： core-user
抄送：
主题： Should there be a way not maintaining the whole namespace structure in memory?
In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this is done when setting up the FSNameSpacs structure set up at namenode startup.
the namenode will analyze the fsimage file and edit log file. And if there are milllions of files or directories how it can be handled?
I have done an exprements by making dirs,before i exprements:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
153648 1193868 1275340 3.7
after i creating 10000 directories, it turns:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
RSS SZ VSZ %MEM
169084 1193868 1275340 4.0
I m trying to improve the fsimage file,so that namenode can locate and load the needed information at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading the whole namespace structure at startup.
Best regards,
Yongqiang He
2008-07-01
Email: heyongqiang@software.ict.ac.cn
Tel: 86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology,
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China