[jira] Commented: (MAPREDUCE-865) harchive: Reduce the number of open calls to _index and _masterindex

["Koji Noguchi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743041#action_12743041 ] 

Koji Noguchi commented on MAPREDUCE-865:
----------------------------------------

I believe  _masterindex is probably small enough to fit in memory(cache)
For  _index file, 1 million files can correspond to _index size of 100MBytes. (It depend on the path length)
Creating a local copy could be costly.

In our clusters, most of the files are mapreduce output files. 
/a/b/part-00000
/a/b/part-00001
/a/b/part-00002
...
These show up as a set in _index file in this order since HarFileSystem.getHarHash is written that way.
So instead of open->read->close _index for each part file, thinking of  keeping the index file open when possible.


> harchive: Reduce the number of open calls  to _index and _masterindex 
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-865
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-865
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: harchive
>            Reporter: Koji Noguchi
>            Priority: Minor
>
> When I have har file with 1000 files in it, 
>    % hadoop dfs -lsr har:///user/knoguchi/myhar.har/
> would open/read/close the _index/_masterindex files 1000 times.
> This makes the client slow and add some load to the namenode as well.
> Any ways to reduce this number?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.