You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2015/01/20 22:30:34 UTC

[jira] [Created] (MAPREDUCE-6219) Reduce memory required for FileInputFormat located status optimization

Jason Lowe created MAPREDUCE-6219:
-------------------------------------

             Summary: Reduce memory required for FileInputFormat located status optimization
                 Key: MAPREDUCE-6219
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6219
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 2.1.1-beta
            Reporter: Jason Lowe
            Priority: Minor


MAPREDUCE-1981 introduced an optimization to drastically reduce the number of namenode operations required to compute input splits when processing a directory.  However it requires more memory to perform this optimization as it retains the full LocatedFileStatus object for all input files while computing the splits.  This can lead to odd situations for users where using a directory as input can run the job client out of heap space but using directory/* as the input spec allows it to run within the original heap space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)