[jira] [Commented] (MAPREDUCE-5603) Ability to disable FileInputFormat listLocatedStatus optimization to save client memory

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809537#comment-13809537 ] 

Jason Lowe commented on MAPREDUCE-5603:
---------------------------------------

Recently we ran across a jobclient that failed with an OOM error once we updated the cluster to 0.23.10.  The OOM was triggered by the FileInputFormat listLocatedStatus optimization from MAPREDUCE-1981, as the client now caches the BlockLocations of all files along with the FileStatus objects it was caching before.  Normally the user can bump the heap size of the client to work around this issue.  However if a job has an input with a particularly large number of BlockLocations, as this job did, it would be nice if the user had the option to disable the optimization to reduce the required memory necessary for input split calculations.

> Ability to disable FileInputFormat listLocatedStatus optimization to save client memory
> ---------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5603
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5603
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: client, mrv2
>    Affects Versions: 0.23.10, 2.2.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Minor
>
> It would be nice if users had the option to disable the listLocatedStatus optimization in FileInputFormat to save client memory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)