[jira] [Commented] (TEZ-711) Fix memory leak when not reading from inputs due to caching

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886097#comment-13886097 ] 

Rohini Palaniswamy commented on TEZ-711:
----------------------------------------

[~sseth],
    Tested TEZ-711-5.txt and it works. No OOM anymore. Took a cursory look at the patch as well. Code looks fine.

> Fix memory leak when not reading from inputs due to caching
> -----------------------------------------------------------
>
>                 Key: TEZ-711
>                 URL: https://issues.apache.org/jira/browse/TEZ-711
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: Rohini Palaniswamy
>            Assignee: Siddharth Seth
>            Priority: Critical
>         Attachments: TEZ-711.5.txt, TEZ-711.wip.1.txt, TEZ-711.wip.2.txt, TEZ-711.wip.3.txt, TEZ-711.wip.4.txt
>
>
>   When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)