[jira] [Updated] (TEZ-711) Fix memory leak when not reading from inputs due to caching

["Hitesh Shah (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hitesh Shah updated TEZ-711:
----------------------------
    Affects Version/s:     (was: 0.2.0)

> Fix memory leak when not reading from inputs due to caching
> -----------------------------------------------------------
>
>                 Key: TEZ-711
>                 URL: https://issues.apache.org/jira/browse/TEZ-711
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Siddharth Seth
>            Priority: Critical
>         Attachments: OOM-threaddump-711-5-patch.txt, OOM-threaddump-till-TEZ-752.txt, TEZ-711.5.txt, TEZ-711.wip.1.txt, TEZ-711.wip.2.txt, TEZ-711.wip.3.txt, TEZ-711.wip.4.txt
>
>
>   When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)