[jira] [Updated] (TEZ-711) API changes for Reader to handle case of caching

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated TEZ-711:
-----------------------------------

    Assignee: Rohini Palaniswamy

Picking up this jira for fixing. Just planning to add

{code}
public void close() throws IOException;
{code}

to the org.apache.tez.runtime.api.Reader interface and implement the method for existing reader implementations in Tez. close() is generic enough to be in Reader interface and does not have to be delegated to KeyValueReader interface. Any one skipping reading from a input, can close() the input to free up resources.  Is that fine?

> API changes for Reader to handle case of caching
> ------------------------------------------------
>
>                 Key: TEZ-711
>                 URL: https://issues.apache.org/jira/browse/TEZ-711
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>
>   When you are reading from inputs and caching objects with vertex scope, you don't have to read the input again when container is reused. But it allocates memory and that leaks causing OOM. KeyValueReader does not have a API to close the reader to clear allotted memory without reading from it. Also if there was a option to pre-close inputs in Processor and not fetch input at all over the wire and do shuffle/sort it would be a good optimization.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)