[jira] Updated: (PIG-445) Null Pointer Exceptions in the mappers leading to lot of retries

["Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shravan Matthur Narayanamurthy updated PIG-445:
-----------------------------------------------

    Attachment: mq.patch

> Null Pointer Exceptions in the mappers leading to lot of retries
> ----------------------------------------------------------------
>
>                 Key: PIG-445
>                 URL: https://issues.apache.org/jira/browse/PIG-445
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shravan Matthur Narayanamurthy
>            Assignee: Shravan Matthur Narayanamurthy
>         Attachments: mq.patch
>
>
> Even with successfully completed jobs, usually with a large data set, we see that there are NPE produced in the mappers which lead to task failure. However, this problem goes away on retries. The problem occurs at places where we access the reporter to report progress. 
> From the analysis, this should happen with jobs that use combiner. The combiner is called whenever the mapper outputs a buffer full of data. So the combiner is called multiple times in between a map task. In the Combiner.close method we currently set the reporter to null as it was assumed that combiner is called only after the entire output of map is produced. 
> The fix is to not set the reporter to null in the Combiner.close() method

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.