You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2007/11/28 21:14:43 UTC
[jira] Updated: (PIG-14) large key cause pig reduce jobs to die
[ https://issues.apache.org/jira/browse/PIG-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-14:
------------------------------
Attachment: heartbeat.patch
Please review the patch. All unit tests including the new ones are passing. about 80 end-to-end tests are passing. Finally and large data test that used to have this problem now does now show it.
> large key cause pig reduce jobs to die
> --------------------------------------
>
> Key: PIG-14
> URL: https://issues.apache.org/jira/browse/PIG-14
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Olga Natkovich
> Attachments: heartbeat.patch
>
>
> The reducer sends a heartbeat to the task tracker every time it starts processing new key. The task tracker expects to
> get a message every 10 minutes. If processing of an individual key takes longer, which could be the case for your job,
> the task tracker would not get a heartbeat in time and would kill the task.
> The current patch is to add <property>
> <name>mapred.task.timeout</name>
> <value>0</value>
> <description>timeout value</description>
> </property>
> to the cluster's hadoop-site.xml. This results in disabling heartbeat functionality which might not be what we want
> long term.
> A more flexible approach is to periodically report from map and reduce job via
> http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/Reporter.html#setStatus(java.lang.String)
> As a workaround for a UDF, call: PigMapReduce.reporter.progress() every 1000th time
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.