You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2008/06/02 12:11:46 UTC
[jira] Updated: (HADOOP-2393) TaskTracker locks up removing job
files within a synchronized method
[ https://issues.apache.org/jira/browse/HADOOP-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-2393:
--------------------------------------------
Attachment: patch-2393.txt
Here is patch adding a daemon thread to the tasktracker delete files that are queued up. TaskTracker queues up all the file deletions to this new thread. This should remove the io operations in purgeJob().
> TaskTracker locks up removing job files within a synchronized method
> ---------------------------------------------------------------------
>
> Key: HADOOP-2393
> URL: https://issues.apache.org/jira/browse/HADOOP-2393
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.4
> Environment: 0.13.1, quad-code x86-64, FC-linux. -xmx2048
> ipc.client.timeout = 10000
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Priority: Critical
> Fix For: 0.18.0
>
> Attachments: patch-2393.txt
>
>
> we have some bad jobs where the reduces are getting stalled (for unknown reason). The task tracker kills these processes from time to time.
> Everytime one of these events happens - other (healthy) map tasks in the same node are also killed. Looking at the logs and code up to 0.14.3 - it seems like the child tasks pings to the task tracker are timed out and the child task self-terminates.
> tasktracker log:
> // notice the good 10+ second gap in logs on otherwise busy node:
> 2007-12-10 09:26:53,047 INFO org.apache.hadoop.mapred.TaskRunner: task_0120_r_000001_47 done; removing files.
> 2007-12-10 09:27:26,878 INFO org.apache.hadoop.mapred.TaskRunner: task_0120_m_000618_0 done; removing files.
> 2007-12-10 09:27:26,883 INFO org.apache.hadoop.ipc.Server: Process Thread Dump: Discarding call ping(task_0149_m_000007_0) from 10.16.158.113:43941
> 24 active threads
> ... huge stack trace dump in logfile ...
> something was going on at this time which caused to the tasktracker to essentially stall. all the pings are discarded. after stack trace dump:
> 2007-12-10 09:27:26,883 WARN org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50050, call ping(task_0149_m_000007_0) from 10.16.158.113:43941:\
> discarded for being too old (21380)
> 2007-12-10 09:27:26,883 WARN org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50050, call ping(task_0149_m_000002_1) from 10.16.158.113:44183:\
> discarded for being too old (21380)
> 2007-12-10 09:27:26,883 WARN org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50050, call ping(task_0149_m_000007_0) from 10.16.158.113:43941:\
> discarded for being too old (10367)
> 2007-12-10 09:27:26,883 WARN org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50050, call ping(task_0149_m_000002_1) from 10.16.158.113:44183:\
> discarded for being too old (10360)
> 2007-12-10 09:27:26,982 WARN org.apache.hadoop.mapred.TaskRunner: task_0149_m_000002_1 Child Error
> looking at code, failure of client to ping causes termination:
> else {
> // send ping
> taskFound = umbilical.ping(taskId);
> }
> ...
> catch (Throwable t) {
> LOG.info("Communication exception: " + StringUtils.stringifyException(t));
> remainingRetries -=1;
> if (remainingRetries == 0) {
> ReflectionUtils.logThreadInfo(LOG, "Communication exception", 0);
> LOG.warn("Last retry, killing "+taskId);
> System.exit(65);
> exit code is 65 as reported by task tracker.
> i don't see an option to turn off stack trace dump (which could be a likely cause) - and i would hate to bump up timeout because of this. Crap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.