You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2010/05/05 19:04:04 UTC
[jira] Created: (MAPREDUCE-1755) Zombie tasks kept alive by logging
system
Zombie tasks kept alive by logging system
-----------------------------------------
Key: MAPREDUCE-1755
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1755
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.2
Reporter: Allen Wittenauer
I'm currently looking at a task that, as far as the task tracker is concerned, is dead. Like long long long ago dead. It was a failed task that ran out of heap. Rather than just kill it, I thought I would see what it was doing, since it was clearly using system resources. It would appear the system is trying to log but failing. I'm guessing we're missing an error condition and not doing the appropriate thing. See the comments for more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1755) Zombie tasks kept alive by
logging system
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864469#action_12864469 ]
Allen Wittenauer commented on MAPREDUCE-1755:
---------------------------------------------
FWIW, all of the other zombie tasks appear to have the same loop.
It would be good if someone can confirm the same pattern on non-Solaris. I have a suspicion that the zombie tasks at Y! are showing the same behavio(u)r.
> Zombie tasks kept alive by logging system
> -----------------------------------------
>
> Key: MAPREDUCE-1755
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1755
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Allen Wittenauer
> Attachments: stderr.txt, syslog.txt, tightloop.txt
>
>
> I'm currently looking at a task that, as far as the task tracker is concerned, is dead. Like long long long ago dead. It was a failed task that ran out of heap. Rather than just kill it, I thought I would see what it was doing, since it was clearly using system resources. It would appear the system is trying to log but failing. I'm guessing we're missing an error condition and not doing the appropriate thing. See the comments for more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1755) Zombie tasks kept alive by logging
system
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-1755:
----------------------------------------
Attachment: tightloop.txt
stderr.txt
syslog.txt
Stripping out the threading info (pollsys, etc) shows that the tasks are in a tight loop around manipulating the log files (see tightloop.txt).
I'm also attaching the syslog and stderr logs from this task to see if that helps to figure out where we're missing a thread needing to get killed.
> Zombie tasks kept alive by logging system
> -----------------------------------------
>
> Key: MAPREDUCE-1755
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1755
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Allen Wittenauer
> Attachments: stderr.txt, syslog.txt, tightloop.txt
>
>
> I'm currently looking at a task that, as far as the task tracker is concerned, is dead. Like long long long ago dead. It was a failed task that ran out of heap. Rather than just kill it, I thought I would see what it was doing, since it was clearly using system resources. It would appear the system is trying to log but failing. I'm guessing we're missing an error condition and not doing the appropriate thing. See the comments for more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1755) Zombie tasks kept alive by logging
system
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-1755:
----------------------------------------
Attachment: jstack.txt
As reqeusted, a jstack.
> Zombie tasks kept alive by logging system
> -----------------------------------------
>
> Key: MAPREDUCE-1755
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1755
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Allen Wittenauer
> Attachments: jstack.txt, stderr.txt, syslog.txt, tightloop.txt
>
>
> I'm currently looking at a task that, as far as the task tracker is concerned, is dead. Like long long long ago dead. It was a failed task that ran out of heap. Rather than just kill it, I thought I would see what it was doing, since it was clearly using system resources. It would appear the system is trying to log but failing. I'm guessing we're missing an error condition and not doing the appropriate thing. See the comments for more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1755) Zombie tasks kept alive by
logging system
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864479#action_12864479 ]
Todd Lipcon commented on MAPREDUCE-1755:
----------------------------------------
Do you have a jstack from the JVM?
> Zombie tasks kept alive by logging system
> -----------------------------------------
>
> Key: MAPREDUCE-1755
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1755
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Allen Wittenauer
> Attachments: stderr.txt, syslog.txt, tightloop.txt
>
>
> I'm currently looking at a task that, as far as the task tracker is concerned, is dead. Like long long long ago dead. It was a failed task that ran out of heap. Rather than just kill it, I thought I would see what it was doing, since it was clearly using system resources. It would appear the system is trying to log but failing. I'm guessing we're missing an error condition and not doing the appropriate thing. See the comments for more.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.