You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/09/23 12:45:16 UTC
[jira] Created: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
-----------------------------------------------------------------------------------------------
Key: MAPREDUCE-1028
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.21.0
Reporter: Hemanth Yamijala
Assignee: Ravi Gummadi
Priority: Blocker
Fix For: 0.21.0
A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1028:
------------------------------------
Attachment: MR-1028.patch
Attaching patch with the fix. Writing testcase is in progress.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1028:
------------------------------------
Attachment: MR-1028.v1.1.patch
Attaching new patch with the suggested changes.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1028:
-----------------------------------------
Status: Open (was: Patch Available)
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758782#action_12758782 ]
Arun C Murthy commented on MAPREDUCE-1028:
------------------------------------------
bq. On second thoughts [...]
To clarify, my comment was pointing out scenarios where we jvm-reuse is on and we use the JVM of a successful task to run the cleanup task of an unrelated failed task.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759651#action_12759651 ]
Nigel Daley commented on MAPREDUCE-1028:
----------------------------------------
Ok, this class is package-private so I guess this is moot. Still, package-private is always better than protected if at all possible.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758744#action_12758744 ]
Arun C Murthy commented on MAPREDUCE-1028:
------------------------------------------
So, I assume the fix is to ensure that the cleanup task uses only 1 slot?
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759500#action_12759500 ]
Ravi Gummadi commented on MAPREDUCE-1028:
-----------------------------------------
Unit tests passed on my local machine with the latest patch.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759676#action_12759676 ]
Hudson commented on MAPREDUCE-1028:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #66 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/66/])
. Fixed number of slots occupied by cleanup tasks to one irrespective of slot size for the job. Contributed by Ravi Gummadi.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759295#action_12759295 ]
Devaraj Das commented on MAPREDUCE-1028:
----------------------------------------
The changes in the patch look fine. This should work with JVM reuse set to ON too.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758771#action_12758771 ]
Arun C Murthy commented on MAPREDUCE-1028:
------------------------------------------
On second thoughts we need to be a little careful here... if we are re-using the existing JVM we should not bother with manipulating the #slots occupied by the JVM, OTOH if we need a new JVM we should be using just 1. Right?
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759498#action_12759498 ]
Hadoop QA commented on MAPREDUCE-1028:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420542/MR-1028.v1.patch
against trunk revision 818674.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h1.grid.sp2.yahoo.net/2/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h1.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h1.grid.sp2.yahoo.net/2/console
This message is automatically generated.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1028:
------------------------------------
Release Note: Makes taskCleanup tasks to use 1 slot even for high memory jobs.
Status: Patch Available (was: Open)
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1028:
-----------------------------------------
Attachment: yhadoop-0.20-MR1028.patch
Patch for the Y-Hadoop distribution
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759058#action_12759058 ]
Devaraj Das commented on MAPREDUCE-1028:
----------------------------------------
After some thought, it seems like decrementing the slot count on a per task-used-slot count basis is harmless.. So, for now, let's just ensure that all special tasks (job-setup, task-cleanup and job-cleanup) take exactly one slot. I couldn't come up with a counter-example where this would lead to inconsistencies in the slot counts on the TT, or, would lead to fewer/more tasks to be launched than should be as per the slot count and the #slots required by tasks scheduled on that TT.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759621#action_12759621 ]
Jothi Padmanabhan commented on MAPREDUCE-1028:
----------------------------------------------
+1. Patch looks fine to me.
Also, the failed test, TestCopyFiles is a known issue.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-1028:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk and the Hadoop 0.21 branch. Thanks, Ravi !
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1028:
------------------------------------
Attachment: MR-1028.v1.patch
Attaching patch with the testcase.
Please review and provide your comments.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759633#action_12759633 ]
Nigel Daley commented on MAPREDUCE-1028:
----------------------------------------
Why is taskStatuses now protected? You've now made it part of the public API.
{quote}
+ protected TreeMap<TaskAttemptID,TaskStatus> taskStatuses =
{quote}
It should be package-private, no? That should still enable the unit test to inspect it.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759709#action_12759709 ]
Hadoop QA commented on MAPREDUCE-1028:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420550/MR-1028.v1.1.patch
against trunk revision 818830.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/134/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/134/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/134/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/134/console
This message is automatically generated.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1028:
-----------------------------------------
Status: Patch Available (was: Open)
For some reason, Hudson did not pick this up. Retrying.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-1028:
-----------------------------------------
Status: Open (was: Patch Available)
Looks good. Some minor nits.
# Change the name of isMap arugment in TestSetupTaskScheduling.addNewTaskStatus
# Add the error strings to asserts
# Minor typo in one of the comments (should read reduce instead of map) in testNumSlotsUsedForTaskCleanup
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759677#action_12759677 ]
Hemanth Yamijala commented on MAPREDUCE-1028:
---------------------------------------------
bq. Ok, this class is package-private so I guess this is moot. Still, package-private is always better than protected if at all possible.
Filed MAPREDUCE-1041 to fix this.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760378#action_12760378 ]
Arun C Murthy commented on MAPREDUCE-1028:
------------------------------------------
The yahoo-20 patch is missing test-cases...
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch, yhadoop-0.20-MR1028.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1028) Cleanup tasks are scheduled using
high memory configuration, leaving tasks in unassigned state.
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-1028:
------------------------------------
Status: Patch Available (was: Open)
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758851#action_12758851 ]
Devaraj Das commented on MAPREDUCE-1028:
----------------------------------------
Hmm.. It should be okay to have a JVM occupying higher number of slots run a task that requires fewer slots. However, we need to fix the TaskTracker.TaskInProgress.releaseSlot. I am thinking that it might make sense to keep track of the slot count per JVM (long term, we anyway should be monitoring the resources being used by the JVM per se). Today, in releaseSlot, we release #slots equal to the number of slots that the task took to run. Instead it could just decrement the slot count by the number of slots the JVM took to run the task. Also, when the task is assigned to the TT, the JobInProgress.{obtainTaskCleanupTask,obtainJobCleanupTask,obtainJobSetupTask} methods should specifically sets the #slots required to 1 (today that's the only way to let the TT know that the task would require 1 slot).
The other option is to have the JobTracker be aware of slot counts for the special tasks. Since the special tasks are scheduled directly by the JobTracker, that would be required to be done.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759612#action_12759612 ]
Hadoop QA commented on MAPREDUCE-1028:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420550/MR-1028.v1.1.patch
against trunk revision 818830.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/133/console
This message is automatically generated.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MR-1028.patch, MR-1028.v1.1.patch, MR-1028.v1.patch
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1028) Cleanup tasks are scheduled
using high memory configuration, leaving tasks in unassigned state.
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758752#action_12758752 ]
Devaraj Das commented on MAPREDUCE-1028:
----------------------------------------
I think that should be it. That would also lead to non-wastage of slots.
> Cleanup tasks are scheduled using high memory configuration, leaving tasks in unassigned state.
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Assignee: Ravi Gummadi
> Priority: Blocker
> Fix For: 0.21.0
>
>
> A cleanup task is launched for a failed task of a job. This task is created based on the TIP of the failed task, and so is marked as requiring as many slots to run as the original task itself. For instance, if a high RAM job requires 2 slots per task, a cleanup task of the high RAM jobs requires 2 slots as well.
> Further, a cleanup task is scheduled to a tasktracker by the jobtracker itself and not the scheduler. While doing so, the JT doesn't check if the TT has enough slots free to run a high RAM cleanup task - always assuming 1 slot is enough. Thus, a task is oversubscribed to the TT.
> However, on the TT, before launch, we check that the task can actually run, and wait for so many slots to become available. If the slots don't get freed quickly, we will have tasks stuck in an unassigned state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.