[jira] Updated: (HADOOP-3581) Prevent memory intensive user tasks from taking down nodes

["Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod Kumar Vavilapalli updated HADOOP-3581:
--------------------------------------------

    Attachment: HADOOP-3581.20080901.2.txt

Included all the above comments. Also removed the ProcessTree abstraction as it proved to be a premature abstraction, and is not panning out nicely while passing things like pid, sigKillInterval.
 * monitoringInterval is configurable via mapred.tasktracker.taskMemoryManager.monitoringInterval and sigkill interval via mapred.tasktracker.procfsBasedProcessTree.sleepTimeBeforeSigKill.
 * Removed the processtreeimpl configuration parameter.
 * Added createProcessTreeInfo(TaskAttemptID tid, long memLimit) and removeProcessTreeInfo(TaskAttemptID tid) to help handle synchronization better. These are respectively called by startNewTask(LaunchTaskAction action) and reportTaskFinished(TaskAttemptID taskid) which update the processTreeInfo map accordingly.
 * A bit of refactoring - made ProcessInfo an inner static class of ProcfsBasedProcessTree.

> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>
>                 Key: HADOOP-3581
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3581
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: HADOOP-3581-final.txt, HADOOP-3581.20080901.2.txt, HADOOP-3581.6.0.txt, patch_3581_0.1.txt, patch_3581_3.3.txt, patch_3581_4.3.txt, patch_3581_4.4.txt, patch_3581_5.0.txt, patch_3581_5.2.txt
>
>
> Sometimes user Map/Reduce applications can get extremely memory intensive, maybe due to some inadvertent bugs in the user code, or the amount of data processed. When this happens, the user tasks start to interfere with the proper execution of other processes on the node, including other Hadoop daemons like the DataNode and TaskTracker. Thus, the node would become unusable for any Hadoop tasks. There should be a way to prevent such tasks from bringing down the node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.