[jira] Commented: (HADOOP-5883) TaskMemoryMonitorThread might shoot down tasks even if their processes momentarily exceed the requested memory

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712630#action_12712630 ] 

Hemanth Yamijala commented on HADOOP-5883:
------------------------------------------

As per some offline discussions with Eric, Arun and Vinod, we decided to follow the approach mentioned below for improving the monitoring logic:

The particular case in HADOOP-5059 indicates that any process that executes a program from the Java code could occupy (momentarily) twice the amount of memory, due to the JVM's fork()+exec() approach. This happens during the fork() and before the exec() completes.

- Hence, if a process tree is over twice the configured memory limit, then it is shot down immediately.
- Also, if there are processes in the tree older than a certain interval and these processes alone contribute to being over limit, then the task is shot down.

By corollary, this means that if there are no processes in the tree older than a certain interval and the tree is still over limit, we give it a benefit of doubt to accomodate the case in HADOOP-5059.

> TaskMemoryMonitorThread might shoot down tasks even if their processes momentarily exceed the requested memory
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5883
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Hemanth Yamijala
>
> Currently the TaskMemoryMonitorThread kills tasks as soon as it detects they are consuming more memory than the max value specified. There are valid cases (see HADOOP-5059) where if a program is executed from the task, it might momentarily occupy twice the amount of memory for a short time. Ideally the monitoring thread should handle this case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.