[jira] Created: (HADOOP-5567) Add support for scheduling jobs based on memory requirements to the Fairscheduler

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

Add support for scheduling jobs based on memory requirements to the Fairscheduler
---------------------------------------------------------------------------------

                 Key: HADOOP-5567
                 URL: https://issues.apache.org/jira/browse/HADOOP-5567
             Project: Hadoop Core
          Issue Type: New Feature
          Components: contrib/fair-share
            Reporter: Hemanth Yamijala


In HADOOP-3759, we added the ability for users to specify jobs requesting for a certain amount of virtual memory. For e.g. users can say that their jobs require 2GB of memory to run. In HADOOP-4035, functionality was added to the capacity scheduler to schedule jobs based on this specified amount. This JIRA is to add similar support to the Fairshare scheduler.

The basic use case is that there are jobs that require a certain known amount of virtual memory, usually more than the JVM's heap size. This happens specifically for streaming jobs that can launch several processes from the child. Without being aware of these requirements, if tasks are scheduled on nodes just based on available slots, they have a potential of affecting the other processes running on the node, or if memory protection features are enabled (HADOOP-3581), they could result in the task being killed by the tasktracker.

The scheduler must take into account the requested amount of memory by the job, the amount of memory that can be committed to by a tracker, and schedule based on these inputs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.