[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539180#comment-13539180 ] 

Arun C Murthy commented on YARN-2:
----------------------------------

Ok, I finally got around to finishing this up.

W.r.t cores, on some more thinking, I'm inclined to go along with the concept of integral 'virtual cores' instead of a float-precision 'cores' for the following reasons:
* It provides a level on indirection to deal with heterogenous cores viz. much more important for CPUs (as opposed to memory, disk b/w etc.). I've also added a notion of physical-to-virtual cores translation (yarn.nodemanager.vcores-pcores-ratio), per NodeManager, similar to physical-to-virtual memory translation (yarn.nodemanager.vmem-pmem-ratio) that we already have in place.
* It ensures we do minimal floating-point operations in the inner-most loop which are very expensive (for e.g. we dropped usage of Math.ceil in MAPREDUCE-1354 for JobTracker - Math.ceil is a JNI call). This is something I've been very focussed on since the dawn, which explains the integral implementations of divideAndCeil we've had since the beginning.

To make it clear, I've also renamed the apis to be (get,set)VirtualCores and marked them evolving - in future we can add (get,set)Cores after we finalize how we specify not just #cores, but also their capabilities (gigahertz?).

Thoughts?
                
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.3-alpha
>
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira