[jira] [Created] (MYRIAD-127) Recommend nodes to launch NodeManagers optimized for locality based on input data for Yarn jobs

["Swapnil Daingade (JIRA)" <ji...@apache.org>]

Swapnil Daingade created MYRIAD-127:
---------------------------------------

             Summary: Recommend nodes to launch NodeManagers optimized for locality based on input data for Yarn jobs
                 Key: MYRIAD-127
                 URL: https://issues.apache.org/jira/browse/MYRIAD-127
             Project: Myriad
          Issue Type: New Feature
            Reporter: Swapnil Daingade
            Assignee: Swapnil Daingade


Hadoop/Yarn works on the principle of taking computation to the data. Thus data locality is important for getting optimal performance. When a yarn job is launched, the user specifies the dfs location of the data being operated on. Looking at all these locations (from various running yarn jobs) being operated on, we can try to predict the best location for launching NodeManagers optimized for locality.

For. e.g We have a 20 node mesos cluster and a user has a 5 node yarn cluster running a few jobs. Looking at the data being operated on by the yarn jobs, we can come up with a recommendation for which 5 nodes to launch NodeManagers on optimized for locality. 

In a more autonomous mode, we could flex down some of the existing NM's and flexup new ones on new nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)