[jira] [Resolved] (MYRIAD-7) Run MyriadMesosScheduler inside YARN's resource manager JVM.

["Santosh Marella (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MYRIAD-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santosh Marella resolved MYRIAD-7.
----------------------------------
    Resolution: Fixed

> Run MyriadMesosScheduler inside YARN's resource manager JVM.
> ------------------------------------------------------------
>
>                 Key: MYRIAD-7
>                 URL: https://issues.apache.org/jira/browse/MYRIAD-7
>             Project: Myriad
>          Issue Type: Improvement
>            Reporter: Santosh Marella
>            Assignee: Santosh Marella
>
> The objective of this change is to run MyriadMesosScheduler inside YARN's resource manager JVM.
> 1. Added Myriad{Fair,Capacity,Fifo}Scheduler classes that extend from Yarn's {Fair,Capacity,Fifo}Scheduler classes respectively.
> 2. Added build dependencies on Hadoop.
> 3. Encountered slf4j's "multiple binding" conflicts because drop wizard uses
>    logback implementation while YARN uses slf4j-log4j12 implementation for slf4j API.
>    After discussions with Mohit, we've decided to remove dependencies on drop wizard.
> 4. Refactored the rest of the code to that effect. Retained non-conflicting deps like
>    codahale metrics, jackson's dataformat/databind/annotations/yaml and hibernate validator.
> 5. Added build rules to package the myriad jar and the dependencies to build/libs dir.
>    These DO NOT INCLUDE hadoop jars and their dependencies as the intent is to
>    deploy myriad jar and it's deps into an existing YARN installation.
> Build and Deployment guidelines:
> 1. From myriad dir in the local git repo, run "./gradlew jar".
> 2. This should produce myriad-0.0.1.jar and other deps under build/libs dir.
> 3. Copy build/libs/*.jar to HADOOP_HOME/share/hadoop/yarn.
> 4. Modify HADOOP_HOME/etc/hadoop/yarn-site.xml to have the following entry:
> {code:xml}
>   <property>
>     <name>yarn.resourcemanager.scheduler.class</name>
>     <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
>   </property>
> {code}
> 5. Restart Resource Manager process.
> CAVEATS:
> - The REST API to myriad is currently broken as we eliminated the dependencies on dropwizard.
>   We need to bring up a web app for myriad and expose the
>   REST API through that.
> - Mesos also requires a native library (libmesos.so). Our build process currently
>   does not add that to myriad jar. We need to manually add that to YARN's native lib dir
>   $HADOOP_HOME/lib/native.
> I'll open tasks for both the above and track them separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)