Application Dependency Managaement ala YARN

[Sam Taha <ta...@gmail.com>]

I am looking to implement a custom executor, but I do not want to require
users to distribute the JARs and other related resources associated with my
executor onto every slave node.

I see from the code that I can addUris and even ".tgz" that get downloaded
and extracted during the execution of the executor, but I am not clear if
this will be practical if my JARs and other resources needed by the java
executor will be in the 10s or 100s of megabytes.

It is important that I don't require the user to install anything on the
slave nodes other than the base Mesos package since my Framework
application is very dynamic and code jars can vary by job in most cases. So
I really want everything dynamically downloaded per job run and I would
also like the resources to be cached if they have not changed...etc.

Is there any plan to support something like what exist in YARN which deals
with these of lifecycle resource manage application dependencies across the
cluster? Here is theYARN overview of what I am kind of asking about:

http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/

Thanks in advance.

Thanks,
Sam Taha

http://www.grandlogic.com

Re: Application Dependency Managaement ala YARN

[Vinod Kone <vi...@gmail.com>]

Sam, following are the tickets you want to follow (or contribute to :)).

MESOS-700 <https://issues.apache.org/jira/browse/MESOS-700>
MESOS-336 <https://issues.apache.org/jira/browse/MESOS-336>


On Tue, Oct 22, 2013 at 3:34 PM, Sam Taha <ta...@gmail.com> wrote:

> I am looking to implement a custom executor, but I do not want to require
> users to distribute the JARs and other related resources associated with my
> executor onto every slave node.
>
> I see from the code that I can addUris and even ".tgz" that get downloaded
> and extracted during the execution of the executor, but I am not clear if
> this will be practical if my JARs and other resources needed by the java
> executor will be in the 10s or 100s of megabytes.
>
> It is important that I don't require the user to install anything on the
> slave nodes other than the base Mesos package since my Framework
> application is very dynamic and code jars can vary by job in most cases. So
> I really want everything dynamically downloaded per job run and I would
> also like the resources to be cached if they have not changed...etc.
>
> Is there any plan to support something like what exist in YARN which deals
> with these of lifecycle resource manage application dependencies across the
> cluster? Here is theYARN overview of what I am kind of asking about:
>
> http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
>
> Thanks in advance.
>
> Thanks,
> Sam Taha
>
> http://www.grandlogic.com
>