Design advice

[Sam Taha <ta...@gmail.com>]

I am hoping to get some advice on how to build out my framework scheduler
for a job scheduling/processing platform, so I don't paint myself in a
corner with any potential limitations in mesos resource allocation or the
java SchedulerDriver capabilities. Some background:

- I have a JVM based job scheduling engine that schedules jobs based on
repeating time patterns (like cron) and also allows for on-demand job
execution.
- The system supports users creating an number of partitions/queues that
have resource preferences defined by the user. So one partitions may run a
certain set of jobs and another partition may run another set of jobs.
Currently the resources are statically allocated to one or more computers
and the user can define the max number of jobs that can be run on each host
in the Partition.
- The scheduling engine is all run out of a single JVM process that is
heavily threaded/concurrent.

So my initial thinking is to create a mesos Sched/Framework instance per
partition/queue thread in the my engine (there could be many partitions
since this is user created/controlled via GUI). Each SchedulerDriver
(living in the JVM) would request some set of resources from Mesos (as
defined by the size of of the queue of jobs running in the Partition) when
jobs become ready to run on reoccurring time schedules.

Now my question is, if it is practical to have multiple mesos Schedulers
Frameworks in a single JVM and each with different resource requirements.
Or should I build a single central mesos scheduler in the JVM and have all
my partitions/queues make request to this central mesos Scheduler that
talks to the mesos master?

I am not sure how mesos scales if for example you have many framework
schedulers running out of the same JVM/process? And in general if you have
a large number of Scheduler instances will this cause resource distribution
problems with on one scheduler getting too many resources allocated....etc.

So the bottom line question is should have create/instantiate a single
central Framework Scheduler in my engine (that proxies requests/offers with
mesos) or can I create one per Partition thread.

The per partition thread approach can give me finer resource control
(request, Filters,....etc) but I am not sure if this is practical.

Thanks,
Sam Taha
http://www.grandlogic.com

Re: Design advice

[Sam Taha <ta...@gmail.com>]

Yes that was my thinking for the slave/execution side of things. My
partitions should map pretty well to slave executors/tasks and the
cgroup/isolation capabilities of mesos will be great benefit. However, I
still have some cases where users want to run each job (even jobs in the
same partition grouping) to under its own linux process so I am hoping I
can make this a runtime decision whether to use a shared partition executor
(on the given slave) or launch a new executor just for that one job and
then terminate the executor at then end of job run (I think I can use the
default command executor for this...I think).

I will take your advice and build a central framework for now and have it
dispatch resources to each Partition thread.

Thanks for you help.

Sam Taha
http://www.grandlogic.com


On Tue, Sep 10, 2013 at 1:01 PM, Vinod Kone <vi...@gmail.com> wrote:

> Sam,
>
> Glad to see you are interested in building a framework on top of Mesos!
>
> From your description, it looks like your "partitions" can be directly
> mapped to Mesos "executors" and "jobs" to Mesos "tasks". In Mesos, each
> executor is run under a cgroup with a given set of resources. This enables
> multiple executors to be run on the same host with isolation. Also, an
> executor can run multiple jobs/tasks underneath it.
>
> In your system, do users need to specifically control the partitions?  I
> would imagine it would be one less thing for users to worry about if Mesos
> can take care of it? A nice thing about using executors is that the total
> no.of resources (for executor and tasks underneath it) can be dynamically
> adjusted. So you don't have to statically over/under provision resources
> for your partitions.
>
> As per multiple frameworks per jvm, while its definitely possible I
> wouldn't recommend it. I think you would be much better off running one
> central framework and running different partitions as executors.
>
> HTH,
>
>
> On Tue, Sep 10, 2013 at 9:41 AM, Sam Taha <ta...@gmail.com> wrote:
>
>> I am hoping to get some advice on how to build out my framework scheduler
>> for a job scheduling/processing platform, so I don't paint myself in a
>> corner with any potential limitations in mesos resource allocation or the
>> java SchedulerDriver capabilities. Some background:
>>
>> - I have a JVM based job scheduling engine that schedules jobs based on
>> repeating time patterns (like cron) and also allows for on-demand job
>> execution.
>> - The system supports users creating an number of partitions/queues that
>> have resource preferences defined by the user. So one partitions may run a
>> certain set of jobs and another partition may run another set of jobs.
>> Currently the resources are statically allocated to one or more computers
>> and the user can define the max number of jobs that can be run on each host
>> in the Partition.
>> - The scheduling engine is all run out of a single JVM process that is
>> heavily threaded/concurrent.
>>
>> So my initial thinking is to create a mesos Sched/Framework instance per
>> partition/queue thread in the my engine (there could be many partitions
>> since this is user created/controlled via GUI). Each SchedulerDriver
>> (living in the JVM) would request some set of resources from Mesos (as
>> defined by the size of of the queue of jobs running in the Partition) when
>> jobs become ready to run on reoccurring time schedules.
>>
>> Now my question is, if it is practical to have multiple mesos Schedulers
>> Frameworks in a single JVM and each with different resource requirements.
>> Or should I build a single central mesos scheduler in the JVM and have all
>> my partitions/queues make request to this central mesos Scheduler that
>> talks to the mesos master?
>>
>> I am not sure how mesos scales if for example you have many framework
>> schedulers running out of the same JVM/process? And in general if you have
>> a large number of Scheduler instances will this cause resource distribution
>> problems with on one scheduler getting too many resources allocated....etc.
>>
>> So the bottom line question is should have create/instantiate a single
>> central Framework Scheduler in my engine (that proxies requests/offers with
>> mesos) or can I create one per Partition thread.
>>
>> The per partition thread approach can give me finer resource control
>> (request, Filters,....etc) but I am not sure if this is practical.
>>
>> Thanks,
>> Sam Taha
>> http://www.grandlogic.com
>>
>>
>>
>

Re: Design advice

[Vinod Kone <vi...@gmail.com>]

Sam,

Glad to see you are interested in building a framework on top of Mesos!

>From your description, it looks like your "partitions" can be directly
mapped to Mesos "executors" and "jobs" to Mesos "tasks". In Mesos, each
executor is run under a cgroup with a given set of resources. This enables
multiple executors to be run on the same host with isolation. Also, an
executor can run multiple jobs/tasks underneath it.

In your system, do users need to specifically control the partitions?  I
would imagine it would be one less thing for users to worry about if Mesos
can take care of it? A nice thing about using executors is that the total
no.of resources (for executor and tasks underneath it) can be dynamically
adjusted. So you don't have to statically over/under provision resources
for your partitions.

As per multiple frameworks per jvm, while its definitely possible I
wouldn't recommend it. I think you would be much better off running one
central framework and running different partitions as executors.

HTH,


On Tue, Sep 10, 2013 at 9:41 AM, Sam Taha <ta...@gmail.com> wrote:

> I am hoping to get some advice on how to build out my framework scheduler
> for a job scheduling/processing platform, so I don't paint myself in a
> corner with any potential limitations in mesos resource allocation or the
> java SchedulerDriver capabilities. Some background:
>
> - I have a JVM based job scheduling engine that schedules jobs based on
> repeating time patterns (like cron) and also allows for on-demand job
> execution.
> - The system supports users creating an number of partitions/queues that
> have resource preferences defined by the user. So one partitions may run a
> certain set of jobs and another partition may run another set of jobs.
> Currently the resources are statically allocated to one or more computers
> and the user can define the max number of jobs that can be run on each host
> in the Partition.
> - The scheduling engine is all run out of a single JVM process that is
> heavily threaded/concurrent.
>
> So my initial thinking is to create a mesos Sched/Framework instance per
> partition/queue thread in the my engine (there could be many partitions
> since this is user created/controlled via GUI). Each SchedulerDriver
> (living in the JVM) would request some set of resources from Mesos (as
> defined by the size of of the queue of jobs running in the Partition) when
> jobs become ready to run on reoccurring time schedules.
>
> Now my question is, if it is practical to have multiple mesos Schedulers
> Frameworks in a single JVM and each with different resource requirements.
> Or should I build a single central mesos scheduler in the JVM and have all
> my partitions/queues make request to this central mesos Scheduler that
> talks to the mesos master?
>
> I am not sure how mesos scales if for example you have many framework
> schedulers running out of the same JVM/process? And in general if you have
> a large number of Scheduler instances will this cause resource distribution
> problems with on one scheduler getting too many resources allocated....etc.
>
> So the bottom line question is should have create/instantiate a single
> central Framework Scheduler in my engine (that proxies requests/offers with
> mesos) or can I create one per Partition thread.
>
> The per partition thread approach can give me finer resource control
> (request, Filters,....etc) but I am not sure if this is practical.
>
> Thanks,
> Sam Taha
> http://www.grandlogic.com
>
>
>