Resource Reservations from Python framework

[Cosmo King <co...@spacecurve.com>]

Hi all,

I'm building a framework to execute short or long-running test suites against a mesos cluster, and one of my goals is to discover a set of available nodes and reserve these for a specific test suite and not release them until test execution is complete. The test execution may consist of hundreds of separate remote command executions by different processes (these might be achieved through multiple mesos task submissions). Once reserved, it's important that no other mesos jobs execute on those particular hosts.  

I have heard mention of a new resource reservation feature in mesos.  Is this type of use case possible with resource reservation, and are there any hints as to how to implement it?

I'm using a custom framework based loosely on the original mesos_submit.py and test_framework.py.  I have basic scheduler and executor functionality working well, but I would like a way to dedicate a set of resources from start to finish.

If i have to use another language to achieve the reservation, that is fine.

Any suggestions?

Thanks!
Cosmo

Re: Resource Reservations from Python framework

[Cosmo King <co...@spacecurve.com>]

Benjamin,

Our product is designed to run on bare metal, and as performance is our top selling point so much of our testing revolves around correctly measuring the performance of workflows against our distributed database. We create test clusters at varying scale points on a shared pool of physical cluster nodes.  When running a performance test we have to ensure nothing but this test  and a dedicated product installation is running on the resources, so we can most reliably simulate the environment in which the product will be running in the field for our test workflows.

I'm envisioning a scenario where we can support many concurrent dynamically allocated test clusters on pools of physical nodes, reserving and releasing the resources as tests come and go.  And of course queueing up tests where resources are insufficient and allowing them to run when available (that part at least seems like a very mesos friendly workflow).

Of course we always have power users requesting nodes throughout the day for demos & POCs as well and they would not appreciate my test coming along and stopming on their system.  

We have a lot of automation already using a python test environment and Jenkins continuous integration, but a weakpoint is remote command execution over ssh and static test configuration of target clusters.

Oh yeah, I have to deploy the same test scenarios to EC2 and similar external clouds as well as internal private cloud :)

I'm hoping mesos will help, but looks like I have to write some code to make it happen.  I mean, I get to write some code:)

Cheers,
Cosmo

On Oct 23, 2013, at 11:01 AM, Benjamin Mahler <be...@gmail.com> wrote:

> Just to satisfy my curiosity, why is it important for there to be no other mesos executors running on the hosts? Is this due of a lack of a particular kind of isolation? (I'd like to understand this requirement better, as with proper isolation it should not be necessary).
> 
> Yes, you can use reservations to do this, however, at the current time there is no dynamic mechanism for changing reservations. You can start a slave reserving all the resources for your test suite framework, but the reservation will live for the entire lifetime of the slave.
> 
> 
> On Mon, Oct 21, 2013 at 11:56 PM, Cosmo King <co...@spacecurve.com> wrote:
> Hi all,
> 
> I'm building a framework to execute short or long-running test suites against a mesos cluster, and one of my goals is to discover a set of available nodes and reserve these for a specific test suite and not release them until test execution is complete. The test execution may consist of hundreds of separate remote command executions by different processes (these might be achieved through multiple mesos task submissions). Once reserved, it's important that no other mesos jobs execute on those particular hosts.
> 
> I have heard mention of a new resource reservation feature in mesos.  Is this type of use case possible with resource reservation, and are there any hints as to how to implement it?
> 
> I'm using a custom framework based loosely on the original mesos_submit.py and test_framework.py.  I have basic scheduler and executor functionality working well, but I would like a way to dedicate a set of resources from start to finish.
> 
> If i have to use another language to achieve the reservation, that is fine.
> 
> Any suggestions?
> 
> Thanks!
> Cosmo
> 

Re: Resource Reservations from Python framework

[Benjamin Mahler <be...@gmail.com>]

Just to satisfy my curiosity, why is it important for there to be no other
mesos executors running on the hosts? Is this due of a lack of a particular
kind of isolation? (I'd like to understand this requirement better, as with
proper isolation it should not be necessary).

Yes, you can use reservations to do this, however, at the current time
there is no dynamic mechanism for changing reservations. You can start a
slave reserving all the resources for your test suite framework, but the
reservation will live for the entire lifetime of the slave.


On Mon, Oct 21, 2013 at 11:56 PM, Cosmo King <co...@spacecurve.com> wrote:

> Hi all,
>
> I'm building a framework to execute short or long-running test suites
> against a mesos cluster, and one of my goals is to discover a set of
> available nodes and reserve these for a specific test suite and not release
> them until test execution is complete. The test execution may consist of
> hundreds of separate remote command executions by different processes
> (these might be achieved through multiple mesos task submissions). Once
> reserved, it's important that no other mesos jobs execute on those
> particular hosts.
>
> I have heard mention of a new resource reservation feature in mesos.  Is
> this type of use case possible with resource reservation, and are there any
> hints as to how to implement it?
>
> I'm using a custom framework based loosely on the original mesos_submit.py
> and test_framework.py.  I have basic scheduler and executor functionality
> working well, but I would like a way to dedicate a set of resources from
> start to finish.
>
> If i have to use another language to achieve the reservation, that is fine.
>
> Any suggestions?
>
> Thanks!
> Cosmo