Experimentation harnesses?

[Michael Browning <in...@gmail.com>]

Hi all,

I was curious if anyone with an active Mesos deployment knows of, has used,
or has developed a harness for integration and exploratory testing against
your installations. The sort of capabilities I'm after include:

   - Sufficient flexibility to allow the launch of multiple frameworks in
   test setup -- this could involve deployment logic, e.g. fetching and
   installing a package or a provisioning script on a host.
   - Sufficient flexibility to allow the orchestration of multiple
   frameworks in a test run, to e.g. understand how different combinations of
   frameworks interact with each other under various usage scenarios, how they
   interact under quota, etc.
   - Sufficient flexibility to allow the gathering of many different
   metrics -- depending on what's being investigated, we might want to be able
   to include various host-level metrics in addition to the metrics and gauges
   that Mesos itself exposes.

This set of capabilities is something I'd expect from a distributed testing
framework, but Googling around hasn't yielded any immediately convincing
open source offerings -- things like Locust or Tsung are focused on
stress-testing and seem to lack the orchestration and provisioning
abilities I'm looking for. LinkedIn seems to have their own open source
offering for this, called Zopkio, that seems like it could hit the above
points, but it doesn't seem to be widely-used and I'm not sure how mature
it is.

Does anyone have any leads in this area? Have you implemented your own
solution? I'd be curious to hear how you've approached this problem.

Regards,
Michael

Re: Experimentation harnesses?

[Niklas Nielsen <ni...@qni.dk>]

Hi Michael,

We are working on this in context of generating workloads (with many
different combinations of latency critical workloads co-located with
best-effort ones) and testing scenarios for oversubscription and would love
to chat

Cheers,
Niklas

On Mon, Feb 22, 2016 at 4:14 PM, James Peach <jo...@gmail.com> wrote:

>
> > On Feb 22, 2016, at 11:57 AM, Michael Browning <in...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I was curious if anyone with an active Mesos deployment knows of, has
> used,
> > or has developed a harness for integration and exploratory testing
> against
> > your installations. The sort of capabilities I'm after include:
> >
> >   - Sufficient flexibility to allow the launch of multiple frameworks in
> >   test setup -- this could involve deployment logic, e.g. fetching and
> >   installing a package or a provisioning script on a host.
> >   - Sufficient flexibility to allow the orchestration of multiple
> >   frameworks in a test run, to e.g. understand how different
> combinations of
> >   frameworks interact with each other under various usage scenarios, how
> they
> >   interact under quota, etc.
> >   - Sufficient flexibility to allow the gathering of many different
> >   metrics -- depending on what's being investigated, we might want to be
> able
> >   to include various host-level metrics in addition to the metrics and
> gauges
> >   that Mesos itself exposes.
>
> At one point I wrote a small framework similar to mesos-execute to poke
> what I needed at the time. I considered adding Lua bindings to this so make
> it easier to make bespoke schedulers to hit different scenarios but never
> got around to investing the time :-/
>
> >
> > This set of capabilities is something I'd expect from a distributed
> testing
> > framework, but Googling around hasn't yielded any immediately convincing
> > open source offerings -- things like Locust or Tsung are focused on
> > stress-testing and seem to lack the orchestration and provisioning
> > abilities I'm looking for. LinkedIn seems to have their own open source
> > offering for this, called Zopkio, that seems like it could hit the above
> > points, but it doesn't seem to be widely-used and I'm not sure how mature
> > it is.
> >
> > Does anyone have any leads in this area? Have you implemented your own
> > solution? I'd be curious to hear how you've approached this problem.
> >
> > Regards,
> > Michael
>
>


-- 
Niklas

Re: Experimentation harnesses?

[James Peach <jo...@gmail.com>]

> On Feb 22, 2016, at 11:57 AM, Michael Browning <in...@gmail.com> wrote:
> 
> Hi all,
> 
> I was curious if anyone with an active Mesos deployment knows of, has used,
> or has developed a harness for integration and exploratory testing against
> your installations. The sort of capabilities I'm after include:
> 
>   - Sufficient flexibility to allow the launch of multiple frameworks in
>   test setup -- this could involve deployment logic, e.g. fetching and
>   installing a package or a provisioning script on a host.
>   - Sufficient flexibility to allow the orchestration of multiple
>   frameworks in a test run, to e.g. understand how different combinations of
>   frameworks interact with each other under various usage scenarios, how they
>   interact under quota, etc.
>   - Sufficient flexibility to allow the gathering of many different
>   metrics -- depending on what's being investigated, we might want to be able
>   to include various host-level metrics in addition to the metrics and gauges
>   that Mesos itself exposes.

At one point I wrote a small framework similar to mesos-execute to poke what I needed at the time. I considered adding Lua bindings to this so make it easier to make bespoke schedulers to hit different scenarios but never got around to investing the time :-/

> 
> This set of capabilities is something I'd expect from a distributed testing
> framework, but Googling around hasn't yielded any immediately convincing
> open source offerings -- things like Locust or Tsung are focused on
> stress-testing and seem to lack the orchestration and provisioning
> abilities I'm looking for. LinkedIn seems to have their own open source
> offering for this, called Zopkio, that seems like it could hit the above
> points, but it doesn't seem to be widely-used and I'm not sure how mature
> it is.
> 
> Does anyone have any leads in this area? Have you implemented your own
> solution? I'd be curious to hear how you've approached this problem.
> 
> Regards,
> Michael