You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Srikanth Viswanathan <sr...@gmail.com> on 2016/02/25 18:00:44 UTC
"Chaos monkey" for mesos?
Has there been any work done to develop a "chaos monkey
<https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey>" analogue for
Mesos? I have been researching on how to write one, but I wanted to know if
there's any work already available that I can take a look at for
comparison, and possibly re-use.
The end goal would be something loaded into Mesos or separate from Mesos
that randomly kills tasks. Could it be something as simple as an
application that uses the KILL HTTP request from the scheduler API to kill
tasks?
Thanks.
Srikanth
Re: "Chaos monkey" for mesos?
Posted by Olaf Magnetic <ol...@magnetic.io>.
Hi everybody, we will release our "workflows" feature for Vamp (www.vamp.io) in our upcoming 0.9.0 release in a few weeks time.
Here you can create your own workflows that run as Cronos (or plain Quartz) jobs and can access all underlying API's in a simple and consolidated way.
Using the workflow feature to create a chaos-monkey functionality (including metrics and dashboarding) will be a few hours time max.
If interest we can give access to the experimental build.
Cheers, Olaf
> On 25 Feb 2016, at 22:58, Connor Doyle <co...@gmail.com> wrote:
>
> I think you could approximate that tool's behavior with some scripting
> plus `mesos-execute` (ships with the distribution) or by writing a
> really simple framework that just turns things off.
>
> On Thu, Feb 25, 2016 at 1:14 PM, Srikanth Viswanathan
> <sr...@gmail.com> wrote:
>> Thanks. Craig and David. I'm curious about the design and use of that tool.
>> Based on the video, it looks close to what I hope to do.
>>
>> A web search didn't yield any results about it, however. Does anyone here
>> know more about the dcos chaos tool?
>>
>> Thanks again.
>> Srikanth
>>
>>> On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
>>>
>>> here's a direct link in the video
>>> https://youtu.be/0I6qG9RQUnY?t=389
>>>
>>>> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com> wrote:
>>>>
>>>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
>>>> sure if that's what your looking for, but it might be something to follow up
>>>> on somehow.
>>>>
>>>> https://mesosphere.com/learn/
>>>>
>>>> David Wood
>>>> Computing Systems for Wireless Networks
>>>> IBM TJ Watson Research Center
>>>> dawood@us.ibm.com
>>>> 914-945-4923 (office), 914-396-6515 (mobile)
>>>>
>>>>
>>>>
>>>>
>>>> From: Srikanth Viswanathan <sr...@gmail.com>
>>>> To: user@mesos.apache.org
>>>> Date: 02/25/2016 12:01 PM
>>>> Subject: "Chaos monkey" for mesos?
>>>> ________________________________
>>>>
>>>>
>>>>
>>>> Has there been any work done to develop a "chaos monkey" analogue for
>>>> Mesos? I have been researching on how to write one, but I wanted to know if
>>>> there's any work already available that I can take a look at for comparison,
>>>> and possibly re-use.
>>>>
>>>> The end goal would be something loaded into Mesos or separate from Mesos
>>>> that randomly kills tasks. Could it be something as simple as an application
>>>> that uses the KILL HTTP request from the scheduler API to kill tasks?
>>>>
>>>> Thanks.
>>>>
>>>> Srikanth
>>>
>>>
>>>
>>> --
>>>
>>> https://github.com/mindscratch
>>> https://www.google.com/+CraigWickesser
>>> https://twitter.com/mind_scratch
>>> https://twitter.com/craig_links
>
>
>
> --
> connor
Re: "Chaos monkey" for mesos?
Posted by Connor Doyle <co...@gmail.com>.
There's no way to kill a single task through the Mesos control
surfaces, but if you let the "chaos" framework launch tasks as a
privileged user, you can run wild.
On Thu, Feb 25, 2016 at 2:49 PM, Srikanth Viswanathan
<sr...@gmail.com> wrote:
> Sorry, ignore my first question. A framework can obviously kill tasks. I was
> just unsure as to whether it can kill foreign tasks, which leaves only my
> second question.
>
> On Thu, Feb 25, 2016 at 5:23 PM, Srikanth Viswanathan <sr...@gmail.com>
> wrote:
>>
>> Appreciate all the responses here. I'll look into `mesos-execute`.
>>
>> I was thinking about the framework idea in passing but my mesos knowledge
>> isn't up to scratch yet, so I haven't been able pursue it yet. There are
>> many questions in my mind w.r.t designing this as a framework:
>> * Doesn't a framework only receive offers from mesos and launch tasks? How
>> would a framework kill tasks? Can it also kill slaves?
>> * Is it legal in mesos for one framework to kill tasks belonging to
>> another framework?
>>
>> Thanks.
>> Srikanth
>>
>> On Thu, Feb 25, 2016 at 4:58 PM, Connor Doyle <co...@gmail.com>
>> wrote:
>>>
>>> I think you could approximate that tool's behavior with some scripting
>>> plus `mesos-execute` (ships with the distribution) or by writing a
>>> really simple framework that just turns things off.
>>>
>>> On Thu, Feb 25, 2016 at 1:14 PM, Srikanth Viswanathan
>>> <sr...@gmail.com> wrote:
>>> > Thanks. Craig and David. I'm curious about the design and use of that
>>> > tool.
>>> > Based on the video, it looks close to what I hope to do.
>>> >
>>> > A web search didn't yield any results about it, however. Does anyone
>>> > here
>>> > know more about the dcos chaos tool?
>>> >
>>> > Thanks again.
>>> > Srikanth
>>> >
>>> > On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
>>> >>
>>> >> here's a direct link in the video
>>> >> https://youtu.be/0I6qG9RQUnY?t=389
>>> >>
>>> >> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com>
>>> >> wrote:
>>> >>>
>>> >>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
>>> >>> sure if that's what your looking for, but it might be something to
>>> >>> follow up
>>> >>> on somehow.
>>> >>>
>>> >>> https://mesosphere.com/learn/
>>> >>>
>>> >>> David Wood
>>> >>> Computing Systems for Wireless Networks
>>> >>> IBM TJ Watson Research Center
>>> >>> dawood@us.ibm.com
>>> >>> 914-945-4923 (office), 914-396-6515 (mobile)
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> From: Srikanth Viswanathan <sr...@gmail.com>
>>> >>> To: user@mesos.apache.org
>>> >>> Date: 02/25/2016 12:01 PM
>>> >>> Subject: "Chaos monkey" for mesos?
>>> >>> ________________________________
>>> >>>
>>> >>>
>>> >>>
>>> >>> Has there been any work done to develop a "chaos monkey" analogue for
>>> >>> Mesos? I have been researching on how to write one, but I wanted to
>>> >>> know if
>>> >>> there's any work already available that I can take a look at for
>>> >>> comparison,
>>> >>> and possibly re-use.
>>> >>>
>>> >>> The end goal would be something loaded into Mesos or separate from
>>> >>> Mesos
>>> >>> that randomly kills tasks. Could it be something as simple as an
>>> >>> application
>>> >>> that uses the KILL HTTP request from the scheduler API to kill tasks?
>>> >>>
>>> >>> Thanks.
>>> >>>
>>> >>> Srikanth
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> https://github.com/mindscratch
>>> >> https://www.google.com/+CraigWickesser
>>> >> https://twitter.com/mind_scratch
>>> >> https://twitter.com/craig_links
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> connor
>>
>>
>
--
connor
Re: "Chaos monkey" for mesos?
Posted by Srikanth Viswanathan <sr...@gmail.com>.
Sorry, ignore my first question. A framework can obviously kill tasks. I
was just unsure as to whether it can kill foreign tasks, which leaves only
my second question.
On Thu, Feb 25, 2016 at 5:23 PM, Srikanth Viswanathan <sr...@gmail.com>
wrote:
> Appreciate all the responses here. I'll look into `mesos-execute`.
>
> I was thinking about the framework idea in passing but my mesos knowledge
> isn't up to scratch yet, so I haven't been able pursue it yet. There are
> many questions in my mind w.r.t designing this as a framework:
> * Doesn't a framework only receive offers from mesos and launch tasks? How
> would a framework kill tasks? Can it also kill slaves?
> * Is it legal in mesos for one framework to kill tasks belonging to
> another framework?
>
> Thanks.
> Srikanth
>
> On Thu, Feb 25, 2016 at 4:58 PM, Connor Doyle <co...@gmail.com>
> wrote:
>
>> I think you could approximate that tool's behavior with some scripting
>> plus `mesos-execute` (ships with the distribution) or by writing a
>> really simple framework that just turns things off.
>>
>> On Thu, Feb 25, 2016 at 1:14 PM, Srikanth Viswanathan
>> <sr...@gmail.com> wrote:
>> > Thanks. Craig and David. I'm curious about the design and use of that
>> tool.
>> > Based on the video, it looks close to what I hope to do.
>> >
>> > A web search didn't yield any results about it, however. Does anyone
>> here
>> > know more about the dcos chaos tool?
>> >
>> > Thanks again.
>> > Srikanth
>> >
>> > On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
>> >>
>> >> here's a direct link in the video
>> >> https://youtu.be/0I6qG9RQUnY?t=389
>> >>
>> >> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com>
>> wrote:
>> >>>
>> >>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
>> >>> sure if that's what your looking for, but it might be something to
>> follow up
>> >>> on somehow.
>> >>>
>> >>> https://mesosphere.com/learn/
>> >>>
>> >>> David Wood
>> >>> Computing Systems for Wireless Networks
>> >>> IBM TJ Watson Research Center
>> >>> dawood@us.ibm.com
>> >>> 914-945-4923 (office), 914-396-6515 (mobile)
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> From: Srikanth Viswanathan <sr...@gmail.com>
>> >>> To: user@mesos.apache.org
>> >>> Date: 02/25/2016 12:01 PM
>> >>> Subject: "Chaos monkey" for mesos?
>> >>> ________________________________
>> >>>
>> >>>
>> >>>
>> >>> Has there been any work done to develop a "chaos monkey" analogue for
>> >>> Mesos? I have been researching on how to write one, but I wanted to
>> know if
>> >>> there's any work already available that I can take a look at for
>> comparison,
>> >>> and possibly re-use.
>> >>>
>> >>> The end goal would be something loaded into Mesos or separate from
>> Mesos
>> >>> that randomly kills tasks. Could it be something as simple as an
>> application
>> >>> that uses the KILL HTTP request from the scheduler API to kill tasks?
>> >>>
>> >>> Thanks.
>> >>>
>> >>> Srikanth
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> https://github.com/mindscratch
>> >> https://www.google.com/+CraigWickesser
>> >> https://twitter.com/mind_scratch
>> >> https://twitter.com/craig_links
>> >
>> >
>>
>>
>>
>> --
>> connor
>>
>
>
Re: "Chaos monkey" for mesos?
Posted by Srikanth Viswanathan <sr...@gmail.com>.
Appreciate all the responses here. I'll look into `mesos-execute`.
I was thinking about the framework idea in passing but my mesos knowledge
isn't up to scratch yet, so I haven't been able pursue it yet. There are
many questions in my mind w.r.t designing this as a framework:
* Doesn't a framework only receive offers from mesos and launch tasks? How
would a framework kill tasks? Can it also kill slaves?
* Is it legal in mesos for one framework to kill tasks belonging to another
framework?
Thanks.
Srikanth
On Thu, Feb 25, 2016 at 4:58 PM, Connor Doyle <co...@gmail.com> wrote:
> I think you could approximate that tool's behavior with some scripting
> plus `mesos-execute` (ships with the distribution) or by writing a
> really simple framework that just turns things off.
>
> On Thu, Feb 25, 2016 at 1:14 PM, Srikanth Viswanathan
> <sr...@gmail.com> wrote:
> > Thanks. Craig and David. I'm curious about the design and use of that
> tool.
> > Based on the video, it looks close to what I hope to do.
> >
> > A web search didn't yield any results about it, however. Does anyone here
> > know more about the dcos chaos tool?
> >
> > Thanks again.
> > Srikanth
> >
> > On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
> >>
> >> here's a direct link in the video
> >> https://youtu.be/0I6qG9RQUnY?t=389
> >>
> >> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com> wrote:
> >>>
> >>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
> >>> sure if that's what your looking for, but it might be something to
> follow up
> >>> on somehow.
> >>>
> >>> https://mesosphere.com/learn/
> >>>
> >>> David Wood
> >>> Computing Systems for Wireless Networks
> >>> IBM TJ Watson Research Center
> >>> dawood@us.ibm.com
> >>> 914-945-4923 (office), 914-396-6515 (mobile)
> >>>
> >>>
> >>>
> >>>
> >>> From: Srikanth Viswanathan <sr...@gmail.com>
> >>> To: user@mesos.apache.org
> >>> Date: 02/25/2016 12:01 PM
> >>> Subject: "Chaos monkey" for mesos?
> >>> ________________________________
> >>>
> >>>
> >>>
> >>> Has there been any work done to develop a "chaos monkey" analogue for
> >>> Mesos? I have been researching on how to write one, but I wanted to
> know if
> >>> there's any work already available that I can take a look at for
> comparison,
> >>> and possibly re-use.
> >>>
> >>> The end goal would be something loaded into Mesos or separate from
> Mesos
> >>> that randomly kills tasks. Could it be something as simple as an
> application
> >>> that uses the KILL HTTP request from the scheduler API to kill tasks?
> >>>
> >>> Thanks.
> >>>
> >>> Srikanth
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> https://github.com/mindscratch
> >> https://www.google.com/+CraigWickesser
> >> https://twitter.com/mind_scratch
> >> https://twitter.com/craig_links
> >
> >
>
>
>
> --
> connor
>
Re: "Chaos monkey" for mesos?
Posted by Connor Doyle <co...@gmail.com>.
I think you could approximate that tool's behavior with some scripting
plus `mesos-execute` (ships with the distribution) or by writing a
really simple framework that just turns things off.
On Thu, Feb 25, 2016 at 1:14 PM, Srikanth Viswanathan
<sr...@gmail.com> wrote:
> Thanks. Craig and David. I'm curious about the design and use of that tool.
> Based on the video, it looks close to what I hope to do.
>
> A web search didn't yield any results about it, however. Does anyone here
> know more about the dcos chaos tool?
>
> Thanks again.
> Srikanth
>
> On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
>>
>> here's a direct link in the video
>> https://youtu.be/0I6qG9RQUnY?t=389
>>
>> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com> wrote:
>>>
>>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
>>> sure if that's what your looking for, but it might be something to follow up
>>> on somehow.
>>>
>>> https://mesosphere.com/learn/
>>>
>>> David Wood
>>> Computing Systems for Wireless Networks
>>> IBM TJ Watson Research Center
>>> dawood@us.ibm.com
>>> 914-945-4923 (office), 914-396-6515 (mobile)
>>>
>>>
>>>
>>>
>>> From: Srikanth Viswanathan <sr...@gmail.com>
>>> To: user@mesos.apache.org
>>> Date: 02/25/2016 12:01 PM
>>> Subject: "Chaos monkey" for mesos?
>>> ________________________________
>>>
>>>
>>>
>>> Has there been any work done to develop a "chaos monkey" analogue for
>>> Mesos? I have been researching on how to write one, but I wanted to know if
>>> there's any work already available that I can take a look at for comparison,
>>> and possibly re-use.
>>>
>>> The end goal would be something loaded into Mesos or separate from Mesos
>>> that randomly kills tasks. Could it be something as simple as an application
>>> that uses the KILL HTTP request from the scheduler API to kill tasks?
>>>
>>> Thanks.
>>>
>>> Srikanth
>>>
>>
>>
>>
>> --
>>
>> https://github.com/mindscratch
>> https://www.google.com/+CraigWickesser
>> https://twitter.com/mind_scratch
>> https://twitter.com/craig_links
>
>
--
connor
Re: "Chaos monkey" for mesos?
Posted by Srikanth Viswanathan <sr...@gmail.com>.
Thanks. Craig and David. I'm curious about the design and use of that tool.
Based on the video, it looks close to what I hope to do.
A web search didn't yield any results about it, however. Does anyone here
know more about the dcos chaos tool?
Thanks again.
Srikanth
On Thu, Feb 25, 2016 at 12:21 PM, craig w <co...@gmail.com> wrote:
> here's a direct link in the video
> https://youtu.be/0I6qG9RQUnY?t=389
>
> On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com> wrote:
>
>> The DCOS tutorial mentions a chaos tool at the end of the video. Not
>> sure if that's what your looking for, but it might be something to follow
>> up on somehow.
>>
>> https://mesosphere.com/learn/
>>
>> David Wood
>> Computing Systems for Wireless Networks
>> IBM TJ Watson Research Center
>> dawood@us.ibm.com
>> 914-945-4923 (office), 914-396-6515 (mobile)
>>
>>
>>
>>
>> From: Srikanth Viswanathan <sr...@gmail.com>
>> To: user@mesos.apache.org
>> Date: 02/25/2016 12:01 PM
>> Subject: "Chaos monkey" for mesos?
>> ------------------------------
>>
>>
>>
>> Has there been any work done to develop a "*chaos monkey*
>> <https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey>" analogue for
>> Mesos? I have been researching on how to write one, but I wanted to know if
>> there's any work already available that I can take a look at for
>> comparison, and possibly re-use.
>>
>> The end goal would be something loaded into Mesos or separate from Mesos
>> that randomly kills tasks. Could it be something as simple as an
>> application that uses the KILL HTTP request from the scheduler API to kill
>> tasks?
>>
>> Thanks.
>>
>> Srikanth
>>
>>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>
Re: "Chaos monkey" for mesos?
Posted by craig w <co...@gmail.com>.
here's a direct link in the video
https://youtu.be/0I6qG9RQUnY?t=389
On Thu, Feb 25, 2016 at 12:17 PM, David Wood <da...@us.ibm.com> wrote:
> The DCOS tutorial mentions a chaos tool at the end of the video. Not sure
> if that's what your looking for, but it might be something to follow up on
> somehow.
>
> https://mesosphere.com/learn/
>
> David Wood
> Computing Systems for Wireless Networks
> IBM TJ Watson Research Center
> dawood@us.ibm.com
> 914-945-4923 (office), 914-396-6515 (mobile)
>
>
>
>
> From: Srikanth Viswanathan <sr...@gmail.com>
> To: user@mesos.apache.org
> Date: 02/25/2016 12:01 PM
> Subject: "Chaos monkey" for mesos?
> ------------------------------
>
>
>
> Has there been any work done to develop a "*chaos monkey*
> <https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey>" analogue for
> Mesos? I have been researching on how to write one, but I wanted to know if
> there's any work already available that I can take a look at for
> comparison, and possibly re-use.
>
> The end goal would be something loaded into Mesos or separate from Mesos
> that randomly kills tasks. Could it be something as simple as an
> application that uses the KILL HTTP request from the scheduler API to kill
> tasks?
>
> Thanks.
>
> Srikanth
>
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: "Chaos monkey" for mesos?
Posted by David Wood <da...@us.ibm.com>.
The DCOS tutorial mentions a chaos tool at the end of the video. Not sure
if that's what your looking for, but it might be something to follow up on
somehow.
https://mesosphere.com/learn/
David Wood
Computing Systems for Wireless Networks
IBM TJ Watson Research Center
dawood@us.ibm.com
914-945-4923 (office), 914-396-6515 (mobile)
From: Srikanth Viswanathan <sr...@gmail.com>
To: user@mesos.apache.org
Date: 02/25/2016 12:01 PM
Subject: "Chaos monkey" for mesos?
Has there been any work done to develop a "chaos monkey" analogue for
Mesos? I have been researching on how to write one, but I wanted to know
if there's any work already available that I can take a look at for
comparison, and possibly re-use.
The end goal would be something loaded into Mesos or separate from Mesos
that randomly kills tasks. Could it be something as simple as an
application that uses the KILL HTTP request from the scheduler API to kill
tasks?
Thanks.
Srikanth