Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Dick Davies <di...@hellooperator.net>]

If it _needs_ to be there always then I'd roll it out with whatever
automation you use to deploy the mesos workers ; depending on
the scale you're running at launching it as a task is likely to be less
reliable due to outages etc.

( I understand the 'maybe all hosts' constraint but if it's 'up to one per
host', it sounds like a CM issue to me. )

On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
> hi Mesos folks!
>
> My team is largely responsible for maintaining the Storm-on-Mesos framework.
> It suffers from a problem related to log retrieval:  Storm has a process
> called the "logviewer" that is assumed to exist on every host, and the Storm
> UI provides links to contact this process to download logs (and other
> debugging artifacts).   Our team manually runs this process on each Mesos
> host, but it would be nice to launch it automatically onto any Mesos host
> where Storm work gets allocated. [0]
>
> I have read that Mesos has added support for Kubernetes-esque "pods" as of
> version 1.1.0, but that feature seems somewhat insufficient for implementing
> our desired behavior from my naive understanding.  Specifically, Storm only
> has support for connecting to 1 logviewer per host, so unless pods can have
> separate containers inside each pod [1], and also dynamically change the set
> of executors and tasks inside of the pod [2], then I don't see how we'd be
> able to use them.
>
> Is there any existing feature in Mesos that might help us accomplish our
> goal?  Or any upcoming features?
>
> Thanks!!
>
> - Erik
>
> [0] Thus the "all" in quotes in the subject of this email, because it
> *might* be all hosts, but it definitely would be all hosts where Storm gets
> work assigned.
>
> [1] The Storm-on-Mesos framework leverages separate containers for each
> topology's Supervisor and Worker processes, to provide isolation between
> topologies.
>
> [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm Workers
> (a Mesos Task) onto hosts is ever changing in a given instance of a
> Storm-on-Mesos framework.  i.e., as topologies get launched and die, or have
> their worker processes die, the processes are dynamically distributed to the
> various Mesos Worker hosts.  So existing containers often have more tasks
> assigned into them (thus growing their footprint) or removed from them (thus
> shrinking the footprint).

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers <ew...@groupon.com>]

Ahhhh.  That would be satisfactory yes.

The framework is already "smart" (read as "overly complicated") since we
handle reassembly of fragmented offers, amongst other intricate coding
details.  So it does seem possible to add the kind of logic you suggest!
Thanks for the idea James!

My team currently uses the mesos containerizer, so the log access between
containers issue is not a problem like it might be with the docker
containerizer.

- Erik

On Sat, Jul 1, 2017 at 4:20 PM James Peach <jo...@gmail.com> wrote:

>
> > On Jul 1, 2017, at 11:14 AM, Erik Weathers <ew...@groupon.com>
> wrote:
> >
> > Thanks for the info Kevin.  Seems there's no JIRAs nor design docs
> floating about yet for "admin tasks" or "daemon sets".
> >
> > Just FYI, this is the ticket in Storm for the problem I've been
> mentioning:
> >
> > https://issues.apache.org/jira/browse/STORM-1342
> >
> > I'll update it with the info you've provided below, so for now we'll
> rely on manually deploying logviewers.
>
> ISTM that this should be possible with a smart framework. If the framework
> keeps track of which agents it gets offers for, it could ensure that it
> launches a Storm logviewer task on the agent before launching any other
> Storm tasks. I expect that it might be a little tricky to get the
> containerization right so that the Storm tasks can rendezvous with the
> logviewer, but in principle it could be made to work?
>
> >
> > Thanks!
> >
> > - Erik
> >
> > On Sat, Jul 1, 2017 at 10:09 AM Kevin Klues <kl...@gmail.com> wrote:
> > What you are describing is a feature we call 'admin tasks' or 'daemon
> sets'.
> >
> > Unfortunately, there is no direct support for these yet, but we do have
> plans in the (relatively) near future to start working on it.
> >
> > One of our use cases is exactly what you describe with the logging
> service. On DC/OS we currently run our logging service as a systemd unit
> outside of mesos since we can't guarantee it gets launched everywhere (the
> same is true for a bunch of other services as well, namely metrics).
> >
> > We don't have an exact timeline for when we will build this support yet,
> but we will certainly announce it once we start actively working on it.
> >
> >
> > Erik Weathers <ew...@groupon.com> schrieb am Sa. 1. Juli 2017 um
> 09:45:
> > That works for our particular use case, and is effectively what *we* do,
> but renders storm a "strange bird" amongst mesos frameworks.  Is there no
> trickery that could be played with mesos roles and/or reservations?
> >
> > - Erik
> >
> > On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <di...@hellooperator.net>
> wrote:
> > If it _needs_ to be there always then I'd roll it out with whatever
> > automation you use to deploy the mesos workers ; depending on
> > the scale you're running at launching it as a task is likely to be less
> > reliable due to outages etc.
> >
> > ( I understand the 'maybe all hosts' constraint but if it's 'up to one
> per
> > host', it sounds like a CM issue to me. )
> >
> > On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
> > > hi Mesos folks!
> > >
> > > My team is largely responsible for maintaining the Storm-on-Mesos
> framework.
> > > It suffers from a problem related to log retrieval:  Storm has a
> process
> > > called the "logviewer" that is assumed to exist on every host, and the
> Storm
> > > UI provides links to contact this process to download logs (and other
> > > debugging artifacts).   Our team manually runs this process on each
> Mesos
> > > host, but it would be nice to launch it automatically onto any Mesos
> host
> > > where Storm work gets allocated. [0]
> > >
> > > I have read that Mesos has added support for Kubernetes-esque "pods"
> as of
> > > version 1.1.0, but that feature seems somewhat insufficient for
> implementing
> > > our desired behavior from my naive understanding.  Specifically, Storm
> only
> > > has support for connecting to 1 logviewer per host, so unless pods can
> have
> > > separate containers inside each pod [1], and also dynamically change
> the set
> > > of executors and tasks inside of the pod [2], then I don't see how
> we'd be
> > > able to use them.
> > >
> > > Is there any existing feature in Mesos that might help us accomplish
> our
> > > goal?  Or any upcoming features?
> > >
> > > Thanks!!
> > >
> > > - Erik
> > >
> > > [0] Thus the "all" in quotes in the subject of this email, because it
> > > *might* be all hosts, but it definitely would be all hosts where Storm
> gets
> > > work assigned.
> > >
> > > [1] The Storm-on-Mesos framework leverages separate containers for each
> > > topology's Supervisor and Worker processes, to provide isolation
> between
> > > topologies.
> > >
> > > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
> Workers
> > > (a Mesos Task) onto hosts is ever changing in a given instance of a
> > > Storm-on-Mesos framework.  i.e., as topologies get launched and die,
> or have
> > > their worker processes die, the processes are dynamically distributed
> to the
> > > various Mesos Worker hosts.  So existing containers often have more
> tasks
> > > assigned into them (thus growing their footprint) or removed from them
> (thus
> > > shrinking the footprint).
> > --
> > ~Kevin
>
>

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[James Peach <jo...@gmail.com>]

> On Jul 1, 2017, at 11:14 AM, Erik Weathers <ew...@groupon.com> wrote:
> 
> Thanks for the info Kevin.  Seems there's no JIRAs nor design docs floating about yet for "admin tasks" or "daemon sets".
> 
> Just FYI, this is the ticket in Storm for the problem I've been mentioning:
> 
> https://issues.apache.org/jira/browse/STORM-1342
> 
> I'll update it with the info you've provided below, so for now we'll rely on manually deploying logviewers.

ISTM that this should be possible with a smart framework. If the framework keeps track of which agents it gets offers for, it could ensure that it launches a Storm logviewer task on the agent before launching any other Storm tasks. I expect that it might be a little tricky to get the containerization right so that the Storm tasks can rendezvous with the logviewer, but in principle it could be made to work?

> 
> Thanks!
> 
> - Erik
> 
> On Sat, Jul 1, 2017 at 10:09 AM Kevin Klues <kl...@gmail.com> wrote:
> What you are describing is a feature we call 'admin tasks' or 'daemon sets'. 
> 
> Unfortunately, there is no direct support for these yet, but we do have plans in the (relatively) near future to start working on it.
> 
> One of our use cases is exactly what you describe with the logging service. On DC/OS we currently run our logging service as a systemd unit outside of mesos since we can't guarantee it gets launched everywhere (the same is true for a bunch of other services as well, namely metrics).
> 
> We don't have an exact timeline for when we will build this support yet, but we will certainly announce it once we start actively working on it.
> 
> 
> Erik Weathers <ew...@groupon.com> schrieb am Sa. 1. Juli 2017 um 09:45:
> That works for our particular use case, and is effectively what *we* do, but renders storm a "strange bird" amongst mesos frameworks.  Is there no trickery that could be played with mesos roles and/or reservations?
> 
> - Erik
> 
> On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <di...@hellooperator.net> wrote:
> If it _needs_ to be there always then I'd roll it out with whatever
> automation you use to deploy the mesos workers ; depending on
> the scale you're running at launching it as a task is likely to be less
> reliable due to outages etc.
> 
> ( I understand the 'maybe all hosts' constraint but if it's 'up to one per
> host', it sounds like a CM issue to me. )
> 
> On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
> > hi Mesos folks!
> >
> > My team is largely responsible for maintaining the Storm-on-Mesos framework.
> > It suffers from a problem related to log retrieval:  Storm has a process
> > called the "logviewer" that is assumed to exist on every host, and the Storm
> > UI provides links to contact this process to download logs (and other
> > debugging artifacts).   Our team manually runs this process on each Mesos
> > host, but it would be nice to launch it automatically onto any Mesos host
> > where Storm work gets allocated. [0]
> >
> > I have read that Mesos has added support for Kubernetes-esque "pods" as of
> > version 1.1.0, but that feature seems somewhat insufficient for implementing
> > our desired behavior from my naive understanding.  Specifically, Storm only
> > has support for connecting to 1 logviewer per host, so unless pods can have
> > separate containers inside each pod [1], and also dynamically change the set
> > of executors and tasks inside of the pod [2], then I don't see how we'd be
> > able to use them.
> >
> > Is there any existing feature in Mesos that might help us accomplish our
> > goal?  Or any upcoming features?
> >
> > Thanks!!
> >
> > - Erik
> >
> > [0] Thus the "all" in quotes in the subject of this email, because it
> > *might* be all hosts, but it definitely would be all hosts where Storm gets
> > work assigned.
> >
> > [1] The Storm-on-Mesos framework leverages separate containers for each
> > topology's Supervisor and Worker processes, to provide isolation between
> > topologies.
> >
> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm Workers
> > (a Mesos Task) onto hosts is ever changing in a given instance of a
> > Storm-on-Mesos framework.  i.e., as topologies get launched and die, or have
> > their worker processes die, the processes are dynamically distributed to the
> > various Mesos Worker hosts.  So existing containers often have more tasks
> > assigned into them (thus growing their footprint) or removed from them (thus
> > shrinking the footprint).
> -- 
> ~Kevin

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers <ew...@groupon.com>]

Thanks for the info Kevin.  Seems there's no JIRAs nor design docs floating
about yet for "admin tasks" or "daemon sets".

Just FYI, this is the ticket in Storm for the problem I've been mentioning:

https://issues.apache.org/jira/browse/STORM-1342

I'll update it with the info you've provided below, so for now we'll rely
on manually deploying logviewers.

Thanks!

- Erik

On Sat, Jul 1, 2017 at 10:09 AM Kevin Klues <kl...@gmail.com> wrote:

> What you are describing is a feature we call 'admin tasks' or 'daemon
> sets'.
>
> Unfortunately, there is no direct support for these yet, but we do have
> plans in the (relatively) near future to start working on it.
>
> One of our use cases is exactly what you describe with the logging
> service. On DC/OS we currently run our logging service as a systemd unit
> outside of mesos since we can't guarantee it gets launched everywhere (the
> same is true for a bunch of other services as well, namely metrics).
>
> We don't have an exact timeline for when we will build this support yet,
> but we will certainly announce it once we start actively working on it.
>
>
> Erik Weathers <ew...@groupon.com> schrieb am Sa. 1. Juli 2017 um
> 09:45:
>
>> That works for our particular use case, and is effectively what *we* do,
>> but renders storm a "strange bird" amongst mesos frameworks.  Is there no
>> trickery that could be played with mesos roles and/or reservations?
>>
>> - Erik
>>
>> On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <di...@hellooperator.net>
>> wrote:
>>
>>> If it _needs_ to be there always then I'd roll it out with whatever
>>> automation you use to deploy the mesos workers ; depending on
>>> the scale you're running at launching it as a task is likely to be less
>>> reliable due to outages etc.
>>>
>>> ( I understand the 'maybe all hosts' constraint but if it's 'up to one
>>> per
>>> host', it sounds like a CM issue to me. )
>>>
>>> On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
>>> > hi Mesos folks!
>>> >
>>> > My team is largely responsible for maintaining the Storm-on-Mesos
>>> framework.
>>> > It suffers from a problem related to log retrieval:  Storm has a
>>> process
>>> > called the "logviewer" that is assumed to exist on every host, and the
>>> Storm
>>> > UI provides links to contact this process to download logs (and other
>>> > debugging artifacts).   Our team manually runs this process on each
>>> Mesos
>>> > host, but it would be nice to launch it automatically onto any Mesos
>>> host
>>> > where Storm work gets allocated. [0]
>>> >
>>> > I have read that Mesos has added support for Kubernetes-esque "pods"
>>> as of
>>> > version 1.1.0, but that feature seems somewhat insufficient for
>>> implementing
>>> > our desired behavior from my naive understanding.  Specifically, Storm
>>> only
>>> > has support for connecting to 1 logviewer per host, so unless pods can
>>> have
>>> > separate containers inside each pod [1], and also dynamically change
>>> the set
>>> > of executors and tasks inside of the pod [2], then I don't see how
>>> we'd be
>>> > able to use them.
>>> >
>>> > Is there any existing feature in Mesos that might help us accomplish
>>> our
>>> > goal?  Or any upcoming features?
>>> >
>>> > Thanks!!
>>> >
>>> > - Erik
>>> >
>>> > [0] Thus the "all" in quotes in the subject of this email, because it
>>> > *might* be all hosts, but it definitely would be all hosts where Storm
>>> gets
>>> > work assigned.
>>> >
>>> > [1] The Storm-on-Mesos framework leverages separate containers for each
>>> > topology's Supervisor and Worker processes, to provide isolation
>>> between
>>> > topologies.
>>> >
>>> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
>>> Workers
>>> > (a Mesos Task) onto hosts is ever changing in a given instance of a
>>> > Storm-on-Mesos framework.  i.e., as topologies get launched and die,
>>> or have
>>> > their worker processes die, the processes are dynamically distributed
>>> to the
>>> > various Mesos Worker hosts.  So existing containers often have more
>>> tasks
>>> > assigned into them (thus growing their footprint) or removed from them
>>> (thus
>>> > shrinking the footprint).
>>>
>> --
> ~Kevin
>

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Kevin Klues <kl...@gmail.com>]

What you are describing is a feature we call 'admin tasks' or 'daemon
sets'.

Unfortunately, there is no direct support for these yet, but we do have
plans in the (relatively) near future to start working on it.

One of our use cases is exactly what you describe with the logging service.
On DC/OS we currently run our logging service as a systemd unit outside of
mesos since we can't guarantee it gets launched everywhere (the same is
true for a bunch of other services as well, namely metrics).

We don't have an exact timeline for when we will build this support yet,
but we will certainly announce it once we start actively working on it.


Erik Weathers <ew...@groupon.com> schrieb am Sa. 1. Juli 2017 um 09:45:

> That works for our particular use case, and is effectively what *we* do,
> but renders storm a "strange bird" amongst mesos frameworks.  Is there no
> trickery that could be played with mesos roles and/or reservations?
>
> - Erik
>
> On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <di...@hellooperator.net> wrote:
>
>> If it _needs_ to be there always then I'd roll it out with whatever
>> automation you use to deploy the mesos workers ; depending on
>> the scale you're running at launching it as a task is likely to be less
>> reliable due to outages etc.
>>
>> ( I understand the 'maybe all hosts' constraint but if it's 'up to one per
>> host', it sounds like a CM issue to me. )
>>
>> On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
>> > hi Mesos folks!
>> >
>> > My team is largely responsible for maintaining the Storm-on-Mesos
>> framework.
>> > It suffers from a problem related to log retrieval:  Storm has a process
>> > called the "logviewer" that is assumed to exist on every host, and the
>> Storm
>> > UI provides links to contact this process to download logs (and other
>> > debugging artifacts).   Our team manually runs this process on each
>> Mesos
>> > host, but it would be nice to launch it automatically onto any Mesos
>> host
>> > where Storm work gets allocated. [0]
>> >
>> > I have read that Mesos has added support for Kubernetes-esque "pods" as
>> of
>> > version 1.1.0, but that feature seems somewhat insufficient for
>> implementing
>> > our desired behavior from my naive understanding.  Specifically, Storm
>> only
>> > has support for connecting to 1 logviewer per host, so unless pods can
>> have
>> > separate containers inside each pod [1], and also dynamically change
>> the set
>> > of executors and tasks inside of the pod [2], then I don't see how we'd
>> be
>> > able to use them.
>> >
>> > Is there any existing feature in Mesos that might help us accomplish our
>> > goal?  Or any upcoming features?
>> >
>> > Thanks!!
>> >
>> > - Erik
>> >
>> > [0] Thus the "all" in quotes in the subject of this email, because it
>> > *might* be all hosts, but it definitely would be all hosts where Storm
>> gets
>> > work assigned.
>> >
>> > [1] The Storm-on-Mesos framework leverages separate containers for each
>> > topology's Supervisor and Worker processes, to provide isolation between
>> > topologies.
>> >
>> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
>> Workers
>> > (a Mesos Task) onto hosts is ever changing in a given instance of a
>> > Storm-on-Mesos framework.  i.e., as topologies get launched and die, or
>> have
>> > their worker processes die, the processes are dynamically distributed
>> to the
>> > various Mesos Worker hosts.  So existing containers often have more
>> tasks
>> > assigned into them (thus growing their footprint) or removed from them
>> (thus
>> > shrinking the footprint).
>>
> --
~Kevin

Re: ensuring a particular task is deployed to "all" Mesos Worker hosts

[Erik Weathers <ew...@groupon.com>]

That works for our particular use case, and is effectively what *we* do,
but renders storm a "strange bird" amongst mesos frameworks.  Is there no
trickery that could be played with mesos roles and/or reservations?

- Erik

On Sat, Jul 1, 2017 at 3:57 AM Dick Davies <di...@hellooperator.net> wrote:

> If it _needs_ to be there always then I'd roll it out with whatever
> automation you use to deploy the mesos workers ; depending on
> the scale you're running at launching it as a task is likely to be less
> reliable due to outages etc.
>
> ( I understand the 'maybe all hosts' constraint but if it's 'up to one per
> host', it sounds like a CM issue to me. )
>
> On 30 June 2017 at 23:58, Erik Weathers <ew...@groupon.com> wrote:
> > hi Mesos folks!
> >
> > My team is largely responsible for maintaining the Storm-on-Mesos
> framework.
> > It suffers from a problem related to log retrieval:  Storm has a process
> > called the "logviewer" that is assumed to exist on every host, and the
> Storm
> > UI provides links to contact this process to download logs (and other
> > debugging artifacts).   Our team manually runs this process on each Mesos
> > host, but it would be nice to launch it automatically onto any Mesos host
> > where Storm work gets allocated. [0]
> >
> > I have read that Mesos has added support for Kubernetes-esque "pods" as
> of
> > version 1.1.0, but that feature seems somewhat insufficient for
> implementing
> > our desired behavior from my naive understanding.  Specifically, Storm
> only
> > has support for connecting to 1 logviewer per host, so unless pods can
> have
> > separate containers inside each pod [1], and also dynamically change the
> set
> > of executors and tasks inside of the pod [2], then I don't see how we'd
> be
> > able to use them.
> >
> > Is there any existing feature in Mesos that might help us accomplish our
> > goal?  Or any upcoming features?
> >
> > Thanks!!
> >
> > - Erik
> >
> > [0] Thus the "all" in quotes in the subject of this email, because it
> > *might* be all hosts, but it definitely would be all hosts where Storm
> gets
> > work assigned.
> >
> > [1] The Storm-on-Mesos framework leverages separate containers for each
> > topology's Supervisor and Worker processes, to provide isolation between
> > topologies.
> >
> > [2] The assignment of Storm Supervisors (a Mesos Executor) + Storm
> Workers
> > (a Mesos Task) onto hosts is ever changing in a given instance of a
> > Storm-on-Mesos framework.  i.e., as topologies get launched and die, or
> have
> > their worker processes die, the processes are dynamically distributed to
> the
> > various Mesos Worker hosts.  So existing containers often have more tasks
> > assigned into them (thus growing their footprint) or removed from them
> (thus
> > shrinking the footprint).
>