Re: [proposal] Module extension: hooks

[Benjamin Mahler <be...@gmail.com>]

> That aside, our use-case involves hanging meta-data off the task with
> labels which we cannot do with an event stream alone.
> The metadata we need is produced by a 3rd party security infrastructure
> which we invoke and use when setting up the executor environment in the
> slave.

Consulting the 3rd party security infrastructure synchronously in the
launch path sounds expensive..?

I still have a hard time understanding the use case. Could this all just be
implemented as a security isolator that consults the 3rd party security
infrastructure? Or a containerizer that wraps our containerizer with the
additional security requirements?

On Tue, Nov 25, 2014 at 5:25 PM, Niklas Nielsen <ni...@mesosphere.io>
wrote:

> Hi again,
>
> (Sorry for the copy paste, but the discussion has been going on in both
> JIRA and this thread)
>
> Just had a chat with BenH and Kapil, and think that we came up with a safer
> solution which would fit our needs (at the cost of some general
> flexibility).
>
> Here is the idea; instead of thinking about generic hooks which can change
> arbitrary fields in task infos (which may cause severe problems if, say
> resource fields are changed) we care about being able to
> 1) Have call-backs be invoked and
> 2) Tag tasks with meta-data in labels at several points during the task
> life-cycle.
> So instead of thinking about generic hooks, we wanted to talk about "Task
> Decorators" instead.
>
> Labels labels = LaunchTaskDecorators(task, framework, ...);
> task.mutable_labels()->CopyFrom(labels);
>
> Only const references to, say TaskInfo, FrameworkInfo and so forth is
> provided to the decorator.
> So the task processing remains untouched and only labels (which are not
> interpreted by Mesos) can be modified.
>
> We also need to be able to hang off data in the executor environments
> which, in turn, can be done with "Environment Decorators".
>
> The term "Decorators" was just our first thought on the name of the
> concept, but that is still open for discussion.
>
> On 25 November 2014 at 15:41, Niklas Nielsen <ni...@mesosphere.io> wrote:
>
> >
> >
> > On 24 November 2014 at 12:24, Vinod Kone <vi...@gmail.com> wrote:
> >
> >> On Fri, Nov 21, 2014 at 4:56 PM, Niklas Nielsen <ni...@mesosphere.io>
> >> wrote:
> >>
> >> > That aside, our use-case involves hanging meta-data off the task with
> >> > labels which we cannot do with an event stream alone.
> >> > The metadata we need is produced by a 3rd party security
> infrastructure
> >> > which we invoke and use when setting up the executor environment in
> the
> >> > slave.
> >> > We actually only need the pre hook / filter mechanism to do this, but
> >> > wanted to come up with a generalized solution.
> >> >
> >>
> >> I still don't quite understand the use case. Why can't a framework(s)
> talk
> >> to the security infrastructure and setup the TaskInfo appropriately and
> >> opaquely to master? It sounds like the executor(s) need to be modified
> to
> >> understand the updated security information anyway.
> >>
> >
> > The frameworks themselves doesn't not have permission to do that and it
> > needs to work transparently across frameworks that we may not
> know/control
> > how constructs task infos.
> > You are right that hooks need to be on both sides (on masters during task
> > launch and on slaves during executor launch).
> >
> >
>

Re: [proposal] Module extension: hooks

[Benjamin Hindman <be...@eecs.berkeley.edu>]

>
> Why can only the master obtain the information?
>

I think it's fair to say that an organization might prefer to limit the
exposure of certain security credentials to fewer machines, such as just
the masters.


> My impression is that because these constraints are rigid, we are forced to
> have a complex solution in mesos.
>

This is a pretty minimal addition to the Mesos masters, and generally, I
think it opens up the ability to do interesting label decorating beyond
just for security. In particular, I'd love to see someone use this to
perform "dtrace" like functionality for tasks that flow through a cluster!


> Introducing asynchronous blocks makes the master much harder to reason
> about. Couldn't this hook just be at the message level? This way, the
> asynchronous implementation is really easy to reason about.
>

We could move the decorator from after task validation to before the
launchTask install'ed handler even gets invoked, but I don't see the upside
there, since we'll still want to do this asynchronously at some point.

Two important points here:

* Decorators/hooks will be enabled on a per organization/cluster basis.
Most users will see no impact here.
* Like modules and isolators, this gives us the opportunity to try out
functionality that at some point we might want to move directly into Mesos
without having to make those changes up front. This is a huge win from a
productivity standpoint.


On Mon, Dec 1, 2014 at 3:24 PM, Niklas Nielsen <ni...@mesosphere.io> wrote:
>
> > The weight of the labels transformation hasn't been a problem so far.
> > In a second pass, we can put an upper limit (in seconds) to the
> > hook/decorator and do it asynchronously.
> > That will free the master actor up too.
> >
> > If the slaves had permissions to get hold of the meta data themselves,
> then
> > yes; we could have done another containerizer (and get it in without
> "any"
> > changes with containerizer modules - TBD).
> > However, only the master nodes can generate the meta-data that we need to
> > track the tasks with. So that alone isn't enough.
> >
> > Niklas
> >
> >
> > On 1 December 2014 at 12:46, Benjamin Mahler <be...@gmail.com>
> > wrote:
> >
> > > > That aside, our use-case involves hanging meta-data off the task with
> > > > labels which we cannot do with an event stream alone.
> > > > The metadata we need is produced by a 3rd party security
> infrastructure
> > > > which we invoke and use when setting up the executor environment in
> the
> > > > slave.
> > >
> > > Consulting the 3rd party security infrastructure synchronously in the
> > > launch path sounds expensive..?
> > >
> > > I still have a hard time understanding the use case. Could this all
> just
> > be
> > > implemented as a security isolator that consults the 3rd party security
> > > infrastructure? Or a containerizer that wraps our containerizer with
> the
> > > additional security requirements?
> > >
> > > On Tue, Nov 25, 2014 at 5:25 PM, Niklas Nielsen <ni...@mesosphere.io>
> > > wrote:
> > >
> > > > Hi again,
> > > >
> > > > (Sorry for the copy paste, but the discussion has been going on in
> both
> > > > JIRA and this thread)
> > > >
> > > > Just had a chat with BenH and Kapil, and think that we came up with a
> > > safer
> > > > solution which would fit our needs (at the cost of some general
> > > > flexibility).
> > > >
> > > > Here is the idea; instead of thinking about generic hooks which can
> > > change
> > > > arbitrary fields in task infos (which may cause severe problems if,
> say
> > > > resource fields are changed) we care about being able to
> > > > 1) Have call-backs be invoked and
> > > > 2) Tag tasks with meta-data in labels at several points during the
> task
> > > > life-cycle.
> > > > So instead of thinking about generic hooks, we wanted to talk about
> > "Task
> > > > Decorators" instead.
> > > >
> > > > Labels labels = LaunchTaskDecorators(task, framework, ...);
> > > > task.mutable_labels()->CopyFrom(labels);
> > > >
> > > > Only const references to, say TaskInfo, FrameworkInfo and so forth is
> > > > provided to the decorator.
> > > > So the task processing remains untouched and only labels (which are
> not
> > > > interpreted by Mesos) can be modified.
> > > >
> > > > We also need to be able to hang off data in the executor environments
> > > > which, in turn, can be done with "Environment Decorators".
> > > >
> > > > The term "Decorators" was just our first thought on the name of the
> > > > concept, but that is still open for discussion.
> > > >
> > > > On 25 November 2014 at 15:41, Niklas Nielsen <ni...@mesosphere.io>
> > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On 24 November 2014 at 12:24, Vinod Kone <vi...@gmail.com>
> > wrote:
> > > > >
> > > > >> On Fri, Nov 21, 2014 at 4:56 PM, Niklas Nielsen <
> > niklas@mesosphere.io
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > That aside, our use-case involves hanging meta-data off the task
> > > with
> > > > >> > labels which we cannot do with an event stream alone.
> > > > >> > The metadata we need is produced by a 3rd party security
> > > > infrastructure
> > > > >> > which we invoke and use when setting up the executor environment
> > in
> > > > the
> > > > >> > slave.
> > > > >> > We actually only need the pre hook / filter mechanism to do
> this,
> > > but
> > > > >> > wanted to come up with a generalized solution.
> > > > >> >
> > > > >>
> > > > >> I still don't quite understand the use case. Why can't a
> > framework(s)
> > > > talk
> > > > >> to the security infrastructure and setup the TaskInfo
> appropriately
> > > and
> > > > >> opaquely to master? It sounds like the executor(s) need to be
> > modified
> > > > to
> > > > >> understand the updated security information anyway.
> > > > >>
> > > > >
> > > > > The frameworks themselves doesn't not have permission to do that
> and
> > it
> > > > > needs to work transparently across frameworks that we may not
> > > > know/control
> > > > > how constructs task infos.
> > > > > You are right that hooks need to be on both sides (on masters
> during
> > > task
> > > > > launch and on slaves during executor launch).
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [proposal] Module extension: hooks

[Benjamin Mahler <be...@gmail.com>]

> However, only the master nodes can generate the meta-data that we need to track
the tasks with. So that alone isn't enough.

Why can only the master obtain the information?

I can understand making a distinction between mesos and the frameworks in
terms of security. The distinction between master and slave security
permissions you mentioned I don't understand. If anything, shouldn't the
master be more restricted security-wise? The slave is the one that needs to
be doing execution under various user accounts, that seems to be where the
permissions to talk to the relevant 3rd party security infrastructure
should lie, no?

My impression is that because these constraints are rigid, we are forced to
have a complex solution in mesos.

> we can put an upper limit (in seconds) to the hook/decorator and do it
asynchronously.

Introducing asynchronous blocks makes the master much harder to reason
about. Couldn't this hook just be at the message level? This way, the
asynchronous implementation is really easy to reason about.

On Mon, Dec 1, 2014 at 3:24 PM, Niklas Nielsen <ni...@mesosphere.io> wrote:

> The weight of the labels transformation hasn't been a problem so far.
> In a second pass, we can put an upper limit (in seconds) to the
> hook/decorator and do it asynchronously.
> That will free the master actor up too.
>
> If the slaves had permissions to get hold of the meta data themselves, then
> yes; we could have done another containerizer (and get it in without "any"
> changes with containerizer modules - TBD).
> However, only the master nodes can generate the meta-data that we need to
> track the tasks with. So that alone isn't enough.
>
> Niklas
>
>
> On 1 December 2014 at 12:46, Benjamin Mahler <be...@gmail.com>
> wrote:
>
> > > That aside, our use-case involves hanging meta-data off the task with
> > > labels which we cannot do with an event stream alone.
> > > The metadata we need is produced by a 3rd party security infrastructure
> > > which we invoke and use when setting up the executor environment in the
> > > slave.
> >
> > Consulting the 3rd party security infrastructure synchronously in the
> > launch path sounds expensive..?
> >
> > I still have a hard time understanding the use case. Could this all just
> be
> > implemented as a security isolator that consults the 3rd party security
> > infrastructure? Or a containerizer that wraps our containerizer with the
> > additional security requirements?
> >
> > On Tue, Nov 25, 2014 at 5:25 PM, Niklas Nielsen <ni...@mesosphere.io>
> > wrote:
> >
> > > Hi again,
> > >
> > > (Sorry for the copy paste, but the discussion has been going on in both
> > > JIRA and this thread)
> > >
> > > Just had a chat with BenH and Kapil, and think that we came up with a
> > safer
> > > solution which would fit our needs (at the cost of some general
> > > flexibility).
> > >
> > > Here is the idea; instead of thinking about generic hooks which can
> > change
> > > arbitrary fields in task infos (which may cause severe problems if, say
> > > resource fields are changed) we care about being able to
> > > 1) Have call-backs be invoked and
> > > 2) Tag tasks with meta-data in labels at several points during the task
> > > life-cycle.
> > > So instead of thinking about generic hooks, we wanted to talk about
> "Task
> > > Decorators" instead.
> > >
> > > Labels labels = LaunchTaskDecorators(task, framework, ...);
> > > task.mutable_labels()->CopyFrom(labels);
> > >
> > > Only const references to, say TaskInfo, FrameworkInfo and so forth is
> > > provided to the decorator.
> > > So the task processing remains untouched and only labels (which are not
> > > interpreted by Mesos) can be modified.
> > >
> > > We also need to be able to hang off data in the executor environments
> > > which, in turn, can be done with "Environment Decorators".
> > >
> > > The term "Decorators" was just our first thought on the name of the
> > > concept, but that is still open for discussion.
> > >
> > > On 25 November 2014 at 15:41, Niklas Nielsen <ni...@mesosphere.io>
> > wrote:
> > >
> > > >
> > > >
> > > > On 24 November 2014 at 12:24, Vinod Kone <vi...@gmail.com>
> wrote:
> > > >
> > > >> On Fri, Nov 21, 2014 at 4:56 PM, Niklas Nielsen <
> niklas@mesosphere.io
> > >
> > > >> wrote:
> > > >>
> > > >> > That aside, our use-case involves hanging meta-data off the task
> > with
> > > >> > labels which we cannot do with an event stream alone.
> > > >> > The metadata we need is produced by a 3rd party security
> > > infrastructure
> > > >> > which we invoke and use when setting up the executor environment
> in
> > > the
> > > >> > slave.
> > > >> > We actually only need the pre hook / filter mechanism to do this,
> > but
> > > >> > wanted to come up with a generalized solution.
> > > >> >
> > > >>
> > > >> I still don't quite understand the use case. Why can't a
> framework(s)
> > > talk
> > > >> to the security infrastructure and setup the TaskInfo appropriately
> > and
> > > >> opaquely to master? It sounds like the executor(s) need to be
> modified
> > > to
> > > >> understand the updated security information anyway.
> > > >>
> > > >
> > > > The frameworks themselves doesn't not have permission to do that and
> it
> > > > needs to work transparently across frameworks that we may not
> > > know/control
> > > > how constructs task infos.
> > > > You are right that hooks need to be on both sides (on masters during
> > task
> > > > launch and on slaves during executor launch).
> > > >
> > > >
> > >
> >
>

Re: [proposal] Module extension: hooks

[Niklas Nielsen <ni...@mesosphere.io>]

The weight of the labels transformation hasn't been a problem so far.
In a second pass, we can put an upper limit (in seconds) to the
hook/decorator and do it asynchronously.
That will free the master actor up too.

If the slaves had permissions to get hold of the meta data themselves, then
yes; we could have done another containerizer (and get it in without "any"
changes with containerizer modules - TBD).
However, only the master nodes can generate the meta-data that we need to
track the tasks with. So that alone isn't enough.

Niklas


On 1 December 2014 at 12:46, Benjamin Mahler <be...@gmail.com>
wrote:

> > That aside, our use-case involves hanging meta-data off the task with
> > labels which we cannot do with an event stream alone.
> > The metadata we need is produced by a 3rd party security infrastructure
> > which we invoke and use when setting up the executor environment in the
> > slave.
>
> Consulting the 3rd party security infrastructure synchronously in the
> launch path sounds expensive..?
>
> I still have a hard time understanding the use case. Could this all just be
> implemented as a security isolator that consults the 3rd party security
> infrastructure? Or a containerizer that wraps our containerizer with the
> additional security requirements?
>
> On Tue, Nov 25, 2014 at 5:25 PM, Niklas Nielsen <ni...@mesosphere.io>
> wrote:
>
> > Hi again,
> >
> > (Sorry for the copy paste, but the discussion has been going on in both
> > JIRA and this thread)
> >
> > Just had a chat with BenH and Kapil, and think that we came up with a
> safer
> > solution which would fit our needs (at the cost of some general
> > flexibility).
> >
> > Here is the idea; instead of thinking about generic hooks which can
> change
> > arbitrary fields in task infos (which may cause severe problems if, say
> > resource fields are changed) we care about being able to
> > 1) Have call-backs be invoked and
> > 2) Tag tasks with meta-data in labels at several points during the task
> > life-cycle.
> > So instead of thinking about generic hooks, we wanted to talk about "Task
> > Decorators" instead.
> >
> > Labels labels = LaunchTaskDecorators(task, framework, ...);
> > task.mutable_labels()->CopyFrom(labels);
> >
> > Only const references to, say TaskInfo, FrameworkInfo and so forth is
> > provided to the decorator.
> > So the task processing remains untouched and only labels (which are not
> > interpreted by Mesos) can be modified.
> >
> > We also need to be able to hang off data in the executor environments
> > which, in turn, can be done with "Environment Decorators".
> >
> > The term "Decorators" was just our first thought on the name of the
> > concept, but that is still open for discussion.
> >
> > On 25 November 2014 at 15:41, Niklas Nielsen <ni...@mesosphere.io>
> wrote:
> >
> > >
> > >
> > > On 24 November 2014 at 12:24, Vinod Kone <vi...@gmail.com> wrote:
> > >
> > >> On Fri, Nov 21, 2014 at 4:56 PM, Niklas Nielsen <niklas@mesosphere.io
> >
> > >> wrote:
> > >>
> > >> > That aside, our use-case involves hanging meta-data off the task
> with
> > >> > labels which we cannot do with an event stream alone.
> > >> > The metadata we need is produced by a 3rd party security
> > infrastructure
> > >> > which we invoke and use when setting up the executor environment in
> > the
> > >> > slave.
> > >> > We actually only need the pre hook / filter mechanism to do this,
> but
> > >> > wanted to come up with a generalized solution.
> > >> >
> > >>
> > >> I still don't quite understand the use case. Why can't a framework(s)
> > talk
> > >> to the security infrastructure and setup the TaskInfo appropriately
> and
> > >> opaquely to master? It sounds like the executor(s) need to be modified
> > to
> > >> understand the updated security information anyway.
> > >>
> > >
> > > The frameworks themselves doesn't not have permission to do that and it
> > > needs to work transparently across frameworks that we may not
> > know/control
> > > how constructs task infos.
> > > You are right that hooks need to be on both sides (on masters during
> task
> > > launch and on slaves during executor launch).
> > >
> > >
> >
>