You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Zameer Manji <zm...@apache.org> on 2016/11/02 20:49:02 UTC
Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo
Joris,
You make a good point. However, I'm not convinced that `CommandInfo` should
be the well defined construct that people use. Can you please describe
different custom executors, and the overlap between them and how
CommandInfo will reduce that overlap? I'm having a hard time seeing where
CommandInfo will solve all of their cases.
Consider the cause of Thermos (Aurora's Executor), it could never use a
`CommandInfo` struct because it executes a processes graph instead of a
single command.
If the project wants to go down this path, I think generalizing
`CommandInfo` that could capture more cases (ie multiple commands or a
graph of commands) would be a better first step.
What do you think?
On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:
> I do think it would be valuable to have a more well defined contract
> between frameworks and custom executors.
>
> As Zameer pointed out a specific framework and accompanying custom executor
> can decide to do that in the data bytes; however, if we started building
> out a few different flavors of executors then it would be great for there
> to be standard way to pass command information to them.
>
> The current model works well in a 1-1 mapping between framework and
> executor binaries. In a world where that is 1-N it means all N executors
> have to use the same method of passing the command.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji <zm...@apache.org> wrote:
>
> > I'm not convinced this is a valid use case.
> >
> > Mesos is supposed to be a generic kernel for launching "tasks", whatever
> > they might be.
> >
> > In some cases it is useful to launch an executable, in other cases it
> might
> > be useful to launch a series of executables, and in some other cases it
> > might be useful to spawn a thread to do some work. Whatever that might
> be,
> > it doesn't matter to Mesos and the executor and framework are free to
> > establish a contract in `ExecutorInfo.data`, completely independent of
> the
> > Mesos API.
> >
> > I think formalizing this contract between executors and frameworks via
> > CommandInfo is going to introduce more problems than what they solve. If
> > the CommandInfo struct is useful, frameworks and executors can just stuff
> > that into ExecutorInfo.data, however it's not something that they need to
> > adhere too.
> >
> > What's the underlying motivation for this?
> >
> >
> >
> > On Thu, Oct 13, 2016 at 10:40 AM, haosdent <ha...@gmail.com> wrote:
> >
> > > For command task, if its `ExecutorInfo` would set with
> `CommandExecutor`
> > as
> > > well?
> > >
> > > Some tickets may relate to this.
> > >
> > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > >
> > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone <vi...@apache.org>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are contemplating whether to allow both CommandInfo and
> ExecutorInfo
> > > on
> > > > TaskInfo (MESOS-6294 <https://issues.apache.org/
> jira/browse/MESOS-6294
> > > >).
> > > > Currently we only allow one or the other. The motivation is to allow
> > > custom
> > > > executors a more structured way to pass information (e.g, command)
> > about
> > > > Task. Right now custom executors have to get this data via
> > > `TaskInfo.bytes`
> > > > which is not ideal.
> > > >
> > > > Are there any custom executors out there that crash if they get Tasks
> > > with
> > > > CommandInfo set?
> > > >
> > > > Thoughts?
> > > >
> > > > Vinod
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Haosdent Huang
> > >
> > > --
> > > Zameer Manji
> > >
> >
>
> --
> Zameer Manji
>
Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo
Posted by David Greenberg <ds...@gmail.com>.
I have worked with executors that perform both conditional & unconditional
execution of a process graph, configurable by the framework. I think that
it would be hard to standardize.
On Wed, Nov 2, 2016 at 2:49 PM Zameer Manji <zm...@apache.org> wrote:
> Joris,
>
> You make a good point. However, I'm not convinced that `CommandInfo` should
> be the well defined construct that people use. Can you please describe
> different custom executors, and the overlap between them and how
> CommandInfo will reduce that overlap? I'm having a hard time seeing where
> CommandInfo will solve all of their cases.
>
> Consider the cause of Thermos (Aurora's Executor), it could never use a
> `CommandInfo` struct because it executes a processes graph instead of a
> single command.
>
> If the project wants to go down this path, I think generalizing
> `CommandInfo` that could capture more cases (ie multiple commands or a
> graph of commands) would be a better first step.
>
> What do you think?
>
> On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> joris@mesosphere.io>
> wrote:
>
> > I do think it would be valuable to have a more well defined contract
> > between frameworks and custom executors.
> >
> > As Zameer pointed out a specific framework and accompanying custom
> executor
> > can decide to do that in the data bytes; however, if we started building
> > out a few different flavors of executors then it would be great for there
> > to be standard way to pass command information to them.
> >
> > The current model works well in a 1-1 mapping between framework and
> > executor binaries. In a world where that is 1-N it means all N executors
> > have to use the same method of passing the command.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji <zm...@apache.org> wrote:
> >
> > > I'm not convinced this is a valid use case.
> > >
> > > Mesos is supposed to be a generic kernel for launching "tasks",
> whatever
> > > they might be.
> > >
> > > In some cases it is useful to launch an executable, in other cases it
> > might
> > > be useful to launch a series of executables, and in some other cases it
> > > might be useful to spawn a thread to do some work. Whatever that might
> > be,
> > > it doesn't matter to Mesos and the executor and framework are free to
> > > establish a contract in `ExecutorInfo.data`, completely independent of
> > the
> > > Mesos API.
> > >
> > > I think formalizing this contract between executors and frameworks via
> > > CommandInfo is going to introduce more problems than what they solve.
> If
> > > the CommandInfo struct is useful, frameworks and executors can just
> stuff
> > > that into ExecutorInfo.data, however it's not something that they need
> to
> > > adhere too.
> > >
> > > What's the underlying motivation for this?
> > >
> > >
> > >
> > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent <ha...@gmail.com> wrote:
> > >
> > > > For command task, if its `ExecutorInfo` would set with
> > `CommandExecutor`
> > > as
> > > > well?
> > > >
> > > > Some tickets may relate to this.
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > >
> > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are contemplating whether to allow both CommandInfo and
> > ExecutorInfo
> > > > on
> > > > > TaskInfo (MESOS-6294 <https://issues.apache.org/
> > jira/browse/MESOS-6294
> > > > >).
> > > > > Currently we only allow one or the other. The motivation is to
> allow
> > > > custom
> > > > > executors a more structured way to pass information (e.g, command)
> > > about
> > > > > Task. Right now custom executors have to get this data via
> > > > `TaskInfo.bytes`
> > > > > which is not ideal.
> > > > >
> > > > > Are there any custom executors out there that crash if they get
> Tasks
> > > > with
> > > > > CommandInfo set?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vinod
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > > > --
> > > > Zameer Manji
> > > >
> > >
> >
> > --
> > Zameer Manji
> >
>
Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo
Posted by Zameer Manji <zm...@apache.org>.
It isn't issue if it a broadening of possibilities.
I would like to point out that, if we do want to have a better contract in
the future (ie a set of CommandInfos), we are setting up ourselves for a
API change in the future.
If we clearly document that the CommandInfo is just passed down to the
executor by the master and agent, I don't see any harm.
On Fri, Nov 4, 2016 at 6:53 AM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:
> @zameer
>
> I think your example makes a lot of sense. I didn't interpret the proposal
> as one that would prevent your case.
>
> The way I read the proposal is that we want to allow setting both, not
> require it. If I misunderstood then please ignore my comments.
>
> When I first heard about this proposal it seemed like a nice way for
> frameworks and executors to start defining a more structured contract _if
> they wanted to_, while still allowing others to keep passing their (from
> Mesos's view) unstructured data as long as both sides agree on the
> serialization / de-serialization.
>
> Does this still seem like an issue if it's not a requirement but just a
> broadening of possibilities?
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji <zm...@apache.org> wrote:
>
> > Joris,
> >
> > You make a good point. However, I'm not convinced that `CommandInfo`
> should
> > be the well defined construct that people use. Can you please describe
> > different custom executors, and the overlap between them and how
> > CommandInfo will reduce that overlap? I'm having a hard time seeing where
> > CommandInfo will solve all of their cases.
> >
> > Consider the cause of Thermos (Aurora's Executor), it could never use a
> > `CommandInfo` struct because it executes a processes graph instead of a
> > single command.
> >
> > If the project wants to go down this path, I think generalizing
> > `CommandInfo` that could capture more cases (ie multiple commands or a
> > graph of commands) would be a better first step.
> >
> > What do you think?
> >
> > On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> > joris@mesosphere.io>
> > wrote:
> >
> > > I do think it would be valuable to have a more well defined contract
> > > between frameworks and custom executors.
> > >
> > > As Zameer pointed out a specific framework and accompanying custom
> > executor
> > > can decide to do that in the data bytes; however, if we started
> building
> > > out a few different flavors of executors then it would be great for
> there
> > > to be standard way to pass command information to them.
> > >
> > > The current model works well in a 1-1 mapping between framework and
> > > executor binaries. In a world where that is 1-N it means all N
> executors
> > > have to use the same method of passing the command.
> > >
> > > —
> > > *Joris Van Remoortere*
> > > Mesosphere
> > >
> > > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji <zm...@apache.org>
> wrote:
> > >
> > > > I'm not convinced this is a valid use case.
> > > >
> > > > Mesos is supposed to be a generic kernel for launching "tasks",
> > whatever
> > > > they might be.
> > > >
> > > > In some cases it is useful to launch an executable, in other cases it
> > > might
> > > > be useful to launch a series of executables, and in some other cases
> it
> > > > might be useful to spawn a thread to do some work. Whatever that
> might
> > > be,
> > > > it doesn't matter to Mesos and the executor and framework are free to
> > > > establish a contract in `ExecutorInfo.data`, completely independent
> of
> > > the
> > > > Mesos API.
> > > >
> > > > I think formalizing this contract between executors and frameworks
> via
> > > > CommandInfo is going to introduce more problems than what they solve.
> > If
> > > > the CommandInfo struct is useful, frameworks and executors can just
> > stuff
> > > > that into ExecutorInfo.data, however it's not something that they
> need
> > to
> > > > adhere too.
> > > >
> > > > What's the underlying motivation for this?
> > > >
> > > >
> > > >
> > > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent <ha...@gmail.com>
> wrote:
> > > >
> > > > > For command task, if its `ExecutorInfo` would set with
> > > `CommandExecutor`
> > > > as
> > > > > well?
> > > > >
> > > > > Some tickets may relate to this.
> > > > >
> > > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > > >
> > > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone <vi...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are contemplating whether to allow both CommandInfo and
> > > ExecutorInfo
> > > > > on
> > > > > > TaskInfo (MESOS-6294 <https://issues.apache.org/
> > > jira/browse/MESOS-6294
> > > > > >).
> > > > > > Currently we only allow one or the other. The motivation is to
> > allow
> > > > > custom
> > > > > > executors a more structured way to pass information (e.g,
> command)
> > > > about
> > > > > > Task. Right now custom executors have to get this data via
> > > > > `TaskInfo.bytes`
> > > > > > which is not ideal.
> > > > > >
> > > > > > Are there any custom executors out there that crash if they get
> > Tasks
> > > > > with
> > > > > > CommandInfo set?
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Vinod
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards,
> > > > > Haosdent Huang
> > > > >
> > > > > --
> > > > > Zameer Manji
> > > > >
> > > >
> > >
> > > --
> > > Zameer Manji
> > >
> >
>
> --
> Zameer Manji
>
Re: Allowing both CommandInfo and ExecutorInfo on TaskInfo
Posted by Joris Van Remoortere <jo...@mesosphere.io>.
@zameer
I think your example makes a lot of sense. I didn't interpret the proposal
as one that would prevent your case.
The way I read the proposal is that we want to allow setting both, not
require it. If I misunderstood then please ignore my comments.
When I first heard about this proposal it seemed like a nice way for
frameworks and executors to start defining a more structured contract _if
they wanted to_, while still allowing others to keep passing their (from
Mesos's view) unstructured data as long as both sides agree on the
serialization / de-serialization.
Does this still seem like an issue if it's not a requirement but just a
broadening of possibilities?
—
*Joris Van Remoortere*
Mesosphere
On Wed, Nov 2, 2016 at 1:49 PM, Zameer Manji <zm...@apache.org> wrote:
> Joris,
>
> You make a good point. However, I'm not convinced that `CommandInfo` should
> be the well defined construct that people use. Can you please describe
> different custom executors, and the overlap between them and how
> CommandInfo will reduce that overlap? I'm having a hard time seeing where
> CommandInfo will solve all of their cases.
>
> Consider the cause of Thermos (Aurora's Executor), it could never use a
> `CommandInfo` struct because it executes a processes graph instead of a
> single command.
>
> If the project wants to go down this path, I think generalizing
> `CommandInfo` that could capture more cases (ie multiple commands or a
> graph of commands) would be a better first step.
>
> What do you think?
>
> On Wed, Oct 26, 2016 at 10:38 AM, Joris Van Remoortere <
> joris@mesosphere.io>
> wrote:
>
> > I do think it would be valuable to have a more well defined contract
> > between frameworks and custom executors.
> >
> > As Zameer pointed out a specific framework and accompanying custom
> executor
> > can decide to do that in the data bytes; however, if we started building
> > out a few different flavors of executors then it would be great for there
> > to be standard way to pass command information to them.
> >
> > The current model works well in a 1-1 mapping between framework and
> > executor binaries. In a world where that is 1-N it means all N executors
> > have to use the same method of passing the command.
> >
> > —
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Mon, Oct 17, 2016 at 4:25 PM, Zameer Manji <zm...@apache.org> wrote:
> >
> > > I'm not convinced this is a valid use case.
> > >
> > > Mesos is supposed to be a generic kernel for launching "tasks",
> whatever
> > > they might be.
> > >
> > > In some cases it is useful to launch an executable, in other cases it
> > might
> > > be useful to launch a series of executables, and in some other cases it
> > > might be useful to spawn a thread to do some work. Whatever that might
> > be,
> > > it doesn't matter to Mesos and the executor and framework are free to
> > > establish a contract in `ExecutorInfo.data`, completely independent of
> > the
> > > Mesos API.
> > >
> > > I think formalizing this contract between executors and frameworks via
> > > CommandInfo is going to introduce more problems than what they solve.
> If
> > > the CommandInfo struct is useful, frameworks and executors can just
> stuff
> > > that into ExecutorInfo.data, however it's not something that they need
> to
> > > adhere too.
> > >
> > > What's the underlying motivation for this?
> > >
> > >
> > >
> > > On Thu, Oct 13, 2016 at 10:40 AM, haosdent <ha...@gmail.com> wrote:
> > >
> > > > For command task, if its `ExecutorInfo` would set with
> > `CommandExecutor`
> > > as
> > > > well?
> > > >
> > > > Some tickets may relate to this.
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/MESOS-2330
> > > > [2]: https://issues.apache.org/jira/browse/MESOS-527
> > > > [3]: https://issues.apache.org/jira/browse/MESOS-5198
> > > >
> > > > On Fri, Oct 14, 2016 at 1:00 AM, Vinod Kone <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are contemplating whether to allow both CommandInfo and
> > ExecutorInfo
> > > > on
> > > > > TaskInfo (MESOS-6294 <https://issues.apache.org/
> > jira/browse/MESOS-6294
> > > > >).
> > > > > Currently we only allow one or the other. The motivation is to
> allow
> > > > custom
> > > > > executors a more structured way to pass information (e.g, command)
> > > about
> > > > > Task. Right now custom executors have to get this data via
> > > > `TaskInfo.bytes`
> > > > > which is not ideal.
> > > > >
> > > > > Are there any custom executors out there that crash if they get
> Tasks
> > > > with
> > > > > CommandInfo set?
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Vinod
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > > > --
> > > > Zameer Manji
> > > >
> > >
> >
> > --
> > Zameer Manji
> >
>