Golang Aurora lib, multiple executor support, integrate mesos task related fields

[Renan DelValle <rd...@binghamton.edu>]

Hello all,

I'd like to (re-)introduce myself. My name's Renan DelValle and I've had
the pleasure of being part of the Aurora community for the last year or so.

Last year I worked to allow Aurora to utilize custom executors. With the
help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature became
a reality and made into Aurora late last year. (Also got to show an early
beta at MesosCon!)

For this year's summer, I have some new goals which I invite anyone to
provide input on.

1. Create a Golang library that works as an alternative to Aurora's Python
client and Pystachio. The initial goal of this library is to support the
most critical job related Thrift API's. (Admin operations can continue to
be done from the Aurora CLI). Since we support custom executors, we need a
way that's not so tied to thermos (as Pystachio is) to send jobs
configurations to Aurora (and by extension the custom executor).

2. Create support for using multiple executors within a single Aurora
scheduler instance. As of right now, only a single executor can exist for
each Aurora scheduler instance. The idea is to allow the Aurora scheduler
to support multiple executors such that they can be used alongside one
another, providing flexibility for running jobs.

3. I'd like to add support for some first class Mesos task related fields.
These would be optional and/or have defaults so that the regular Aurora CLI
does not break. One of the examples I'm interested in integrating is the
Mesos Fetcher so that resources can be downloaded into the sandbox that the
custom executor may need. (The executor path will never be exposed as this
will be defined serverside and be static).

If anyone has any feedback on these, I'd be very glad to hear it.

That's it for me for now, thanks!

-Renan

Re: Aurora now supports multiple executors

[me...@yahoo.com.INVALID]

Renan shared it in user list. Here you go
https://github.com/rdelval/gorealis/blob/master/docs/getting-started.md

More features are getting added.


Thx

PS: btw compose executor docs would be updated as well (it's stale) and it's going through production hardening.


Sent from my iPhone

> On Aug 5, 2016, at 9:23 AM, Zhitao Li <zh...@gmail.com> wrote:
> 
> This sounds awesome!
> 
> Is there a document for how to use the feature?
> 
> On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
> 
>> Renaming the subject.
>> Renan's latest patch in 0.16 has completed the goal of implementing both 2
>> and 3. So, Aurora now can not only support custom executors but run
>> multiple executors in the cluster!! Mesos fetcher has been integrated as
>> well to help the custom executors have access to downloaded artifacts that
>> it needs. We are going to add dedicated documentation on how to enable this
>> in 0.16.
>> This was 2 summers in making with Renan's internship slots. Special thanks
>> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the
>> process.
>> 
>> 
>> As for point 1, we are going to put out the golang client in few weeks to
>> easily test different executors (including thermos) with Aurora with real
>> examples. During the same time we will release production grade
>> docker-compose executor simulating running container pods with Aurora.
>> Expect a detailed blog in a month.
>> Thx
>>      From: David McLaughlin <dm...@apache.org>
>> To: dev@aurora.apache.org
>> Sent: Monday, June 13, 2016 10:39 AM
>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>> mesos task related fields
>> 
>> Thanks, also +1 to supporting points 2 and 3.
>> 
>>> On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid> wrote:
>>> 
>>> Got it. Thx Max.
>>> Renan will starting working on 2 and 3, involving and updating you all.
>>> 
>>> Thx
>>> 
>>> Sent from my iPhone
>>> 
>>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
>>>> wrote:
>>>> 
>>>> David pinged me offline suggesting I misread the proposal for (2) as
>>>> an attempt to support multiple executors per task rather than "per
>>>> cluster". If that's the case, I am happy to change my vote for (2) to
>>>> +1. I don't see much harm (aside from possible UI inconsistencies)
>>>> from multiple executor types sharing the same cluster.
>>>> 
>>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid>
>> wrote:
>>>>> I will leave Renan for a detailed response.
>>>>> 
>>>>> If aurora can schedule Job A with thermos and Job B that requires a
>>> different executor that is a first class concept in mesos, why is there a
>>> objection? Renan's patch of replacing thermos executor is already in
>>> aurora. However right now aurora can statically configure only one
>>> executor. And so we want a job submission dynamically can call out the
>>> executor type.
>>>>> Please clarify your views.
>>>>> Right now the docker integration story in mesos is weak (current and
>>> also the new universal containerizer). We see much value in using
>> executors
>>> like
>>>>> https://github.com/mesos/docker-compose-executor
>>>>> With aurora jobs.
>>>>> 
>>>>> Points 2 and points 3 make the custom executor story stronger. (For
>>> example we will run both thermos and compose executor depending on job
>> type)
>>>>> 
>>>>> 
>>>>> Point 1, I agree. That is outside of aurora and most likely will be a
>>> shim on top of generated go thrift bindings but a helper to easily test
>>> custom executors.
>>>>> Last time when we reviewed with aurora team, it came out current
>> aurora
>>> cli is tightly coupled with thermos objects for jobs and hence keep it as
>>> is.
>>>>> 
>>>>> Thx
>>>>> 
>>>>> 
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>> wrote:
>>>>>> 
>>>>>> I am with Rick and David on this. The (1) and (2) feel like building
>> a
>>>>>> parallel universe within Aurora without a clear justification on the
>>>>>> long-term benefits for the community.
>>>>>> 
>>>>>> I am strong -1 on bringing yet another language into the Aurora
>>>>>> ecosystem without identifying the strategic upside. As David
>>>>>> suggested, any efforts towards building the first-class REST API
>>>>>> instead would go a long way and will be much appreciated by everyone.
>>>>>> 
>>>>>> I also don't see a reason to support multiple executors per task.
>> That
>>>>>> feels like re-creating thermos in a thermos-less world.
>>>>>> 
>>>>>> As for (3), I'd like to see more details on the mechanics here but
>>>>>> generally positive towards supporting more TaskInfo features in
>>>>>> Aurora.
>>>>>> 
>>>>>> 
>>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
>>>>>>> I generally shy away from technical goals that are based on a choice
>>> of language. What is it about a go client that can't be done by extending
>>> the existing client?
>>>>>>> 
>>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
>> rdelval1@binghamton.edu>
>>> wrote:
>>>>>>>> 
>>>>>>>> Hi David,
>>>>>>>> 
>>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
>>> david@dmclaughlin.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> Comments inline
>>>>>>>>>> 
>>>>>>>>>>   From: David McLaughlin <dm...@apache.org>
>>>>>>>>>> To: dev@aurora.apache.org
>>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
>>> integrate
>>>>>>>>>> mesos task related fields
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
>>> rdelval1@binghamton.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello all,
>>>>>>>>>>> 
>>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
>>> I've
>>>>>>>>> had
>>>>>>>>>>> the pleasure of being part of the Aurora community for the last
>>> year or
>>>>>>>>>> so.
>>>>>>>>>>> 
>>>>>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
>>> With
>>>>>>>>> the
>>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
>>> feature
>>>>>>>>>> became
>>>>>>>>>>> a reality and made into Aurora late last year. (Also got to show
>>> an
>>>>>>>>> early
>>>>>>>>>>> beta at MesosCon!)
>>>>>>>>>>> 
>>>>>>>>>>> For this year's summer, I have some new goals which I invite
>>> anyone to
>>>>>>>>>>> provide input on.
>>>>>>>>>>> 
>>>>>>>>>>> 1. Create a Golang library that works as an alternative to
>>> Aurora's
>>>>>>>>>> Python
>>>>>>>>>>> client and Pystachio. The initial goal of this library is to
>>> support
>>>>>>>>> the
>>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
>>> continue
>>>>>>>>> to
>>>>>>>>>>> be done from the Aurora CLI). Since we support custom executors,
>>> we
>>>>>>>>> need
>>>>>>>>>> a
>>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>>>>>>>> configurations to Aurora (and by extension the custom executor).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> You can easily add support for a different configuration format
>>> without
>>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own
>>> custom
>>>>>>>>>> noun/verbs to the client or replacing existing commands. For
>>> example, at
>>>>>>>>>> Twitter we have our own version of 'aurora update start' that
>>> plugs into
>>>>>>>>> an
>>>>>>>>>> internal Deploy Orchestration Service and we have a whole other
>>> command
>>>>>>>>> for
>>>>>>>>>> federated deploys. I can show you how we do that.
>>>>>>>>>> 
>>>>>>>>>> The first thing I thought of though was - this seems like a
>> perfect
>>>>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
>>> Having
>>>>>>>>> that
>>>>>>>>>> would make it trivial to do a CLI in any language you want, and
>> the
>>>>>>>>> Python
>>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>>>>>>> CLI integrations is not useful for a lot of different reasons.
>>> We need
>>>>>>>>>> API's from other orchestration points from different components
>>> and hence
>>>>>>>>>> we would package it in a go library. Uber, I believe has taken
>> the
>>> same
>>>>>>>>>> route (info from this mesoscon)
>>>>>>>>>> We are not writing a replacement cli tool. As noted admin
>>> operations
>>>>>>>>> would
>>>>>>>>>> heavily leverage current aurora cli.
>>>>>>>>>> I think REST work has been postponed and attempted few times in
>>> past. We
>>>>>>>>>> cannot wait for that.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This is the dev list for contributions to the project, so I
>> assumed
>>> we were
>>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and
>>> Thrift
>>>>>>>>> has Go bindings so you're all set for whatever custom tooling
>> you're
>>>>>>>>> building. Please use the users list for feedback on that.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> While I'm very thankful for your input on the matter, my original
>>> text very
>>>>>>>> clearly said:
>>>>>>>> "1. Create a Golang library that works as an *alternative* to
>>>>>>>> Aurora's Python client and Pystachio."
>>>>>>>> 
>>>>>>>> I believe every single item I have listed has the potential to
>>> become a
>>>>>>>> useful contribution to this project.
>>>>>>>> Therefore, I feel that it is prudent continue the discussion here
>>> (unless
>>>>>>>> others feel similarly).
>>>>>>>> 
>>>>>>>> -Renan
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 2. Create support for using multiple executors within a single
>>> Aurora
>>>>>>>>>>> scheduler instance. As of right now, only a single executor can
>>> exist
>>>>>>>>> for
>>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>>>>>>> scheduler
>>>>>>>>>>> to support multiple executors such that they can be used
>>> alongside one
>>>>>>>>>>> another, providing flexibility for running jobs.
>>>>>>>>>> 
>>>>>>>>>> Just for my own understanding - what problems are you solving
>> with
>>> your
>>>>>>>>>> custom executor? Sorry if you've explained this to the lists
>>> before.
>>>>>>>>>> 
>>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
>>> executor
>>>>>>>>>> config as an opaque blob and being more opinionated. The
>>>>>>>>> Thermos/Scheduler
>>>>>>>>>> abstraction really hinders what type of user experience (UI) we
>> can
>>>>>>>>> build,
>>>>>>>>>> and other frameworks do a much better job by being more
>>> opinionated and
>>>>>>>>>> pulling executor data into the scheduler UI.
>>>>>>>>>>>> We probably don't need to debate on this point. Executor is a
>>> first
>>>>>>>>>> class mesos citizen and time is right for aurora to have good
>>> support for
>>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
>>>>>>>>> docker-compose
>>>>>>>>>> and the executor manages that. Scheduler UI main features should
>>> not get
>>>>>>>>>> bogged down or be held back for a particular executor. That feels
>>>>>>>>>> incredibly restrictive. If a special UI mode for thermos executor
>>> is
>>>>>>>>>> created, that should be fine. We do have to differentiate the
>>> scheduler
>>>>>>>>> and
>>>>>>>>>> thermos and aurora team has done a great job of not hard coupling
>>> the
>>>>>>>>> two.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
>> related
>>>>>>>>>> fields.
>>>>>>>>>>> These would be optional and/or have defaults so that the regular
>>> Aurora
>>>>>>>>>> CLI
>>>>>>>>>>> does not break. One of the examples I'm interested in
>> integrating
>>> is
>>>>>>>>> the
>>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
>>> sandbox that
>>>>>>>>>> the
>>>>>>>>>>> custom executor may need. (The executor path will never be
>>> exposed as
>>>>>>>>>> this
>>>>>>>>>>> will be defined serverside and be static).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
>>> passed to
>>>>>>>>>> your executor? I have to admit I don't know enough about how the
>>> Mesos
>>>>>>>>>> fetcher works.
>>>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> We may add other first class fields.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
>> it.
>>>>>>>>>>> 
>>>>>>>>>>> That's it for me for now, thanks!
>>>>>>>>>>> 
>>>>>>>>>>> -Renan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |    |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |
>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>>> |  |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> 
>>>>>>>>>> |
> 
> 
> 
> 
> -- 
> Cheers,
> 
> Zhitao Li

Re: Aurora now supports multiple executors

[Renan DelValle <rd...@binghamton.edu>]

Thanks! Truth be told, it wouldn't have been possible without the mentoring
and support from so many in the community. Looking forward to making
further contributions in the future.

-Renan

On Aug 5, 2016 10:26 AM, "David McLaughlin" <dm...@apache.org> wrote:

> Congratulations Renan on contributing such a major feature back to the
> community. Looking forward to reading the blog post.
>
>
> On Fri, Aug 5, 2016 at 10:15 AM, <me...@yahoo.com.invalid> wrote:
>
> > Expect that to be up by mid next week and put in 0.16 docs.
> >
> > Sent from my iPhone
> >
> > > On Aug 5, 2016, at 9:23 AM, Zhitao Li <zh...@gmail.com> wrote:
> > >
> > > This sounds awesome!
> > >
> > > Is there a document for how to use the feature?
> > >
> > > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> > > meghdoot_b@yahoo.com.invalid> wrote:
> > >
> > >> Renaming the subject.
> > >> Renan's latest patch in 0.16 has completed the goal of implementing
> > both 2
> > >> and 3. So, Aurora now can not only support custom executors but run
> > >> multiple executors in the cluster!! Mesos fetcher has been integrated
> as
> > >> well to help the custom executors have access to downloaded artifacts
> > that
> > >> it needs. We are going to add dedicated documentation on how to enable
> > this
> > >> in 0.16.
> > >> This was 2 summers in making with Renan's internship slots. Special
> > thanks
> > >> to Maxim, Joshua and Bill to make this happen and mentoring Renan in
> the
> > >> process.
> > >>
> > >>
> > >> As for point 1, we are going to put out the golang client in few weeks
> > to
> > >> easily test different executors (including thermos) with Aurora with
> > real
> > >> examples. During the same time we will release production grade
> > >> docker-compose executor simulating running container pods with Aurora.
> > >> Expect a detailed blog in a month.
> > >> Thx
> > >>      From: David McLaughlin <dm...@apache.org>
> > >> To: dev@aurora.apache.org
> > >> Sent: Monday, June 13, 2016 10:39 AM
> > >> Subject: Re: Golang Aurora lib, multiple executor support, integrate
> > >> mesos task related fields
> > >>
> > >> Thanks, also +1 to supporting points 2 and 3.
> > >>
> > >>> On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid>
> > wrote:
> > >>>
> > >>> Got it. Thx Max.
> > >>> Renan will starting working on 2 and 3, involving and updating you
> all.
> > >>>
> > >>> Thx
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> David pinged me offline suggesting I misread the proposal for (2) as
> > >>>> an attempt to support multiple executors per task rather than "per
> > >>>> cluster". If that's the case, I am happy to change my vote for (2)
> to
> > >>>> +1. I don't see much harm (aside from possible UI inconsistencies)
> > >>>> from multiple executor types sharing the same cluster.
> > >>>>
> > >>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid>
> > >> wrote:
> > >>>>> I will leave Renan for a detailed response.
> > >>>>>
> > >>>>> If aurora can schedule Job A with thermos and Job B that requires a
> > >>> different executor that is a first class concept in mesos, why is
> > there a
> > >>> objection? Renan's patch of replacing thermos executor is already in
> > >>> aurora. However right now aurora can statically configure only one
> > >>> executor. And so we want a job submission dynamically can call out
> the
> > >>> executor type.
> > >>>>> Please clarify your views.
> > >>>>> Right now the docker integration story in mesos is weak (current
> and
> > >>> also the new universal containerizer). We see much value in using
> > >> executors
> > >>> like
> > >>>>> https://github.com/mesos/docker-compose-executor
> > >>>>> With aurora jobs.
> > >>>>>
> > >>>>> Points 2 and points 3 make the custom executor story stronger. (For
> > >>> example we will run both thermos and compose executor depending on
> job
> > >> type)
> > >>>>>
> > >>>>>
> > >>>>> Point 1, I agree. That is outside of aurora and most likely will
> be a
> > >>> shim on top of generated go thrift bindings but a helper to easily
> test
> > >>> custom executors.
> > >>>>> Last time when we reviewed with aurora team, it came out current
> > >> aurora
> > >>> cli is tightly coupled with thermos objects for jobs and hence keep
> it
> > as
> > >>> is.
> > >>>>>
> > >>>>> Thx
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Sent from my iPhone
> > >>>>>
> > >>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> > >>> wrote:
> > >>>>>>
> > >>>>>> I am with Rick and David on this. The (1) and (2) feel like
> building
> > >> a
> > >>>>>> parallel universe within Aurora without a clear justification on
> the
> > >>>>>> long-term benefits for the community.
> > >>>>>>
> > >>>>>> I am strong -1 on bringing yet another language into the Aurora
> > >>>>>> ecosystem without identifying the strategic upside. As David
> > >>>>>> suggested, any efforts towards building the first-class REST API
> > >>>>>> instead would go a long way and will be much appreciated by
> > everyone.
> > >>>>>>
> > >>>>>> I also don't see a reason to support multiple executors per task.
> > >> That
> > >>>>>> feels like re-creating thermos in a thermos-less world.
> > >>>>>>
> > >>>>>> As for (3), I'd like to see more details on the mechanics here but
> > >>>>>> generally positive towards supporting more TaskInfo features in
> > >>>>>> Aurora.
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> > >>>>>>> I generally shy away from technical goals that are based on a
> > choice
> > >>> of language. What is it about a go client that can't be done by
> > extending
> > >>> the existing client?
> > >>>>>>>
> > >>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
> > >> rdelval1@binghamton.edu>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi David,
> > >>>>>>>>
> > >>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> > >>> david@dmclaughlin.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> > >>>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Comments inline
> > >>>>>>>>>>
> > >>>>>>>>>>   From: David McLaughlin <dm...@apache.org>
> > >>>>>>>>>> To: dev@aurora.apache.org
> > >>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> > >>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> > >>> integrate
> > >>>>>>>>>> mesos task related fields
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> > >>> rdelval1@binghamton.edu>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello all,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle
> and
> > >>> I've
> > >>>>>>>>> had
> > >>>>>>>>>>> the pleasure of being part of the Aurora community for the
> last
> > >>> year or
> > >>>>>>>>>> so.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Last year I worked to allow Aurora to utilize custom
> executors.
> > >>> With
> > >>>>>>>>> the
> > >>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> > >>> feature
> > >>>>>>>>>> became
> > >>>>>>>>>>> a reality and made into Aurora late last year. (Also got to
> > show
> > >>> an
> > >>>>>>>>> early
> > >>>>>>>>>>> beta at MesosCon!)
> > >>>>>>>>>>>
> > >>>>>>>>>>> For this year's summer, I have some new goals which I invite
> > >>> anyone to
> > >>>>>>>>>>> provide input on.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. Create a Golang library that works as an alternative to
> > >>> Aurora's
> > >>>>>>>>>> Python
> > >>>>>>>>>>> client and Pystachio. The initial goal of this library is to
> > >>> support
> > >>>>>>>>> the
> > >>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
> > >>> continue
> > >>>>>>>>> to
> > >>>>>>>>>>> be done from the Aurora CLI). Since we support custom
> > executors,
> > >>> we
> > >>>>>>>>> need
> > >>>>>>>>>> a
> > >>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send
> > jobs
> > >>>>>>>>>>> configurations to Aurora (and by extension the custom
> > executor).
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> You can easily add support for a different configuration
> format
> > >>> without
> > >>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your
> > own
> > >>> custom
> > >>>>>>>>>> noun/verbs to the client or replacing existing commands. For
> > >>> example, at
> > >>>>>>>>>> Twitter we have our own version of 'aurora update start' that
> > >>> plugs into
> > >>>>>>>>> an
> > >>>>>>>>>> internal Deploy Orchestration Service and we have a whole
> other
> > >>> command
> > >>>>>>>>> for
> > >>>>>>>>>> federated deploys. I can show you how we do that.
> > >>>>>>>>>>
> > >>>>>>>>>> The first thing I thought of though was - this seems like a
> > >> perfect
> > >>>>>>>>>> starting point to get serious about a HTTP+JSON API for
> Aurora.
> > >>> Having
> > >>>>>>>>> that
> > >>>>>>>>>> would make it trivial to do a CLI in any language you want,
> and
> > >> the
> > >>>>>>>>> Python
> > >>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
> > >>>>>>>>>>>> CLI integrations is not useful for a lot of different
> reasons.
> > >>> We need
> > >>>>>>>>>> API's from other orchestration points from different
> components
> > >>> and hence
> > >>>>>>>>>> we would package it in a go library. Uber, I believe has taken
> > >> the
> > >>> same
> > >>>>>>>>>> route (info from this mesoscon)
> > >>>>>>>>>> We are not writing a replacement cli tool. As noted admin
> > >>> operations
> > >>>>>>>>> would
> > >>>>>>>>>> heavily leverage current aurora cli.
> > >>>>>>>>>> I think REST work has been postponed and attempted few times
> in
> > >>> past. We
> > >>>>>>>>>> cannot wait for that.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> This is the dev list for contributions to the project, so I
> > >> assumed
> > >>> we were
> > >>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift
> > and
> > >>> Thrift
> > >>>>>>>>> has Go bindings so you're all set for whatever custom tooling
> > >> you're
> > >>>>>>>>> building. Please use the users list for feedback on that.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> While I'm very thankful for your input on the matter, my
> original
> > >>> text very
> > >>>>>>>> clearly said:
> > >>>>>>>> "1. Create a Golang library that works as an *alternative* to
> > >>>>>>>> Aurora's Python client and Pystachio."
> > >>>>>>>>
> > >>>>>>>> I believe every single item I have listed has the potential to
> > >>> become a
> > >>>>>>>> useful contribution to this project.
> > >>>>>>>> Therefore, I feel that it is prudent continue the discussion
> here
> > >>> (unless
> > >>>>>>>> others feel similarly).
> > >>>>>>>>
> > >>>>>>>> -Renan
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. Create support for using multiple executors within a
> single
> > >>> Aurora
> > >>>>>>>>>>> scheduler instance. As of right now, only a single executor
> can
> > >>> exist
> > >>>>>>>>> for
> > >>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the
> Aurora
> > >>>>>>>>> scheduler
> > >>>>>>>>>>> to support multiple executors such that they can be used
> > >>> alongside one
> > >>>>>>>>>>> another, providing flexibility for running jobs.
> > >>>>>>>>>>
> > >>>>>>>>>> Just for my own understanding - what problems are you solving
> > >> with
> > >>> your
> > >>>>>>>>>> custom executor? Sorry if you've explained this to the lists
> > >>> before.
> > >>>>>>>>>>
> > >>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
> > >>> executor
> > >>>>>>>>>> config as an opaque blob and being more opinionated. The
> > >>>>>>>>> Thermos/Scheduler
> > >>>>>>>>>> abstraction really hinders what type of user experience (UI)
> we
> > >> can
> > >>>>>>>>> build,
> > >>>>>>>>>> and other frameworks do a much better job by being more
> > >>> opinionated and
> > >>>>>>>>>> pulling executor data into the scheduler UI.
> > >>>>>>>>>>>> We probably don't need to debate on this point. Executor is
> a
> > >>> first
> > >>>>>>>>>> class mesos citizen and time is right for aurora to have good
> > >>> support for
> > >>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
> > >>>>>>>>> docker-compose
> > >>>>>>>>>> and the executor manages that. Scheduler UI main features
> should
> > >>> not get
> > >>>>>>>>>> bogged down or be held back for a particular executor. That
> > feels
> > >>>>>>>>>> incredibly restrictive. If a special UI mode for thermos
> > executor
> > >>> is
> > >>>>>>>>>> created, that should be fine. We do have to differentiate the
> > >>> scheduler
> > >>>>>>>>> and
> > >>>>>>>>>> thermos and aurora team has done a great job of not hard
> > coupling
> > >>> the
> > >>>>>>>>> two.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
> > >> related
> > >>>>>>>>>> fields.
> > >>>>>>>>>>> These would be optional and/or have defaults so that the
> > regular
> > >>> Aurora
> > >>>>>>>>>> CLI
> > >>>>>>>>>>> does not break. One of the examples I'm interested in
> > >> integrating
> > >>> is
> > >>>>>>>>> the
> > >>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> > >>> sandbox that
> > >>>>>>>>>> the
> > >>>>>>>>>>> custom executor may need. (The executor path will never be
> > >>> exposed as
> > >>>>>>>>>> this
> > >>>>>>>>>>> will be defined serverside and be static).
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> > >>> passed to
> > >>>>>>>>>> your executor? I have to admit I don't know enough about how
> the
> > >>> Mesos
> > >>>>>>>>>> fetcher works.
> > >>>>>>>>>>>> Apache Mesos - Fetcher
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> We may add other first class fields.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
> > >> it.
> > >>>>>>>>>>>
> > >>>>>>>>>>> That's it for me for now, thanks!
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Renan
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |  |    |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>> |
> > >>>>>>>>>> |  |
> > >>>>>>>>>> Apache Mesos - Fetcher
> > >>>>>>>>>> |  |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >>>>>>>>>>
> > >>>>>>>>>> |
> > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> >
>

Re: Aurora now supports multiple executors

[David McLaughlin <dm...@apache.org>]

Congratulations Renan on contributing such a major feature back to the
community. Looking forward to reading the blog post.


On Fri, Aug 5, 2016 at 10:15 AM, <me...@yahoo.com.invalid> wrote:

> Expect that to be up by mid next week and put in 0.16 docs.
>
> Sent from my iPhone
>
> > On Aug 5, 2016, at 9:23 AM, Zhitao Li <zh...@gmail.com> wrote:
> >
> > This sounds awesome!
> >
> > Is there a document for how to use the feature?
> >
> > On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> > meghdoot_b@yahoo.com.invalid> wrote:
> >
> >> Renaming the subject.
> >> Renan's latest patch in 0.16 has completed the goal of implementing
> both 2
> >> and 3. So, Aurora now can not only support custom executors but run
> >> multiple executors in the cluster!! Mesos fetcher has been integrated as
> >> well to help the custom executors have access to downloaded artifacts
> that
> >> it needs. We are going to add dedicated documentation on how to enable
> this
> >> in 0.16.
> >> This was 2 summers in making with Renan's internship slots. Special
> thanks
> >> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the
> >> process.
> >>
> >>
> >> As for point 1, we are going to put out the golang client in few weeks
> to
> >> easily test different executors (including thermos) with Aurora with
> real
> >> examples. During the same time we will release production grade
> >> docker-compose executor simulating running container pods with Aurora.
> >> Expect a detailed blog in a month.
> >> Thx
> >>      From: David McLaughlin <dm...@apache.org>
> >> To: dev@aurora.apache.org
> >> Sent: Monday, June 13, 2016 10:39 AM
> >> Subject: Re: Golang Aurora lib, multiple executor support, integrate
> >> mesos task related fields
> >>
> >> Thanks, also +1 to supporting points 2 and 3.
> >>
> >>> On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid>
> wrote:
> >>>
> >>> Got it. Thx Max.
> >>> Renan will starting working on 2 and 3, involving and updating you all.
> >>>
> >>> Thx
> >>>
> >>> Sent from my iPhone
> >>>
> >>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> >>>> wrote:
> >>>>
> >>>> David pinged me offline suggesting I misread the proposal for (2) as
> >>>> an attempt to support multiple executors per task rather than "per
> >>>> cluster". If that's the case, I am happy to change my vote for (2) to
> >>>> +1. I don't see much harm (aside from possible UI inconsistencies)
> >>>> from multiple executor types sharing the same cluster.
> >>>>
> >>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid>
> >> wrote:
> >>>>> I will leave Renan for a detailed response.
> >>>>>
> >>>>> If aurora can schedule Job A with thermos and Job B that requires a
> >>> different executor that is a first class concept in mesos, why is
> there a
> >>> objection? Renan's patch of replacing thermos executor is already in
> >>> aurora. However right now aurora can statically configure only one
> >>> executor. And so we want a job submission dynamically can call out the
> >>> executor type.
> >>>>> Please clarify your views.
> >>>>> Right now the docker integration story in mesos is weak (current and
> >>> also the new universal containerizer). We see much value in using
> >> executors
> >>> like
> >>>>> https://github.com/mesos/docker-compose-executor
> >>>>> With aurora jobs.
> >>>>>
> >>>>> Points 2 and points 3 make the custom executor story stronger. (For
> >>> example we will run both thermos and compose executor depending on job
> >> type)
> >>>>>
> >>>>>
> >>>>> Point 1, I agree. That is outside of aurora and most likely will be a
> >>> shim on top of generated go thrift bindings but a helper to easily test
> >>> custom executors.
> >>>>> Last time when we reviewed with aurora team, it came out current
> >> aurora
> >>> cli is tightly coupled with thermos objects for jobs and hence keep it
> as
> >>> is.
> >>>>>
> >>>>> Thx
> >>>>>
> >>>>>
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>> I am with Rick and David on this. The (1) and (2) feel like building
> >> a
> >>>>>> parallel universe within Aurora without a clear justification on the
> >>>>>> long-term benefits for the community.
> >>>>>>
> >>>>>> I am strong -1 on bringing yet another language into the Aurora
> >>>>>> ecosystem without identifying the strategic upside. As David
> >>>>>> suggested, any efforts towards building the first-class REST API
> >>>>>> instead would go a long way and will be much appreciated by
> everyone.
> >>>>>>
> >>>>>> I also don't see a reason to support multiple executors per task.
> >> That
> >>>>>> feels like re-creating thermos in a thermos-less world.
> >>>>>>
> >>>>>> As for (3), I'd like to see more details on the mechanics here but
> >>>>>> generally positive towards supporting more TaskInfo features in
> >>>>>> Aurora.
> >>>>>>
> >>>>>>
> >>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> >>>>>>> I generally shy away from technical goals that are based on a
> choice
> >>> of language. What is it about a go client that can't be done by
> extending
> >>> the existing client?
> >>>>>>>
> >>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
> >> rdelval1@binghamton.edu>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> Hi David,
> >>>>>>>>
> >>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> >>> david@dmclaughlin.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> >>>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
> >>>>>>>>>
> >>>>>>>>>> Comments inline
> >>>>>>>>>>
> >>>>>>>>>>   From: David McLaughlin <dm...@apache.org>
> >>>>>>>>>> To: dev@aurora.apache.org
> >>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> >>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> >>> integrate
> >>>>>>>>>> mesos task related fields
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> >>> rdelval1@binghamton.edu>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello all,
> >>>>>>>>>>>
> >>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
> >>> I've
> >>>>>>>>> had
> >>>>>>>>>>> the pleasure of being part of the Aurora community for the last
> >>> year or
> >>>>>>>>>> so.
> >>>>>>>>>>>
> >>>>>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
> >>> With
> >>>>>>>>> the
> >>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> >>> feature
> >>>>>>>>>> became
> >>>>>>>>>>> a reality and made into Aurora late last year. (Also got to
> show
> >>> an
> >>>>>>>>> early
> >>>>>>>>>>> beta at MesosCon!)
> >>>>>>>>>>>
> >>>>>>>>>>> For this year's summer, I have some new goals which I invite
> >>> anyone to
> >>>>>>>>>>> provide input on.
> >>>>>>>>>>>
> >>>>>>>>>>> 1. Create a Golang library that works as an alternative to
> >>> Aurora's
> >>>>>>>>>> Python
> >>>>>>>>>>> client and Pystachio. The initial goal of this library is to
> >>> support
> >>>>>>>>> the
> >>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
> >>> continue
> >>>>>>>>> to
> >>>>>>>>>>> be done from the Aurora CLI). Since we support custom
> executors,
> >>> we
> >>>>>>>>> need
> >>>>>>>>>> a
> >>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send
> jobs
> >>>>>>>>>>> configurations to Aurora (and by extension the custom
> executor).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> You can easily add support for a different configuration format
> >>> without
> >>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your
> own
> >>> custom
> >>>>>>>>>> noun/verbs to the client or replacing existing commands. For
> >>> example, at
> >>>>>>>>>> Twitter we have our own version of 'aurora update start' that
> >>> plugs into
> >>>>>>>>> an
> >>>>>>>>>> internal Deploy Orchestration Service and we have a whole other
> >>> command
> >>>>>>>>> for
> >>>>>>>>>> federated deploys. I can show you how we do that.
> >>>>>>>>>>
> >>>>>>>>>> The first thing I thought of though was - this seems like a
> >> perfect
> >>>>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
> >>> Having
> >>>>>>>>> that
> >>>>>>>>>> would make it trivial to do a CLI in any language you want, and
> >> the
> >>>>>>>>> Python
> >>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
> >>>>>>>>>>>> CLI integrations is not useful for a lot of different reasons.
> >>> We need
> >>>>>>>>>> API's from other orchestration points from different components
> >>> and hence
> >>>>>>>>>> we would package it in a go library. Uber, I believe has taken
> >> the
> >>> same
> >>>>>>>>>> route (info from this mesoscon)
> >>>>>>>>>> We are not writing a replacement cli tool. As noted admin
> >>> operations
> >>>>>>>>> would
> >>>>>>>>>> heavily leverage current aurora cli.
> >>>>>>>>>> I think REST work has been postponed and attempted few times in
> >>> past. We
> >>>>>>>>>> cannot wait for that.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This is the dev list for contributions to the project, so I
> >> assumed
> >>> we were
> >>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift
> and
> >>> Thrift
> >>>>>>>>> has Go bindings so you're all set for whatever custom tooling
> >> you're
> >>>>>>>>> building. Please use the users list for feedback on that.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> While I'm very thankful for your input on the matter, my original
> >>> text very
> >>>>>>>> clearly said:
> >>>>>>>> "1. Create a Golang library that works as an *alternative* to
> >>>>>>>> Aurora's Python client and Pystachio."
> >>>>>>>>
> >>>>>>>> I believe every single item I have listed has the potential to
> >>> become a
> >>>>>>>> useful contribution to this project.
> >>>>>>>> Therefore, I feel that it is prudent continue the discussion here
> >>> (unless
> >>>>>>>> others feel similarly).
> >>>>>>>>
> >>>>>>>> -Renan
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 2. Create support for using multiple executors within a single
> >>> Aurora
> >>>>>>>>>>> scheduler instance. As of right now, only a single executor can
> >>> exist
> >>>>>>>>> for
> >>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
> >>>>>>>>> scheduler
> >>>>>>>>>>> to support multiple executors such that they can be used
> >>> alongside one
> >>>>>>>>>>> another, providing flexibility for running jobs.
> >>>>>>>>>>
> >>>>>>>>>> Just for my own understanding - what problems are you solving
> >> with
> >>> your
> >>>>>>>>>> custom executor? Sorry if you've explained this to the lists
> >>> before.
> >>>>>>>>>>
> >>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
> >>> executor
> >>>>>>>>>> config as an opaque blob and being more opinionated. The
> >>>>>>>>> Thermos/Scheduler
> >>>>>>>>>> abstraction really hinders what type of user experience (UI) we
> >> can
> >>>>>>>>> build,
> >>>>>>>>>> and other frameworks do a much better job by being more
> >>> opinionated and
> >>>>>>>>>> pulling executor data into the scheduler UI.
> >>>>>>>>>>>> We probably don't need to debate on this point. Executor is a
> >>> first
> >>>>>>>>>> class mesos citizen and time is right for aurora to have good
> >>> support for
> >>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
> >>>>>>>>> docker-compose
> >>>>>>>>>> and the executor manages that. Scheduler UI main features should
> >>> not get
> >>>>>>>>>> bogged down or be held back for a particular executor. That
> feels
> >>>>>>>>>> incredibly restrictive. If a special UI mode for thermos
> executor
> >>> is
> >>>>>>>>>> created, that should be fine. We do have to differentiate the
> >>> scheduler
> >>>>>>>>> and
> >>>>>>>>>> thermos and aurora team has done a great job of not hard
> coupling
> >>> the
> >>>>>>>>> two.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
> >> related
> >>>>>>>>>> fields.
> >>>>>>>>>>> These would be optional and/or have defaults so that the
> regular
> >>> Aurora
> >>>>>>>>>> CLI
> >>>>>>>>>>> does not break. One of the examples I'm interested in
> >> integrating
> >>> is
> >>>>>>>>> the
> >>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> >>> sandbox that
> >>>>>>>>>> the
> >>>>>>>>>>> custom executor may need. (The executor path will never be
> >>> exposed as
> >>>>>>>>>> this
> >>>>>>>>>>> will be defined serverside and be static).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> >>> passed to
> >>>>>>>>>> your executor? I have to admit I don't know enough about how the
> >>> Mesos
> >>>>>>>>>> fetcher works.
> >>>>>>>>>>>> Apache Mesos - Fetcher
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> We may add other first class fields.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
> >> it.
> >>>>>>>>>>>
> >>>>>>>>>>> That's it for me for now, thanks!
> >>>>>>>>>>>
> >>>>>>>>>>> -Renan
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> |
> >>>>>>>>>> |
> >>>>>>>>>> |
> >>>>>>>>>> |  |    |
> >>>>>>>>>>
> >>>>>>>>>> |
> >>>>>>>>>>
> >>>>>>>>>> |
> >>>>>>>>>> |
> >>>>>>>>>> |  |
> >>>>>>>>>> Apache Mesos - Fetcher
> >>>>>>>>>> |  |
> >>>>>>>>>>
> >>>>>>>>>> |
> >>>>>>>>>>
> >>>>>>>>>> |
> >
> >
> >
> >
> > --
> > Cheers,
> >
> > Zhitao Li
>

Re: Aurora now supports multiple executors

[me...@yahoo.com.INVALID]

Expect that to be up by mid next week and put in 0.16 docs.

Sent from my iPhone

> On Aug 5, 2016, at 9:23 AM, Zhitao Li <zh...@gmail.com> wrote:
> 
> This sounds awesome!
> 
> Is there a document for how to use the feature?
> 
> On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
> 
>> Renaming the subject.
>> Renan's latest patch in 0.16 has completed the goal of implementing both 2
>> and 3. So, Aurora now can not only support custom executors but run
>> multiple executors in the cluster!! Mesos fetcher has been integrated as
>> well to help the custom executors have access to downloaded artifacts that
>> it needs. We are going to add dedicated documentation on how to enable this
>> in 0.16.
>> This was 2 summers in making with Renan's internship slots. Special thanks
>> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the
>> process.
>> 
>> 
>> As for point 1, we are going to put out the golang client in few weeks to
>> easily test different executors (including thermos) with Aurora with real
>> examples. During the same time we will release production grade
>> docker-compose executor simulating running container pods with Aurora.
>> Expect a detailed blog in a month.
>> Thx
>>      From: David McLaughlin <dm...@apache.org>
>> To: dev@aurora.apache.org
>> Sent: Monday, June 13, 2016 10:39 AM
>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>> mesos task related fields
>> 
>> Thanks, also +1 to supporting points 2 and 3.
>> 
>>> On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid> wrote:
>>> 
>>> Got it. Thx Max.
>>> Renan will starting working on 2 and 3, involving and updating you all.
>>> 
>>> Thx
>>> 
>>> Sent from my iPhone
>>> 
>>>>> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
>>>> wrote:
>>>> 
>>>> David pinged me offline suggesting I misread the proposal for (2) as
>>>> an attempt to support multiple executors per task rather than "per
>>>> cluster". If that's the case, I am happy to change my vote for (2) to
>>>> +1. I don't see much harm (aside from possible UI inconsistencies)
>>>> from multiple executor types sharing the same cluster.
>>>> 
>>>>> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid>
>> wrote:
>>>>> I will leave Renan for a detailed response.
>>>>> 
>>>>> If aurora can schedule Job A with thermos and Job B that requires a
>>> different executor that is a first class concept in mesos, why is there a
>>> objection? Renan's patch of replacing thermos executor is already in
>>> aurora. However right now aurora can statically configure only one
>>> executor. And so we want a job submission dynamically can call out the
>>> executor type.
>>>>> Please clarify your views.
>>>>> Right now the docker integration story in mesos is weak (current and
>>> also the new universal containerizer). We see much value in using
>> executors
>>> like
>>>>> https://github.com/mesos/docker-compose-executor
>>>>> With aurora jobs.
>>>>> 
>>>>> Points 2 and points 3 make the custom executor story stronger. (For
>>> example we will run both thermos and compose executor depending on job
>> type)
>>>>> 
>>>>> 
>>>>> Point 1, I agree. That is outside of aurora and most likely will be a
>>> shim on top of generated go thrift bindings but a helper to easily test
>>> custom executors.
>>>>> Last time when we reviewed with aurora team, it came out current
>> aurora
>>> cli is tightly coupled with thermos objects for jobs and hence keep it as
>>> is.
>>>>> 
>>>>> Thx
>>>>> 
>>>>> 
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
>>> wrote:
>>>>>> 
>>>>>> I am with Rick and David on this. The (1) and (2) feel like building
>> a
>>>>>> parallel universe within Aurora without a clear justification on the
>>>>>> long-term benefits for the community.
>>>>>> 
>>>>>> I am strong -1 on bringing yet another language into the Aurora
>>>>>> ecosystem without identifying the strategic upside. As David
>>>>>> suggested, any efforts towards building the first-class REST API
>>>>>> instead would go a long way and will be much appreciated by everyone.
>>>>>> 
>>>>>> I also don't see a reason to support multiple executors per task.
>> That
>>>>>> feels like re-creating thermos in a thermos-less world.
>>>>>> 
>>>>>> As for (3), I'd like to see more details on the mechanics here but
>>>>>> generally positive towards supporting more TaskInfo features in
>>>>>> Aurora.
>>>>>> 
>>>>>> 
>>>>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
>>>>>>> I generally shy away from technical goals that are based on a choice
>>> of language. What is it about a go client that can't be done by extending
>>> the existing client?
>>>>>>> 
>>>>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
>> rdelval1@binghamton.edu>
>>> wrote:
>>>>>>>> 
>>>>>>>> Hi David,
>>>>>>>> 
>>>>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
>>> david@dmclaughlin.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> Comments inline
>>>>>>>>>> 
>>>>>>>>>>   From: David McLaughlin <dm...@apache.org>
>>>>>>>>>> To: dev@aurora.apache.org
>>>>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
>>> integrate
>>>>>>>>>> mesos task related fields
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
>>> rdelval1@binghamton.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello all,
>>>>>>>>>>> 
>>>>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
>>> I've
>>>>>>>>> had
>>>>>>>>>>> the pleasure of being part of the Aurora community for the last
>>> year or
>>>>>>>>>> so.
>>>>>>>>>>> 
>>>>>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
>>> With
>>>>>>>>> the
>>>>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
>>> feature
>>>>>>>>>> became
>>>>>>>>>>> a reality and made into Aurora late last year. (Also got to show
>>> an
>>>>>>>>> early
>>>>>>>>>>> beta at MesosCon!)
>>>>>>>>>>> 
>>>>>>>>>>> For this year's summer, I have some new goals which I invite
>>> anyone to
>>>>>>>>>>> provide input on.
>>>>>>>>>>> 
>>>>>>>>>>> 1. Create a Golang library that works as an alternative to
>>> Aurora's
>>>>>>>>>> Python
>>>>>>>>>>> client and Pystachio. The initial goal of this library is to
>>> support
>>>>>>>>> the
>>>>>>>>>>> most critical job related Thrift API's. (Admin operations can
>>> continue
>>>>>>>>> to
>>>>>>>>>>> be done from the Aurora CLI). Since we support custom executors,
>>> we
>>>>>>>>> need
>>>>>>>>>> a
>>>>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>>>>>>>> configurations to Aurora (and by extension the custom executor).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> You can easily add support for a different configuration format
>>> without
>>>>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own
>>> custom
>>>>>>>>>> noun/verbs to the client or replacing existing commands. For
>>> example, at
>>>>>>>>>> Twitter we have our own version of 'aurora update start' that
>>> plugs into
>>>>>>>>> an
>>>>>>>>>> internal Deploy Orchestration Service and we have a whole other
>>> command
>>>>>>>>> for
>>>>>>>>>> federated deploys. I can show you how we do that.
>>>>>>>>>> 
>>>>>>>>>> The first thing I thought of though was - this seems like a
>> perfect
>>>>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
>>> Having
>>>>>>>>> that
>>>>>>>>>> would make it trivial to do a CLI in any language you want, and
>> the
>>>>>>>>> Python
>>>>>>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>>>>>>> CLI integrations is not useful for a lot of different reasons.
>>> We need
>>>>>>>>>> API's from other orchestration points from different components
>>> and hence
>>>>>>>>>> we would package it in a go library. Uber, I believe has taken
>> the
>>> same
>>>>>>>>>> route (info from this mesoscon)
>>>>>>>>>> We are not writing a replacement cli tool. As noted admin
>>> operations
>>>>>>>>> would
>>>>>>>>>> heavily leverage current aurora cli.
>>>>>>>>>> I think REST work has been postponed and attempted few times in
>>> past. We
>>>>>>>>>> cannot wait for that.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> This is the dev list for contributions to the project, so I
>> assumed
>>> we were
>>>>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and
>>> Thrift
>>>>>>>>> has Go bindings so you're all set for whatever custom tooling
>> you're
>>>>>>>>> building. Please use the users list for feedback on that.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> While I'm very thankful for your input on the matter, my original
>>> text very
>>>>>>>> clearly said:
>>>>>>>> "1. Create a Golang library that works as an *alternative* to
>>>>>>>> Aurora's Python client and Pystachio."
>>>>>>>> 
>>>>>>>> I believe every single item I have listed has the potential to
>>> become a
>>>>>>>> useful contribution to this project.
>>>>>>>> Therefore, I feel that it is prudent continue the discussion here
>>> (unless
>>>>>>>> others feel similarly).
>>>>>>>> 
>>>>>>>> -Renan
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 2. Create support for using multiple executors within a single
>>> Aurora
>>>>>>>>>>> scheduler instance. As of right now, only a single executor can
>>> exist
>>>>>>>>> for
>>>>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>>>>>>> scheduler
>>>>>>>>>>> to support multiple executors such that they can be used
>>> alongside one
>>>>>>>>>>> another, providing flexibility for running jobs.
>>>>>>>>>> 
>>>>>>>>>> Just for my own understanding - what problems are you solving
>> with
>>> your
>>>>>>>>>> custom executor? Sorry if you've explained this to the lists
>>> before.
>>>>>>>>>> 
>>>>>>>>>> I ask because I'd really like to see Aurora stop treating the
>>> executor
>>>>>>>>>> config as an opaque blob and being more opinionated. The
>>>>>>>>> Thermos/Scheduler
>>>>>>>>>> abstraction really hinders what type of user experience (UI) we
>> can
>>>>>>>>> build,
>>>>>>>>>> and other frameworks do a much better job by being more
>>> opinionated and
>>>>>>>>>> pulling executor data into the scheduler UI.
>>>>>>>>>>>> We probably don't need to debate on this point. Executor is a
>>> first
>>>>>>>>>> class mesos citizen and time is right for aurora to have good
>>> support for
>>>>>>>>>> it.In our case, we have kubernetes like pods modeled through
>>>>>>>>> docker-compose
>>>>>>>>>> and the executor manages that. Scheduler UI main features should
>>> not get
>>>>>>>>>> bogged down or be held back for a particular executor. That feels
>>>>>>>>>> incredibly restrictive. If a special UI mode for thermos executor
>>> is
>>>>>>>>>> created, that should be fine. We do have to differentiate the
>>> scheduler
>>>>>>>>> and
>>>>>>>>>> thermos and aurora team has done a great job of not hard coupling
>>> the
>>>>>>>>> two.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 3. I'd like to add support for some first class Mesos task
>> related
>>>>>>>>>> fields.
>>>>>>>>>>> These would be optional and/or have defaults so that the regular
>>> Aurora
>>>>>>>>>> CLI
>>>>>>>>>>> does not break. One of the examples I'm interested in
>> integrating
>>> is
>>>>>>>>> the
>>>>>>>>>>> Mesos Fetcher so that resources can be downloaded into the
>>> sandbox that
>>>>>>>>>> the
>>>>>>>>>>> custom executor may need. (The executor path will never be
>>> exposed as
>>>>>>>>>> this
>>>>>>>>>>> will be defined serverside and be static).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Wouldn't the mesos-fetcher call just be another process to be
>>> passed to
>>>>>>>>>> your executor? I have to admit I don't know enough about how the
>>> Mesos
>>>>>>>>>> fetcher works.
>>>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> We may add other first class fields.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
>> it.
>>>>>>>>>>> 
>>>>>>>>>>> That's it for me for now, thanks!
>>>>>>>>>>> 
>>>>>>>>>>> -Renan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |    |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> |
>>>>>>>>>> |  |
>>>>>>>>>> Apache Mesos - Fetcher
>>>>>>>>>> |  |
>>>>>>>>>> 
>>>>>>>>>> |
>>>>>>>>>> 
>>>>>>>>>> |
> 
> 
> 
> 
> -- 
> Cheers,
> 
> Zhitao Li

Re: Aurora now supports multiple executors

[Zhitao Li <zh...@gmail.com>]

This sounds awesome!

Is there a document for how to use the feature?

On Fri, Aug 5, 2016 at 12:35 AM, meghdoot bhattacharya <
meghdoot_b@yahoo.com.invalid> wrote:

> Renaming the subject.
> Renan's latest patch in 0.16 has completed the goal of implementing both 2
> and 3. So, Aurora now can not only support custom executors but run
> multiple executors in the cluster!! Mesos fetcher has been integrated as
> well to help the custom executors have access to downloaded artifacts that
> it needs. We are going to add dedicated documentation on how to enable this
> in 0.16.
> This was 2 summers in making with Renan's internship slots. Special thanks
> to Maxim, Joshua and Bill to make this happen and mentoring Renan in the
> process.
>
>
> As for point 1, we are going to put out the golang client in few weeks to
> easily test different executors (including thermos) with Aurora with real
> examples. During the same time we will release production grade
> docker-compose executor simulating running container pods with Aurora.
> Expect a detailed blog in a month.
> Thx
>       From: David McLaughlin <dm...@apache.org>
>  To: dev@aurora.apache.org
>  Sent: Monday, June 13, 2016 10:39 AM
>  Subject: Re: Golang Aurora lib, multiple executor support, integrate
> mesos task related fields
>
> Thanks, also +1 to supporting points 2 and 3.
>
> On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid> wrote:
>
> > Got it. Thx Max.
> > Renan will starting working on 2 and 3, involving and updating you all.
> >
> > Thx
> >
> > Sent from my iPhone
> >
> > > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> > wrote:
> > >
> > > David pinged me offline suggesting I misread the proposal for (2) as
> > > an attempt to support multiple executors per task rather than "per
> > > cluster". If that's the case, I am happy to change my vote for (2) to
> > > +1. I don't see much harm (aside from possible UI inconsistencies)
> > > from multiple executor types sharing the same cluster.
> > >
> > >> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid>
> wrote:
> > >> I will leave Renan for a detailed response.
> > >>
> > >> If aurora can schedule Job A with thermos and Job B that requires a
> > different executor that is a first class concept in mesos, why is there a
> > objection? Renan's patch of replacing thermos executor is already in
> > aurora. However right now aurora can statically configure only one
> > executor. And so we want a job submission dynamically can call out the
> > executor type.
> > >> Please clarify your views.
> > >> Right now the docker integration story in mesos is weak (current and
> > also the new universal containerizer). We see much value in using
> executors
> > like
> > >> https://github.com/mesos/docker-compose-executor
> > >> With aurora jobs.
> > >>
> > >> Points 2 and points 3 make the custom executor story stronger. (For
> > example we will run both thermos and compose executor depending on job
> type)
> > >>
> > >>
> > >> Point 1, I agree. That is outside of aurora and most likely will be a
> > shim on top of generated go thrift bindings but a helper to easily test
> > custom executors.
> > >> Last time when we reviewed with aurora team, it came out current
> aurora
> > cli is tightly coupled with thermos objects for jobs and hence keep it as
> > is.
> > >>
> > >> Thx
> > >>
> > >>
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> > wrote:
> > >>>
> > >>> I am with Rick and David on this. The (1) and (2) feel like building
> a
> > >>> parallel universe within Aurora without a clear justification on the
> > >>> long-term benefits for the community.
> > >>>
> > >>> I am strong -1 on bringing yet another language into the Aurora
> > >>> ecosystem without identifying the strategic upside. As David
> > >>> suggested, any efforts towards building the first-class REST API
> > >>> instead would go a long way and will be much appreciated by everyone.
> > >>>
> > >>> I also don't see a reason to support multiple executors per task.
> That
> > >>> feels like re-creating thermos in a thermos-less world.
> > >>>
> > >>> As for (3), I'd like to see more details on the mechanics here but
> > >>> generally positive towards supporting more TaskInfo features in
> > >>> Aurora.
> > >>>
> > >>>
> > >>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> > >>>> I generally shy away from technical goals that are based on a choice
> > of language. What is it about a go client that can't be done by extending
> > the existing client?
> > >>>>
> > >>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <
> rdelval1@binghamton.edu>
> > wrote:
> > >>>>>
> > >>>>> Hi David,
> > >>>>>
> > >>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> > david@dmclaughlin.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> > >>>>>> meghdoot_b@yahoo.com.invalid> wrote:
> > >>>>>>
> > >>>>>>> Comments inline
> > >>>>>>>
> > >>>>>>>    From: David McLaughlin <dm...@apache.org>
> > >>>>>>> To: dev@aurora.apache.org
> > >>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> > >>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> > integrate
> > >>>>>>> mesos task related fields
> > >>>>>>>
> > >>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> > rdelval1@binghamton.edu>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hello all,
> > >>>>>>>>
> > >>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
> > I've
> > >>>>>> had
> > >>>>>>>> the pleasure of being part of the Aurora community for the last
> > year or
> > >>>>>>> so.
> > >>>>>>>>
> > >>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
> > With
> > >>>>>> the
> > >>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> > feature
> > >>>>>>> became
> > >>>>>>>> a reality and made into Aurora late last year. (Also got to show
> > an
> > >>>>>> early
> > >>>>>>>> beta at MesosCon!)
> > >>>>>>>>
> > >>>>>>>> For this year's summer, I have some new goals which I invite
> > anyone to
> > >>>>>>>> provide input on.
> > >>>>>>>>
> > >>>>>>>> 1. Create a Golang library that works as an alternative to
> > Aurora's
> > >>>>>>> Python
> > >>>>>>>> client and Pystachio. The initial goal of this library is to
> > support
> > >>>>>> the
> > >>>>>>>> most critical job related Thrift API's. (Admin operations can
> > continue
> > >>>>>> to
> > >>>>>>>> be done from the Aurora CLI). Since we support custom executors,
> > we
> > >>>>>> need
> > >>>>>>> a
> > >>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
> > >>>>>>>> configurations to Aurora (and by extension the custom executor).
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> You can easily add support for a different configuration format
> > without
> > >>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own
> > custom
> > >>>>>>> noun/verbs to the client or replacing existing commands. For
> > example, at
> > >>>>>>> Twitter we have our own version of 'aurora update start' that
> > plugs into
> > >>>>>> an
> > >>>>>>> internal Deploy Orchestration Service and we have a whole other
> > command
> > >>>>>> for
> > >>>>>>> federated deploys. I can show you how we do that.
> > >>>>>>>
> > >>>>>>> The first thing I thought of though was - this seems like a
> perfect
> > >>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
> > Having
> > >>>>>> that
> > >>>>>>> would make it trivial to do a CLI in any language you want, and
> the
> > >>>>>> Python
> > >>>>>>> CLI would really only be there to do Pystachio evaluation.
> > >>>>>>>>> CLI integrations is not useful for a lot of different reasons.
> > We need
> > >>>>>>> API's from other orchestration points from different components
> > and hence
> > >>>>>>> we would package it in a go library. Uber, I believe has taken
> the
> > same
> > >>>>>>> route (info from this mesoscon)
> > >>>>>>> We are not writing a replacement cli tool. As noted admin
> > operations
> > >>>>>> would
> > >>>>>>> heavily leverage current aurora cli.
> > >>>>>>> I think REST work has been postponed and attempted few times in
> > past. We
> > >>>>>>> cannot wait for that.
> > >>>>>>
> > >>>>>>
> > >>>>>> This is the dev list for contributions to the project, so I
> assumed
> > we were
> > >>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and
> > Thrift
> > >>>>>> has Go bindings so you're all set for whatever custom tooling
> you're
> > >>>>>> building. Please use the users list for feedback on that.
> > >>>>>
> > >>>>>
> > >>>>> While I'm very thankful for your input on the matter, my original
> > text very
> > >>>>> clearly said:
> > >>>>> "1. Create a Golang library that works as an *alternative* to
> > >>>>> Aurora's Python client and Pystachio."
> > >>>>>
> > >>>>> I believe every single item I have listed has the potential to
> > become a
> > >>>>> useful contribution to this project.
> > >>>>> Therefore, I feel that it is prudent continue the discussion here
> > (unless
> > >>>>> others feel similarly).
> > >>>>>
> > >>>>> -Renan
> > >>>>>
> > >>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> 2. Create support for using multiple executors within a single
> > Aurora
> > >>>>>>>> scheduler instance. As of right now, only a single executor can
> > exist
> > >>>>>> for
> > >>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
> > >>>>>> scheduler
> > >>>>>>>> to support multiple executors such that they can be used
> > alongside one
> > >>>>>>>> another, providing flexibility for running jobs.
> > >>>>>>>
> > >>>>>>> Just for my own understanding - what problems are you solving
> with
> > your
> > >>>>>>> custom executor? Sorry if you've explained this to the lists
> > before.
> > >>>>>>>
> > >>>>>>> I ask because I'd really like to see Aurora stop treating the
> > executor
> > >>>>>>> config as an opaque blob and being more opinionated. The
> > >>>>>> Thermos/Scheduler
> > >>>>>>> abstraction really hinders what type of user experience (UI) we
> can
> > >>>>>> build,
> > >>>>>>> and other frameworks do a much better job by being more
> > opinionated and
> > >>>>>>> pulling executor data into the scheduler UI.
> > >>>>>>>>> We probably don't need to debate on this point. Executor is a
> > first
> > >>>>>>> class mesos citizen and time is right for aurora to have good
> > support for
> > >>>>>>> it.In our case, we have kubernetes like pods modeled through
> > >>>>>> docker-compose
> > >>>>>>> and the executor manages that. Scheduler UI main features should
> > not get
> > >>>>>>> bogged down or be held back for a particular executor. That feels
> > >>>>>>> incredibly restrictive. If a special UI mode for thermos executor
> > is
> > >>>>>>> created, that should be fine. We do have to differentiate the
> > scheduler
> > >>>>>> and
> > >>>>>>> thermos and aurora team has done a great job of not hard coupling
> > the
> > >>>>>> two.
> > >>>>>>
> > >>>>>>
> > >>>>>>>>
> > >>>>>>>> 3. I'd like to add support for some first class Mesos task
> related
> > >>>>>>> fields.
> > >>>>>>>> These would be optional and/or have defaults so that the regular
> > Aurora
> > >>>>>>> CLI
> > >>>>>>>> does not break. One of the examples I'm interested in
> integrating
> > is
> > >>>>>> the
> > >>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> > sandbox that
> > >>>>>>> the
> > >>>>>>>> custom executor may need. (The executor path will never be
> > exposed as
> > >>>>>>> this
> > >>>>>>>> will be defined serverside and be static).
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> > passed to
> > >>>>>>> your executor? I have to admit I don't know enough about how the
> > Mesos
> > >>>>>>> fetcher works.
> > >>>>>>>>> Apache Mesos - Fetcher
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> We may add other first class fields.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> If anyone has any feedback on these, I'd be very glad to hear
> it.
> > >>>>>>>>
> > >>>>>>>> That's it for me for now, thanks!
> > >>>>>>>>
> > >>>>>>>> -Renan
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> |
> > >>>>>>> |
> > >>>>>>> |
> > >>>>>>> |  |    |
> > >>>>>>>
> > >>>>>>> |
> > >>>>>>>
> > >>>>>>> |
> > >>>>>>> |
> > >>>>>>> |  |
> > >>>>>>> Apache Mesos - Fetcher
> > >>>>>>> |  |
> > >>>>>>>
> > >>>>>>> |
> > >>>>>>>
> > >>>>>>> |
> > >>>>>>
> >
>
>
>




-- 
Cheers,

Zhitao Li

Aurora now supports multiple executors

[meghdoot bhattacharya <me...@yahoo.com.INVALID>]

Renaming the subject.
Renan's latest patch in 0.16 has completed the goal of implementing both 2 and 3. So, Aurora now can not only support custom executors but run multiple executors in the cluster!! Mesos fetcher has been integrated as well to help the custom executors have access to downloaded artifacts that it needs. We are going to add dedicated documentation on how to enable this in 0.16.
This was 2 summers in making with Renan's internship slots. Special thanks to Maxim, Joshua and Bill to make this happen and mentoring Renan in the process.


As for point 1, we are going to put out the golang client in few weeks to easily test different executors (including thermos) with Aurora with real examples. During the same time we will release production grade docker-compose executor simulating running container pods with Aurora.  
Expect a detailed blog in a month.
Thx
      From: David McLaughlin <dm...@apache.org>
 To: dev@aurora.apache.org 
 Sent: Monday, June 13, 2016 10:39 AM
 Subject: Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
   
Thanks, also +1 to supporting points 2 and 3.

On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid> wrote:

> Got it. Thx Max.
> Renan will starting working on 2 and 3, involving and updating you all.
>
> Thx
>
> Sent from my iPhone
>
> > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
> >
> > David pinged me offline suggesting I misread the proposal for (2) as
> > an attempt to support multiple executors per task rather than "per
> > cluster". If that's the case, I am happy to change my vote for (2) to
> > +1. I don't see much harm (aside from possible UI inconsistencies)
> > from multiple executor types sharing the same cluster.
> >
> >> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid> wrote:
> >> I will leave Renan for a detailed response.
> >>
> >> If aurora can schedule Job A with thermos and Job B that requires a
> different executor that is a first class concept in mesos, why is there a
> objection? Renan's patch of replacing thermos executor is already in
> aurora. However right now aurora can statically configure only one
> executor. And so we want a job submission dynamically can call out the
> executor type.
> >> Please clarify your views.
> >> Right now the docker integration story in mesos is weak (current and
> also the new universal containerizer). We see much value in using executors
> like
> >> https://github.com/mesos/docker-compose-executor
> >> With aurora jobs.
> >>
> >> Points 2 and points 3 make the custom executor story stronger. (For
> example we will run both thermos and compose executor depending on job type)
> >>
> >>
> >> Point 1, I agree. That is outside of aurora and most likely will be a
> shim on top of generated go thrift bindings but a helper to easily test
> custom executors.
> >> Last time when we reviewed with aurora team, it came out current aurora
> cli is tightly coupled with thermos objects for jobs and hence keep it as
> is.
> >>
> >> Thx
> >>
> >>
> >>
> >> Sent from my iPhone
> >>
> >>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
> >>>
> >>> I am with Rick and David on this. The (1) and (2) feel like building a
> >>> parallel universe within Aurora without a clear justification on the
> >>> long-term benefits for the community.
> >>>
> >>> I am strong -1 on bringing yet another language into the Aurora
> >>> ecosystem without identifying the strategic upside. As David
> >>> suggested, any efforts towards building the first-class REST API
> >>> instead would go a long way and will be much appreciated by everyone.
> >>>
> >>> I also don't see a reason to support multiple executors per task. That
> >>> feels like re-creating thermos in a thermos-less world.
> >>>
> >>> As for (3), I'd like to see more details on the mechanics here but
> >>> generally positive towards supporting more TaskInfo features in
> >>> Aurora.
> >>>
> >>>
> >>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> >>>> I generally shy away from technical goals that are based on a choice
> of language. What is it about a go client that can't be done by extending
> the existing client?
> >>>>
> >>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu>
> wrote:
> >>>>>
> >>>>> Hi David,
> >>>>>
> >>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> david@dmclaughlin.com>
> >>>>> wrote:
> >>>>>
> >>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> >>>>>> meghdoot_b@yahoo.com.invalid> wrote:
> >>>>>>
> >>>>>>> Comments inline
> >>>>>>>
> >>>>>>>    From: David McLaughlin <dm...@apache.org>
> >>>>>>> To: dev@aurora.apache.org
> >>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> >>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> integrate
> >>>>>>> mesos task related fields
> >>>>>>>
> >>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> rdelval1@binghamton.edu>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello all,
> >>>>>>>>
> >>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
> I've
> >>>>>> had
> >>>>>>>> the pleasure of being part of the Aurora community for the last
> year or
> >>>>>>> so.
> >>>>>>>>
> >>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
> With
> >>>>>> the
> >>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> feature
> >>>>>>> became
> >>>>>>>> a reality and made into Aurora late last year. (Also got to show
> an
> >>>>>> early
> >>>>>>>> beta at MesosCon!)
> >>>>>>>>
> >>>>>>>> For this year's summer, I have some new goals which I invite
> anyone to
> >>>>>>>> provide input on.
> >>>>>>>>
> >>>>>>>> 1. Create a Golang library that works as an alternative to
> Aurora's
> >>>>>>> Python
> >>>>>>>> client and Pystachio. The initial goal of this library is to
> support
> >>>>>> the
> >>>>>>>> most critical job related Thrift API's. (Admin operations can
> continue
> >>>>>> to
> >>>>>>>> be done from the Aurora CLI). Since we support custom executors,
> we
> >>>>>> need
> >>>>>>> a
> >>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
> >>>>>>>> configurations to Aurora (and by extension the custom executor).
> >>>>>>>
> >>>>>>>
> >>>>>>> You can easily add support for a different configuration format
> without
> >>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own
> custom
> >>>>>>> noun/verbs to the client or replacing existing commands. For
> example, at
> >>>>>>> Twitter we have our own version of 'aurora update start' that
> plugs into
> >>>>>> an
> >>>>>>> internal Deploy Orchestration Service and we have a whole other
> command
> >>>>>> for
> >>>>>>> federated deploys. I can show you how we do that.
> >>>>>>>
> >>>>>>> The first thing I thought of though was - this seems like a perfect
> >>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
> Having
> >>>>>> that
> >>>>>>> would make it trivial to do a CLI in any language you want, and the
> >>>>>> Python
> >>>>>>> CLI would really only be there to do Pystachio evaluation.
> >>>>>>>>> CLI integrations is not useful for a lot of different reasons.
> We need
> >>>>>>> API's from other orchestration points from different components
> and hence
> >>>>>>> we would package it in a go library. Uber, I believe has taken the
> same
> >>>>>>> route (info from this mesoscon)
> >>>>>>> We are not writing a replacement cli tool. As noted admin
> operations
> >>>>>> would
> >>>>>>> heavily leverage current aurora cli.
> >>>>>>> I think REST work has been postponed and attempted few times in
> past. We
> >>>>>>> cannot wait for that.
> >>>>>>
> >>>>>>
> >>>>>> This is the dev list for contributions to the project, so I assumed
> we were
> >>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and
> Thrift
> >>>>>> has Go bindings so you're all set for whatever custom tooling you're
> >>>>>> building. Please use the users list for feedback on that.
> >>>>>
> >>>>>
> >>>>> While I'm very thankful for your input on the matter, my original
> text very
> >>>>> clearly said:
> >>>>> "1. Create a Golang library that works as an *alternative* to
> >>>>> Aurora's Python client and Pystachio."
> >>>>>
> >>>>> I believe every single item I have listed has the potential to
> become a
> >>>>> useful contribution to this project.
> >>>>> Therefore, I feel that it is prudent continue the discussion here
> (unless
> >>>>> others feel similarly).
> >>>>>
> >>>>> -Renan
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> 2. Create support for using multiple executors within a single
> Aurora
> >>>>>>>> scheduler instance. As of right now, only a single executor can
> exist
> >>>>>> for
> >>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
> >>>>>> scheduler
> >>>>>>>> to support multiple executors such that they can be used
> alongside one
> >>>>>>>> another, providing flexibility for running jobs.
> >>>>>>>
> >>>>>>> Just for my own understanding - what problems are you solving with
> your
> >>>>>>> custom executor? Sorry if you've explained this to the lists
> before.
> >>>>>>>
> >>>>>>> I ask because I'd really like to see Aurora stop treating the
> executor
> >>>>>>> config as an opaque blob and being more opinionated. The
> >>>>>> Thermos/Scheduler
> >>>>>>> abstraction really hinders what type of user experience (UI) we can
> >>>>>> build,
> >>>>>>> and other frameworks do a much better job by being more
> opinionated and
> >>>>>>> pulling executor data into the scheduler UI.
> >>>>>>>>> We probably don't need to debate on this point. Executor is a
> first
> >>>>>>> class mesos citizen and time is right for aurora to have good
> support for
> >>>>>>> it.In our case, we have kubernetes like pods modeled through
> >>>>>> docker-compose
> >>>>>>> and the executor manages that. Scheduler UI main features should
> not get
> >>>>>>> bogged down or be held back for a particular executor. That feels
> >>>>>>> incredibly restrictive. If a special UI mode for thermos executor
> is
> >>>>>>> created, that should be fine. We do have to differentiate the
> scheduler
> >>>>>> and
> >>>>>>> thermos and aurora team has done a great job of not hard coupling
> the
> >>>>>> two.
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>> 3. I'd like to add support for some first class Mesos task related
> >>>>>>> fields.
> >>>>>>>> These would be optional and/or have defaults so that the regular
> Aurora
> >>>>>>> CLI
> >>>>>>>> does not break. One of the examples I'm interested in integrating
> is
> >>>>>> the
> >>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> sandbox that
> >>>>>>> the
> >>>>>>>> custom executor may need. (The executor path will never be
> exposed as
> >>>>>>> this
> >>>>>>>> will be defined serverside and be static).
> >>>>>>>
> >>>>>>>
> >>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> passed to
> >>>>>>> your executor? I have to admit I don't know enough about how the
> Mesos
> >>>>>>> fetcher works.
> >>>>>>>>> Apache Mesos - Fetcher
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> We may add other first class fields.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
> >>>>>>>>
> >>>>>>>> That's it for me for now, thanks!
> >>>>>>>>
> >>>>>>>> -Renan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |  |    |
> >>>>>>>
> >>>>>>> |
> >>>>>>>
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |  |
> >>>>>>> Apache Mesos - Fetcher
> >>>>>>> |  |
> >>>>>>>
> >>>>>>> |
> >>>>>>>
> >>>>>>> |
> >>>>>>
>


  

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[David McLaughlin <dm...@apache.org>]

Thanks, also +1 to supporting points 2 and 3.

On Mon, Jun 13, 2016 at 10:33 AM, <me...@yahoo.com.invalid> wrote:

> Got it. Thx Max.
> Renan will starting working on 2 and 3, involving and updating you all.
>
> Thx
>
> Sent from my iPhone
>
> > On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
> >
> > David pinged me offline suggesting I misread the proposal for (2) as
> > an attempt to support multiple executors per task rather than "per
> > cluster". If that's the case, I am happy to change my vote for (2) to
> > +1. I don't see much harm (aside from possible UI inconsistencies)
> > from multiple executor types sharing the same cluster.
> >
> >> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid> wrote:
> >> I will leave Renan for a detailed response.
> >>
> >> If aurora can schedule Job A with thermos and Job B that requires a
> different executor that is a first class concept in mesos, why is there a
> objection? Renan's patch of replacing thermos executor is already in
> aurora. However right now aurora can statically configure only one
> executor. And so we want a job submission dynamically can call out the
> executor type.
> >> Please clarify your views.
> >> Right now the docker integration story in mesos is weak (current and
> also the new universal containerizer). We see much value in using executors
> like
> >> https://github.com/mesos/docker-compose-executor
> >> With aurora jobs.
> >>
> >> Points 2 and points 3 make the custom executor story stronger. (For
> example we will run both thermos and compose executor depending on job type)
> >>
> >>
> >> Point 1, I agree. That is outside of aurora and most likely will be a
> shim on top of generated go thrift bindings but a helper to easily test
> custom executors.
> >> Last time when we reviewed with aurora team, it came out current aurora
> cli is tightly coupled with thermos objects for jobs and hence keep it as
> is.
> >>
> >> Thx
> >>
> >>
> >>
> >> Sent from my iPhone
> >>
> >>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org>
> wrote:
> >>>
> >>> I am with Rick and David on this. The (1) and (2) feel like building a
> >>> parallel universe within Aurora without a clear justification on the
> >>> long-term benefits for the community.
> >>>
> >>> I am strong -1 on bringing yet another language into the Aurora
> >>> ecosystem without identifying the strategic upside. As David
> >>> suggested, any efforts towards building the first-class REST API
> >>> instead would go a long way and will be much appreciated by everyone.
> >>>
> >>> I also don't see a reason to support multiple executors per task. That
> >>> feels like re-creating thermos in a thermos-less world.
> >>>
> >>> As for (3), I'd like to see more details on the mechanics here but
> >>> generally positive towards supporting more TaskInfo features in
> >>> Aurora.
> >>>
> >>>
> >>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> >>>> I generally shy away from technical goals that are based on a choice
> of language. What is it about a go client that can't be done by extending
> the existing client?
> >>>>
> >>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu>
> wrote:
> >>>>>
> >>>>> Hi David,
> >>>>>
> >>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <
> david@dmclaughlin.com>
> >>>>> wrote:
> >>>>>
> >>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> >>>>>> meghdoot_b@yahoo.com.invalid> wrote:
> >>>>>>
> >>>>>>> Comments inline
> >>>>>>>
> >>>>>>>    From: David McLaughlin <dm...@apache.org>
> >>>>>>> To: dev@aurora.apache.org
> >>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
> >>>>>>> Subject: Re: Golang Aurora lib, multiple executor support,
> integrate
> >>>>>>> mesos task related fields
> >>>>>>>
> >>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <
> rdelval1@binghamton.edu>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello all,
> >>>>>>>>
> >>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and
> I've
> >>>>>> had
> >>>>>>>> the pleasure of being part of the Aurora community for the last
> year or
> >>>>>>> so.
> >>>>>>>>
> >>>>>>>> Last year I worked to allow Aurora to utilize custom executors.
> With
> >>>>>> the
> >>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that
> feature
> >>>>>>> became
> >>>>>>>> a reality and made into Aurora late last year. (Also got to show
> an
> >>>>>> early
> >>>>>>>> beta at MesosCon!)
> >>>>>>>>
> >>>>>>>> For this year's summer, I have some new goals which I invite
> anyone to
> >>>>>>>> provide input on.
> >>>>>>>>
> >>>>>>>> 1. Create a Golang library that works as an alternative to
> Aurora's
> >>>>>>> Python
> >>>>>>>> client and Pystachio. The initial goal of this library is to
> support
> >>>>>> the
> >>>>>>>> most critical job related Thrift API's. (Admin operations can
> continue
> >>>>>> to
> >>>>>>>> be done from the Aurora CLI). Since we support custom executors,
> we
> >>>>>> need
> >>>>>>> a
> >>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
> >>>>>>>> configurations to Aurora (and by extension the custom executor).
> >>>>>>>
> >>>>>>>
> >>>>>>> You can easily add support for a different configuration format
> without
> >>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own
> custom
> >>>>>>> noun/verbs to the client or replacing existing commands. For
> example, at
> >>>>>>> Twitter we have our own version of 'aurora update start' that
> plugs into
> >>>>>> an
> >>>>>>> internal Deploy Orchestration Service and we have a whole other
> command
> >>>>>> for
> >>>>>>> federated deploys. I can show you how we do that.
> >>>>>>>
> >>>>>>> The first thing I thought of though was - this seems like a perfect
> >>>>>>> starting point to get serious about a HTTP+JSON API for Aurora.
> Having
> >>>>>> that
> >>>>>>> would make it trivial to do a CLI in any language you want, and the
> >>>>>> Python
> >>>>>>> CLI would really only be there to do Pystachio evaluation.
> >>>>>>>>> CLI integrations is not useful for a lot of different reasons.
> We need
> >>>>>>> API's from other orchestration points from different components
> and hence
> >>>>>>> we would package it in a go library. Uber, I believe has taken the
> same
> >>>>>>> route (info from this mesoscon)
> >>>>>>> We are not writing a replacement cli tool. As noted admin
> operations
> >>>>>> would
> >>>>>>> heavily leverage current aurora cli.
> >>>>>>> I think REST work has been postponed and attempted few times in
> past. We
> >>>>>>> cannot wait for that.
> >>>>>>
> >>>>>>
> >>>>>> This is the dev list for contributions to the project, so I assumed
> we were
> >>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and
> Thrift
> >>>>>> has Go bindings so you're all set for whatever custom tooling you're
> >>>>>> building. Please use the users list for feedback on that.
> >>>>>
> >>>>>
> >>>>> While I'm very thankful for your input on the matter, my original
> text very
> >>>>> clearly said:
> >>>>> "1. Create a Golang library that works as an *alternative* to
> >>>>> Aurora's Python client and Pystachio."
> >>>>>
> >>>>> I believe every single item I have listed has the potential to
> become a
> >>>>> useful contribution to this project.
> >>>>> Therefore, I feel that it is prudent continue the discussion here
> (unless
> >>>>> others feel similarly).
> >>>>>
> >>>>> -Renan
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> 2. Create support for using multiple executors within a single
> Aurora
> >>>>>>>> scheduler instance. As of right now, only a single executor can
> exist
> >>>>>> for
> >>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
> >>>>>> scheduler
> >>>>>>>> to support multiple executors such that they can be used
> alongside one
> >>>>>>>> another, providing flexibility for running jobs.
> >>>>>>>
> >>>>>>> Just for my own understanding - what problems are you solving with
> your
> >>>>>>> custom executor? Sorry if you've explained this to the lists
> before.
> >>>>>>>
> >>>>>>> I ask because I'd really like to see Aurora stop treating the
> executor
> >>>>>>> config as an opaque blob and being more opinionated. The
> >>>>>> Thermos/Scheduler
> >>>>>>> abstraction really hinders what type of user experience (UI) we can
> >>>>>> build,
> >>>>>>> and other frameworks do a much better job by being more
> opinionated and
> >>>>>>> pulling executor data into the scheduler UI.
> >>>>>>>>> We probably don't need to debate on this point. Executor is a
> first
> >>>>>>> class mesos citizen and time is right for aurora to have good
> support for
> >>>>>>> it.In our case, we have kubernetes like pods modeled through
> >>>>>> docker-compose
> >>>>>>> and the executor manages that. Scheduler UI main features should
> not get
> >>>>>>> bogged down or be held back for a particular executor. That feels
> >>>>>>> incredibly restrictive. If a special UI mode for thermos executor
> is
> >>>>>>> created, that should be fine. We do have to differentiate the
> scheduler
> >>>>>> and
> >>>>>>> thermos and aurora team has done a great job of not hard coupling
> the
> >>>>>> two.
> >>>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>>> 3. I'd like to add support for some first class Mesos task related
> >>>>>>> fields.
> >>>>>>>> These would be optional and/or have defaults so that the regular
> Aurora
> >>>>>>> CLI
> >>>>>>>> does not break. One of the examples I'm interested in integrating
> is
> >>>>>> the
> >>>>>>>> Mesos Fetcher so that resources can be downloaded into the
> sandbox that
> >>>>>>> the
> >>>>>>>> custom executor may need. (The executor path will never be
> exposed as
> >>>>>>> this
> >>>>>>>> will be defined serverside and be static).
> >>>>>>>
> >>>>>>>
> >>>>>>> Wouldn't the mesos-fetcher call just be another process to be
> passed to
> >>>>>>> your executor? I have to admit I don't know enough about how the
> Mesos
> >>>>>>> fetcher works.
> >>>>>>>>> Apache Mesos - Fetcher
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> We may add other first class fields.
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
> >>>>>>>>
> >>>>>>>> That's it for me for now, thanks!
> >>>>>>>>
> >>>>>>>> -Renan
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |   |    |
> >>>>>>>
> >>>>>>> |
> >>>>>>>
> >>>>>>> |
> >>>>>>> |
> >>>>>>> |   |
> >>>>>>> Apache Mesos - Fetcher
> >>>>>>> |   |
> >>>>>>>
> >>>>>>> |
> >>>>>>>
> >>>>>>> |
> >>>>>>
>

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[me...@yahoo.com.INVALID]

Got it. Thx Max.
Renan will starting working on 2 and 3, involving and updating you all.

Thx

Sent from my iPhone

> On Jun 13, 2016, at 10:09 AM, Maxim Khutornenko <ma...@apache.org> wrote:
> 
> David pinged me offline suggesting I misread the proposal for (2) as
> an attempt to support multiple executors per task rather than "per
> cluster". If that's the case, I am happy to change my vote for (2) to
> +1. I don't see much harm (aside from possible UI inconsistencies)
> from multiple executor types sharing the same cluster.
> 
>> On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid> wrote:
>> I will leave Renan for a detailed response.
>> 
>> If aurora can schedule Job A with thermos and Job B that requires a different executor that is a first class concept in mesos, why is there a objection? Renan's patch of replacing thermos executor is already in aurora. However right now aurora can statically configure only one executor. And so we want a job submission dynamically can call out the executor type.
>> Please clarify your views.
>> Right now the docker integration story in mesos is weak (current and also the new universal containerizer). We see much value in using executors like
>> https://github.com/mesos/docker-compose-executor
>> With aurora jobs.
>> 
>> Points 2 and points 3 make the custom executor story stronger. (For example we will run both thermos and compose executor depending on job type)
>> 
>> 
>> Point 1, I agree. That is outside of aurora and most likely will be a shim on top of generated go thrift bindings but a helper to easily test custom executors.
>> Last time when we reviewed with aurora team, it came out current aurora cli is tightly coupled with thermos objects for jobs and hence keep it as is.
>> 
>> Thx
>> 
>> 
>> 
>> Sent from my iPhone
>> 
>>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> wrote:
>>> 
>>> I am with Rick and David on this. The (1) and (2) feel like building a
>>> parallel universe within Aurora without a clear justification on the
>>> long-term benefits for the community.
>>> 
>>> I am strong -1 on bringing yet another language into the Aurora
>>> ecosystem without identifying the strategic upside. As David
>>> suggested, any efforts towards building the first-class REST API
>>> instead would go a long way and will be much appreciated by everyone.
>>> 
>>> I also don't see a reason to support multiple executors per task. That
>>> feels like re-creating thermos in a thermos-less world.
>>> 
>>> As for (3), I'd like to see more details on the mechanics here but
>>> generally positive towards supporting more TaskInfo features in
>>> Aurora.
>>> 
>>> 
>>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
>>>> I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?
>>>> 
>>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu> wrote:
>>>>> 
>>>>> Hi David,
>>>>> 
>>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
>>>>> wrote:
>>>>> 
>>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>>>> 
>>>>>>> Comments inline
>>>>>>> 
>>>>>>>    From: David McLaughlin <dm...@apache.org>
>>>>>>> To: dev@aurora.apache.org
>>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>>>>>> mesos task related fields
>>>>>>> 
>>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hello all,
>>>>>>>> 
>>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>>>>>> had
>>>>>>>> the pleasure of being part of the Aurora community for the last year or
>>>>>>> so.
>>>>>>>> 
>>>>>>>> Last year I worked to allow Aurora to utilize custom executors. With
>>>>>> the
>>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>>>>>> became
>>>>>>>> a reality and made into Aurora late last year. (Also got to show an
>>>>>> early
>>>>>>>> beta at MesosCon!)
>>>>>>>> 
>>>>>>>> For this year's summer, I have some new goals which I invite anyone to
>>>>>>>> provide input on.
>>>>>>>> 
>>>>>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>>>>>> Python
>>>>>>>> client and Pystachio. The initial goal of this library is to support
>>>>>> the
>>>>>>>> most critical job related Thrift API's. (Admin operations can continue
>>>>>> to
>>>>>>>> be done from the Aurora CLI). Since we support custom executors, we
>>>>>> need
>>>>>>> a
>>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>>>>> configurations to Aurora (and by extension the custom executor).
>>>>>>> 
>>>>>>> 
>>>>>>> You can easily add support for a different configuration format without
>>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>>>>>> noun/verbs to the client or replacing existing commands. For example, at
>>>>>>> Twitter we have our own version of 'aurora update start' that plugs into
>>>>>> an
>>>>>>> internal Deploy Orchestration Service and we have a whole other command
>>>>>> for
>>>>>>> federated deploys. I can show you how we do that.
>>>>>>> 
>>>>>>> The first thing I thought of though was - this seems like a perfect
>>>>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>>>>>> that
>>>>>>> would make it trivial to do a CLI in any language you want, and the
>>>>>> Python
>>>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>>>>>> API's from other orchestration points from different components and hence
>>>>>>> we would package it in a go library. Uber, I believe has taken the same
>>>>>>> route (info from this mesoscon)
>>>>>>> We are not writing a replacement cli tool. As noted admin operations
>>>>>> would
>>>>>>> heavily leverage current aurora cli.
>>>>>>> I think REST work has been postponed and attempted few times in past. We
>>>>>>> cannot wait for that.
>>>>>> 
>>>>>> 
>>>>>> This is the dev list for contributions to the project, so I assumed we were
>>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>>>>>> has Go bindings so you're all set for whatever custom tooling you're
>>>>>> building. Please use the users list for feedback on that.
>>>>> 
>>>>> 
>>>>> While I'm very thankful for your input on the matter, my original text very
>>>>> clearly said:
>>>>> "1. Create a Golang library that works as an *alternative* to
>>>>> Aurora's Python client and Pystachio."
>>>>> 
>>>>> I believe every single item I have listed has the potential to become a
>>>>> useful contribution to this project.
>>>>> Therefore, I feel that it is prudent continue the discussion here (unless
>>>>> others feel similarly).
>>>>> 
>>>>> -Renan
>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 2. Create support for using multiple executors within a single Aurora
>>>>>>>> scheduler instance. As of right now, only a single executor can exist
>>>>>> for
>>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>>>> scheduler
>>>>>>>> to support multiple executors such that they can be used alongside one
>>>>>>>> another, providing flexibility for running jobs.
>>>>>>> 
>>>>>>> Just for my own understanding - what problems are you solving with your
>>>>>>> custom executor? Sorry if you've explained this to the lists before.
>>>>>>> 
>>>>>>> I ask because I'd really like to see Aurora stop treating the executor
>>>>>>> config as an opaque blob and being more opinionated. The
>>>>>> Thermos/Scheduler
>>>>>>> abstraction really hinders what type of user experience (UI) we can
>>>>>> build,
>>>>>>> and other frameworks do a much better job by being more opinionated and
>>>>>>> pulling executor data into the scheduler UI.
>>>>>>>>> We probably don't need to debate on this point. Executor is a first
>>>>>>> class mesos citizen and time is right for aurora to have good support for
>>>>>>> it.In our case, we have kubernetes like pods modeled through
>>>>>> docker-compose
>>>>>>> and the executor manages that. Scheduler UI main features should not get
>>>>>>> bogged down or be held back for a particular executor. That feels
>>>>>>> incredibly restrictive. If a special UI mode for thermos executor is
>>>>>>> created, that should be fine. We do have to differentiate the scheduler
>>>>>> and
>>>>>>> thermos and aurora team has done a great job of not hard coupling the
>>>>>> two.
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> 3. I'd like to add support for some first class Mesos task related
>>>>>>> fields.
>>>>>>>> These would be optional and/or have defaults so that the regular Aurora
>>>>>>> CLI
>>>>>>>> does not break. One of the examples I'm interested in integrating is
>>>>>> the
>>>>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>>>>>> the
>>>>>>>> custom executor may need. (The executor path will never be exposed as
>>>>>>> this
>>>>>>>> will be defined serverside and be static).
>>>>>>> 
>>>>>>> 
>>>>>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>>>>>> your executor? I have to admit I don't know enough about how the Mesos
>>>>>>> fetcher works.
>>>>>>>>> Apache Mesos - Fetcher
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> We may add other first class fields.
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>>>>> 
>>>>>>>> That's it for me for now, thanks!
>>>>>>>> 
>>>>>>>> -Renan
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> |
>>>>>>> |
>>>>>>> |
>>>>>>> |   |    |
>>>>>>> 
>>>>>>> |
>>>>>>> 
>>>>>>> |
>>>>>>> |
>>>>>>> |   |
>>>>>>> Apache Mesos - Fetcher
>>>>>>> |   |
>>>>>>> 
>>>>>>> |
>>>>>>> 
>>>>>>> |
>>>>>> 

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[Maxim Khutornenko <ma...@apache.org>]

David pinged me offline suggesting I misread the proposal for (2) as
an attempt to support multiple executors per task rather than "per
cluster". If that's the case, I am happy to change my vote for (2) to
+1. I don't see much harm (aside from possible UI inconsistencies)
from multiple executor types sharing the same cluster.

On Mon, Jun 13, 2016 at 9:54 AM,  <me...@yahoo.com.invalid> wrote:
> I will leave Renan for a detailed response.
>
> If aurora can schedule Job A with thermos and Job B that requires a different executor that is a first class concept in mesos, why is there a objection? Renan's patch of replacing thermos executor is already in aurora. However right now aurora can statically configure only one executor. And so we want a job submission dynamically can call out the executor type.
> Please clarify your views.
> Right now the docker integration story in mesos is weak (current and also the new universal containerizer). We see much value in using executors like
> https://github.com/mesos/docker-compose-executor
> With aurora jobs.
>
> Points 2 and points 3 make the custom executor story stronger. (For example we will run both thermos and compose executor depending on job type)
>
>
> Point 1, I agree. That is outside of aurora and most likely will be a shim on top of generated go thrift bindings but a helper to easily test custom executors.
> Last time when we reviewed with aurora team, it came out current aurora cli is tightly coupled with thermos objects for jobs and hence keep it as is.
>
> Thx
>
>
>
> Sent from my iPhone
>
>> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> wrote:
>>
>> I am with Rick and David on this. The (1) and (2) feel like building a
>> parallel universe within Aurora without a clear justification on the
>> long-term benefits for the community.
>>
>> I am strong -1 on bringing yet another language into the Aurora
>> ecosystem without identifying the strategic upside. As David
>> suggested, any efforts towards building the first-class REST API
>> instead would go a long way and will be much appreciated by everyone.
>>
>> I also don't see a reason to support multiple executors per task. That
>> feels like re-creating thermos in a thermos-less world.
>>
>> As for (3), I'd like to see more details on the mechanics here but
>> generally positive towards supporting more TaskInfo features in
>> Aurora.
>>
>>
>>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
>>> I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?
>>>
>>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu> wrote:
>>>>
>>>> Hi David,
>>>>
>>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
>>>> wrote:
>>>>
>>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>>>
>>>>>> Comments inline
>>>>>>
>>>>>>     From: David McLaughlin <dm...@apache.org>
>>>>>> To: dev@aurora.apache.org
>>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>>>>> mesos task related fields
>>>>>>
>>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>>>>> had
>>>>>>> the pleasure of being part of the Aurora community for the last year or
>>>>>> so.
>>>>>>>
>>>>>>> Last year I worked to allow Aurora to utilize custom executors. With
>>>>> the
>>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>>>>> became
>>>>>>> a reality and made into Aurora late last year. (Also got to show an
>>>>> early
>>>>>>> beta at MesosCon!)
>>>>>>>
>>>>>>> For this year's summer, I have some new goals which I invite anyone to
>>>>>>> provide input on.
>>>>>>>
>>>>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>>>>> Python
>>>>>>> client and Pystachio. The initial goal of this library is to support
>>>>> the
>>>>>>> most critical job related Thrift API's. (Admin operations can continue
>>>>> to
>>>>>>> be done from the Aurora CLI). Since we support custom executors, we
>>>>> need
>>>>>> a
>>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>>>> configurations to Aurora (and by extension the custom executor).
>>>>>>
>>>>>>
>>>>>> You can easily add support for a different configuration format without
>>>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>>>>> noun/verbs to the client or replacing existing commands. For example, at
>>>>>> Twitter we have our own version of 'aurora update start' that plugs into
>>>>> an
>>>>>> internal Deploy Orchestration Service and we have a whole other command
>>>>> for
>>>>>> federated deploys. I can show you how we do that.
>>>>>>
>>>>>> The first thing I thought of though was - this seems like a perfect
>>>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>>>>> that
>>>>>> would make it trivial to do a CLI in any language you want, and the
>>>>> Python
>>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>>>>> API's from other orchestration points from different components and hence
>>>>>> we would package it in a go library. Uber, I believe has taken the same
>>>>>> route (info from this mesoscon)
>>>>>> We are not writing a replacement cli tool. As noted admin operations
>>>>> would
>>>>>> heavily leverage current aurora cli.
>>>>>> I think REST work has been postponed and attempted few times in past. We
>>>>>> cannot wait for that.
>>>>>
>>>>>
>>>>> This is the dev list for contributions to the project, so I assumed we were
>>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>>>>> has Go bindings so you're all set for whatever custom tooling you're
>>>>> building. Please use the users list for feedback on that.
>>>>
>>>>
>>>> While I'm very thankful for your input on the matter, my original text very
>>>> clearly said:
>>>> "1. Create a Golang library that works as an *alternative* to
>>>> Aurora's Python client and Pystachio."
>>>>
>>>> I believe every single item I have listed has the potential to become a
>>>> useful contribution to this project.
>>>> Therefore, I feel that it is prudent continue the discussion here (unless
>>>> others feel similarly).
>>>>
>>>> -Renan
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2. Create support for using multiple executors within a single Aurora
>>>>>>> scheduler instance. As of right now, only a single executor can exist
>>>>> for
>>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>>> scheduler
>>>>>>> to support multiple executors such that they can be used alongside one
>>>>>>> another, providing flexibility for running jobs.
>>>>>>
>>>>>> Just for my own understanding - what problems are you solving with your
>>>>>> custom executor? Sorry if you've explained this to the lists before.
>>>>>>
>>>>>> I ask because I'd really like to see Aurora stop treating the executor
>>>>>> config as an opaque blob and being more opinionated. The
>>>>> Thermos/Scheduler
>>>>>> abstraction really hinders what type of user experience (UI) we can
>>>>> build,
>>>>>> and other frameworks do a much better job by being more opinionated and
>>>>>> pulling executor data into the scheduler UI.
>>>>>>>> We probably don't need to debate on this point. Executor is a first
>>>>>> class mesos citizen and time is right for aurora to have good support for
>>>>>> it.In our case, we have kubernetes like pods modeled through
>>>>> docker-compose
>>>>>> and the executor manages that. Scheduler UI main features should not get
>>>>>> bogged down or be held back for a particular executor. That feels
>>>>>> incredibly restrictive. If a special UI mode for thermos executor is
>>>>>> created, that should be fine. We do have to differentiate the scheduler
>>>>> and
>>>>>> thermos and aurora team has done a great job of not hard coupling the
>>>>> two.
>>>>>
>>>>>
>>>>>>>
>>>>>>> 3. I'd like to add support for some first class Mesos task related
>>>>>> fields.
>>>>>>> These would be optional and/or have defaults so that the regular Aurora
>>>>>> CLI
>>>>>>> does not break. One of the examples I'm interested in integrating is
>>>>> the
>>>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>>>>> the
>>>>>>> custom executor may need. (The executor path will never be exposed as
>>>>>> this
>>>>>>> will be defined serverside and be static).
>>>>>>
>>>>>>
>>>>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>>>>> your executor? I have to admit I don't know enough about how the Mesos
>>>>>> fetcher works.
>>>>>>>> Apache Mesos - Fetcher
>>>>>
>>>>>
>>>>>
>>>>>> We may add other first class fields.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>>>>
>>>>>>> That's it for me for now, thanks!
>>>>>>>
>>>>>>> -Renan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |   |    |
>>>>>>
>>>>>>  |
>>>>>>
>>>>>> |
>>>>>> |
>>>>>> |   |
>>>>>> Apache Mesos - Fetcher
>>>>>>  |   |
>>>>>>
>>>>>> |
>>>>>>
>>>>>> |
>>>>>

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[me...@yahoo.com.INVALID]

I will leave Renan for a detailed response.

If aurora can schedule Job A with thermos and Job B that requires a different executor that is a first class concept in mesos, why is there a objection? Renan's patch of replacing thermos executor is already in aurora. However right now aurora can statically configure only one executor. And so we want a job submission dynamically can call out the executor type. 
Please clarify your views.
Right now the docker integration story in mesos is weak (current and also the new universal containerizer). We see much value in using executors like
https://github.com/mesos/docker-compose-executor
With aurora jobs.

Points 2 and points 3 make the custom executor story stronger. (For example we will run both thermos and compose executor depending on job type)


Point 1, I agree. That is outside of aurora and most likely will be a shim on top of generated go thrift bindings but a helper to easily test custom executors.
Last time when we reviewed with aurora team, it came out current aurora cli is tightly coupled with thermos objects for jobs and hence keep it as is.

Thx



Sent from my iPhone

> On Jun 13, 2016, at 9:19 AM, Maxim Khutornenko <ma...@apache.org> wrote:
> 
> I am with Rick and David on this. The (1) and (2) feel like building a
> parallel universe within Aurora without a clear justification on the
> long-term benefits for the community.
> 
> I am strong -1 on bringing yet another language into the Aurora
> ecosystem without identifying the strategic upside. As David
> suggested, any efforts towards building the first-class REST API
> instead would go a long way and will be much appreciated by everyone.
> 
> I also don't see a reason to support multiple executors per task. That
> feels like re-creating thermos in a thermos-less world.
> 
> As for (3), I'd like to see more details on the mechanics here but
> generally positive towards supporting more TaskInfo features in
> Aurora.
> 
> 
>> On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
>> I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?
>> 
>>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu> wrote:
>>> 
>>> Hi David,
>>> 
>>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
>>> wrote:
>>> 
>>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>> 
>>>>> Comments inline
>>>>> 
>>>>>     From: David McLaughlin <dm...@apache.org>
>>>>> To: dev@aurora.apache.org
>>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>>>> mesos task related fields
>>>>> 
>>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
>>>>> wrote:
>>>>> 
>>>>>> Hello all,
>>>>>> 
>>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>>>> had
>>>>>> the pleasure of being part of the Aurora community for the last year or
>>>>> so.
>>>>>> 
>>>>>> Last year I worked to allow Aurora to utilize custom executors. With
>>>> the
>>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>>>> became
>>>>>> a reality and made into Aurora late last year. (Also got to show an
>>>> early
>>>>>> beta at MesosCon!)
>>>>>> 
>>>>>> For this year's summer, I have some new goals which I invite anyone to
>>>>>> provide input on.
>>>>>> 
>>>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>>>> Python
>>>>>> client and Pystachio. The initial goal of this library is to support
>>>> the
>>>>>> most critical job related Thrift API's. (Admin operations can continue
>>>> to
>>>>>> be done from the Aurora CLI). Since we support custom executors, we
>>>> need
>>>>> a
>>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>>> configurations to Aurora (and by extension the custom executor).
>>>>> 
>>>>> 
>>>>> You can easily add support for a different configuration format without
>>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>>>> noun/verbs to the client or replacing existing commands. For example, at
>>>>> Twitter we have our own version of 'aurora update start' that plugs into
>>>> an
>>>>> internal Deploy Orchestration Service and we have a whole other command
>>>> for
>>>>> federated deploys. I can show you how we do that.
>>>>> 
>>>>> The first thing I thought of though was - this seems like a perfect
>>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>>>> that
>>>>> would make it trivial to do a CLI in any language you want, and the
>>>> Python
>>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>>>> API's from other orchestration points from different components and hence
>>>>> we would package it in a go library. Uber, I believe has taken the same
>>>>> route (info from this mesoscon)
>>>>> We are not writing a replacement cli tool. As noted admin operations
>>>> would
>>>>> heavily leverage current aurora cli.
>>>>> I think REST work has been postponed and attempted few times in past. We
>>>>> cannot wait for that.
>>>> 
>>>> 
>>>> This is the dev list for contributions to the project, so I assumed we were
>>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>>>> has Go bindings so you're all set for whatever custom tooling you're
>>>> building. Please use the users list for feedback on that.
>>> 
>>> 
>>> While I'm very thankful for your input on the matter, my original text very
>>> clearly said:
>>> "1. Create a Golang library that works as an *alternative* to
>>> Aurora's Python client and Pystachio."
>>> 
>>> I believe every single item I have listed has the potential to become a
>>> useful contribution to this project.
>>> Therefore, I feel that it is prudent continue the discussion here (unless
>>> others feel similarly).
>>> 
>>> -Renan
>>> 
>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 2. Create support for using multiple executors within a single Aurora
>>>>>> scheduler instance. As of right now, only a single executor can exist
>>>> for
>>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>>> scheduler
>>>>>> to support multiple executors such that they can be used alongside one
>>>>>> another, providing flexibility for running jobs.
>>>>> 
>>>>> Just for my own understanding - what problems are you solving with your
>>>>> custom executor? Sorry if you've explained this to the lists before.
>>>>> 
>>>>> I ask because I'd really like to see Aurora stop treating the executor
>>>>> config as an opaque blob and being more opinionated. The
>>>> Thermos/Scheduler
>>>>> abstraction really hinders what type of user experience (UI) we can
>>>> build,
>>>>> and other frameworks do a much better job by being more opinionated and
>>>>> pulling executor data into the scheduler UI.
>>>>>>> We probably don't need to debate on this point. Executor is a first
>>>>> class mesos citizen and time is right for aurora to have good support for
>>>>> it.In our case, we have kubernetes like pods modeled through
>>>> docker-compose
>>>>> and the executor manages that. Scheduler UI main features should not get
>>>>> bogged down or be held back for a particular executor. That feels
>>>>> incredibly restrictive. If a special UI mode for thermos executor is
>>>>> created, that should be fine. We do have to differentiate the scheduler
>>>> and
>>>>> thermos and aurora team has done a great job of not hard coupling the
>>>> two.
>>>> 
>>>> 
>>>>>> 
>>>>>> 3. I'd like to add support for some first class Mesos task related
>>>>> fields.
>>>>>> These would be optional and/or have defaults so that the regular Aurora
>>>>> CLI
>>>>>> does not break. One of the examples I'm interested in integrating is
>>>> the
>>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>>>> the
>>>>>> custom executor may need. (The executor path will never be exposed as
>>>>> this
>>>>>> will be defined serverside and be static).
>>>>> 
>>>>> 
>>>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>>>> your executor? I have to admit I don't know enough about how the Mesos
>>>>> fetcher works.
>>>>>>> Apache Mesos - Fetcher
>>>> 
>>>> 
>>>> 
>>>>> We may add other first class fields.
>>>>> 
>>>>> 
>>>>>> 
>>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>>> 
>>>>>> That's it for me for now, thanks!
>>>>>> 
>>>>>> -Renan
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> |
>>>>> |
>>>>> |
>>>>> |   |    |
>>>>> 
>>>>>  |
>>>>> 
>>>>> |
>>>>> |
>>>>> |   |
>>>>> Apache Mesos - Fetcher
>>>>>  |   |
>>>>> 
>>>>> |
>>>>> 
>>>>> |
>>>> 

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[Maxim Khutornenko <ma...@apache.org>]

I am with Rick and David on this. The (1) and (2) feel like building a
parallel universe within Aurora without a clear justification on the
long-term benefits for the community.

I am strong -1 on bringing yet another language into the Aurora
ecosystem without identifying the strategic upside. As David
suggested, any efforts towards building the first-class REST API
instead would go a long way and will be much appreciated by everyone.

I also don't see a reason to support multiple executors per task. That
feels like re-creating thermos in a thermos-less world.

As for (3), I'd like to see more details on the mechanics here but
generally positive towards supporting more TaskInfo features in
Aurora.


On Sun, Jun 12, 2016 at 11:46 AM,  <ri...@chartbeat.com> wrote:
> I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?
>
>> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu> wrote:
>>
>> Hi David,
>>
>> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
>> wrote:
>>
>>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>>> meghdoot_b@yahoo.com.invalid> wrote:
>>>
>>>> Comments inline
>>>>
>>>>      From: David McLaughlin <dm...@apache.org>
>>>> To: dev@aurora.apache.org
>>>> Sent: Thursday, June 9, 2016 5:13 PM
>>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>>> mesos task related fields
>>>>
>>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
>>>> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>>> had
>>>>> the pleasure of being part of the Aurora community for the last year or
>>>> so.
>>>>>
>>>>> Last year I worked to allow Aurora to utilize custom executors. With
>>> the
>>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>>> became
>>>>> a reality and made into Aurora late last year. (Also got to show an
>>> early
>>>>> beta at MesosCon!)
>>>>>
>>>>> For this year's summer, I have some new goals which I invite anyone to
>>>>> provide input on.
>>>>>
>>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>>> Python
>>>>> client and Pystachio. The initial goal of this library is to support
>>> the
>>>>> most critical job related Thrift API's. (Admin operations can continue
>>> to
>>>>> be done from the Aurora CLI). Since we support custom executors, we
>>> need
>>>> a
>>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>>> configurations to Aurora (and by extension the custom executor).
>>>>
>>>>
>>>> You can easily add support for a different configuration format without
>>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>>> noun/verbs to the client or replacing existing commands. For example, at
>>>> Twitter we have our own version of 'aurora update start' that plugs into
>>> an
>>>> internal Deploy Orchestration Service and we have a whole other command
>>> for
>>>> federated deploys. I can show you how we do that.
>>>>
>>>> The first thing I thought of though was - this seems like a perfect
>>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>>> that
>>>> would make it trivial to do a CLI in any language you want, and the
>>> Python
>>>> CLI would really only be there to do Pystachio evaluation.
>>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>>> API's from other orchestration points from different components and hence
>>>> we would package it in a go library. Uber, I believe has taken the same
>>>> route (info from this mesoscon)
>>>> We are not writing a replacement cli tool. As noted admin operations
>>> would
>>>> heavily leverage current aurora cli.
>>>> I think REST work has been postponed and attempted few times in past. We
>>>> cannot wait for that.
>>>
>>>
>>> This is the dev list for contributions to the project, so I assumed we were
>>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>>> has Go bindings so you're all set for whatever custom tooling you're
>>> building. Please use the users list for feedback on that.
>>
>>
>> While I'm very thankful for your input on the matter, my original text very
>> clearly said:
>> "1. Create a Golang library that works as an *alternative* to
>> Aurora's Python client and Pystachio."
>>
>> I believe every single item I have listed has the potential to become a
>> useful contribution to this project.
>> Therefore, I feel that it is prudent continue the discussion here (unless
>> others feel similarly).
>>
>> -Renan
>>
>>>
>>>>
>>>>
>>>>>
>>>>> 2. Create support for using multiple executors within a single Aurora
>>>>> scheduler instance. As of right now, only a single executor can exist
>>> for
>>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>>> scheduler
>>>>> to support multiple executors such that they can be used alongside one
>>>>> another, providing flexibility for running jobs.
>>>>
>>>> Just for my own understanding - what problems are you solving with your
>>>> custom executor? Sorry if you've explained this to the lists before.
>>>>
>>>> I ask because I'd really like to see Aurora stop treating the executor
>>>> config as an opaque blob and being more opinionated. The
>>> Thermos/Scheduler
>>>> abstraction really hinders what type of user experience (UI) we can
>>> build,
>>>> and other frameworks do a much better job by being more opinionated and
>>>> pulling executor data into the scheduler UI.
>>>>>> We probably don't need to debate on this point. Executor is a first
>>>> class mesos citizen and time is right for aurora to have good support for
>>>> it.In our case, we have kubernetes like pods modeled through
>>> docker-compose
>>>> and the executor manages that. Scheduler UI main features should not get
>>>> bogged down or be held back for a particular executor. That feels
>>>> incredibly restrictive. If a special UI mode for thermos executor is
>>>> created, that should be fine. We do have to differentiate the scheduler
>>> and
>>>> thermos and aurora team has done a great job of not hard coupling the
>>> two.
>>>
>>>
>>>>>
>>>>> 3. I'd like to add support for some first class Mesos task related
>>>> fields.
>>>>> These would be optional and/or have defaults so that the regular Aurora
>>>> CLI
>>>>> does not break. One of the examples I'm interested in integrating is
>>> the
>>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>>> the
>>>>> custom executor may need. (The executor path will never be exposed as
>>>> this
>>>>> will be defined serverside and be static).
>>>>
>>>>
>>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>>> your executor? I have to admit I don't know enough about how the Mesos
>>>> fetcher works.
>>>>>> Apache Mesos - Fetcher
>>>
>>>
>>>
>>>> We may add other first class fields.
>>>>
>>>>
>>>>>
>>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>>>
>>>>> That's it for me for now, thanks!
>>>>>
>>>>> -Renan
>>>>
>>>>
>>>>
>>>>
>>>> |
>>>> |
>>>> |
>>>> |   |    |
>>>>
>>>>   |
>>>>
>>>>  |
>>>> |
>>>> |   |
>>>> Apache Mesos - Fetcher
>>>>   |   |
>>>>
>>>>  |
>>>>
>>>>  |
>>>

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[ri...@chartbeat.com]

I generally shy away from technical goals that are based on a choice of language. What is it about a go client that can't be done by extending the existing client?

> On Jun 12, 2016, at 2:00 PM, Renan DelValle <rd...@binghamton.edu> wrote:
> 
> Hi David,
> 
> On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
> wrote:
> 
>> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
>> meghdoot_b@yahoo.com.invalid> wrote:
>> 
>>> Comments inline
>>> 
>>>      From: David McLaughlin <dm...@apache.org>
>>> To: dev@aurora.apache.org
>>> Sent: Thursday, June 9, 2016 5:13 PM
>>> Subject: Re: Golang Aurora lib, multiple executor support, integrate
>>> mesos task related fields
>>> 
>>> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
>>> wrote:
>>> 
>>>> Hello all,
>>>> 
>>>> I'd like to (re-)introduce myself. My name's Renan DelValle and I've
>> had
>>>> the pleasure of being part of the Aurora community for the last year or
>>> so.
>>>> 
>>>> Last year I worked to allow Aurora to utilize custom executors. With
>> the
>>>> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
>>> became
>>>> a reality and made into Aurora late last year. (Also got to show an
>> early
>>>> beta at MesosCon!)
>>>> 
>>>> For this year's summer, I have some new goals which I invite anyone to
>>>> provide input on.
>>>> 
>>>> 1. Create a Golang library that works as an alternative to Aurora's
>>> Python
>>>> client and Pystachio. The initial goal of this library is to support
>> the
>>>> most critical job related Thrift API's. (Admin operations can continue
>> to
>>>> be done from the Aurora CLI). Since we support custom executors, we
>> need
>>> a
>>>> way that's not so tied to thermos (as Pystachio is) to send jobs
>>>> configurations to Aurora (and by extension the custom executor).
>>> 
>>> 
>>> You can easily add support for a different configuration format without
>>> having to rewrite the CLI in Go. You'd do that by adding your own custom
>>> noun/verbs to the client or replacing existing commands. For example, at
>>> Twitter we have our own version of 'aurora update start' that plugs into
>> an
>>> internal Deploy Orchestration Service and we have a whole other command
>> for
>>> federated deploys. I can show you how we do that.
>>> 
>>> The first thing I thought of though was - this seems like a perfect
>>> starting point to get serious about a HTTP+JSON API for Aurora. Having
>> that
>>> would make it trivial to do a CLI in any language you want, and the
>> Python
>>> CLI would really only be there to do Pystachio evaluation.
>>>>> CLI integrations is not useful for a lot of different reasons. We need
>>> API's from other orchestration points from different components and hence
>>> we would package it in a go library. Uber, I believe has taken the same
>>> route (info from this mesoscon)
>>> We are not writing a replacement cli tool. As noted admin operations
>> would
>>> heavily leverage current aurora cli.
>>> I think REST work has been postponed and attempted few times in past. We
>>> cannot wait for that.
>> 
>> 
>> This is the dev list for contributions to the project, so I assumed we were
>> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
>> has Go bindings so you're all set for whatever custom tooling you're
>> building. Please use the users list for feedback on that.
> 
> 
> While I'm very thankful for your input on the matter, my original text very
> clearly said:
> "1. Create a Golang library that works as an *alternative* to
> Aurora's Python client and Pystachio."
> 
> I believe every single item I have listed has the potential to become a
> useful contribution to this project.
> Therefore, I feel that it is prudent continue the discussion here (unless
> others feel similarly).
> 
> -Renan
> 
>> 
>>> 
>>> 
>>>> 
>>>> 2. Create support for using multiple executors within a single Aurora
>>>> scheduler instance. As of right now, only a single executor can exist
>> for
>>>> each Aurora scheduler instance. The idea is to allow the Aurora
>> scheduler
>>>> to support multiple executors such that they can be used alongside one
>>>> another, providing flexibility for running jobs.
>>> 
>>> Just for my own understanding - what problems are you solving with your
>>> custom executor? Sorry if you've explained this to the lists before.
>>> 
>>> I ask because I'd really like to see Aurora stop treating the executor
>>> config as an opaque blob and being more opinionated. The
>> Thermos/Scheduler
>>> abstraction really hinders what type of user experience (UI) we can
>> build,
>>> and other frameworks do a much better job by being more opinionated and
>>> pulling executor data into the scheduler UI.
>>>>> We probably don't need to debate on this point. Executor is a first
>>> class mesos citizen and time is right for aurora to have good support for
>>> it.In our case, we have kubernetes like pods modeled through
>> docker-compose
>>> and the executor manages that. Scheduler UI main features should not get
>>> bogged down or be held back for a particular executor. That feels
>>> incredibly restrictive. If a special UI mode for thermos executor is
>>> created, that should be fine. We do have to differentiate the scheduler
>> and
>>> thermos and aurora team has done a great job of not hard coupling the
>> two.
>> 
>> 
>>>> 
>>>> 3. I'd like to add support for some first class Mesos task related
>>> fields.
>>>> These would be optional and/or have defaults so that the regular Aurora
>>> CLI
>>>> does not break. One of the examples I'm interested in integrating is
>> the
>>>> Mesos Fetcher so that resources can be downloaded into the sandbox that
>>> the
>>>> custom executor may need. (The executor path will never be exposed as
>>> this
>>>> will be defined serverside and be static).
>>> 
>>> 
>>> Wouldn't the mesos-fetcher call just be another process to be passed to
>>> your executor? I have to admit I don't know enough about how the Mesos
>>> fetcher works.
>>>>> Apache Mesos - Fetcher
>> 
>> 
>> 
>>> We may add other first class fields.
>>> 
>>> 
>>>> 
>>>> If anyone has any feedback on these, I'd be very glad to hear it.
>>>> 
>>>> That's it for me for now, thanks!
>>>> 
>>>> -Renan
>>> 
>>> 
>>> 
>>> 
>>> |
>>> |
>>> |
>>> |   |    |
>>> 
>>>   |
>>> 
>>>  |
>>> |
>>> |   |
>>> Apache Mesos - Fetcher
>>>   |   |
>>> 
>>>  |
>>> 
>>>  |
>> 

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[Renan DelValle <rd...@binghamton.edu>]

Hi David,

On Fri, Jun 10, 2016 at 12:33 AM, David McLaughlin <da...@dmclaughlin.com>
wrote:

> On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
> meghdoot_b@yahoo.com.invalid> wrote:
>
> > Comments inline
> >
> >       From: David McLaughlin <dm...@apache.org>
> >  To: dev@aurora.apache.org
> >  Sent: Thursday, June 9, 2016 5:13 PM
> >  Subject: Re: Golang Aurora lib, multiple executor support, integrate
> > mesos task related fields
> >
> > On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
> > wrote:
> >
> > > Hello all,
> > >
> > > I'd like to (re-)introduce myself. My name's Renan DelValle and I've
> had
> > > the pleasure of being part of the Aurora community for the last year or
> > so.
> > >
> > > Last year I worked to allow Aurora to utilize custom executors. With
> the
> > > help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
> > became
> > > a reality and made into Aurora late last year. (Also got to show an
> early
> > > beta at MesosCon!)
> > >
> > > For this year's summer, I have some new goals which I invite anyone to
> > > provide input on.
> > >
> > > 1. Create a Golang library that works as an alternative to Aurora's
> > Python
> > > client and Pystachio. The initial goal of this library is to support
> the
> > > most critical job related Thrift API's. (Admin operations can continue
> to
> > > be done from the Aurora CLI). Since we support custom executors, we
> need
> > a
> > > way that's not so tied to thermos (as Pystachio is) to send jobs
> > > configurations to Aurora (and by extension the custom executor).
> > >
> >
> >
> > You can easily add support for a different configuration format without
> > having to rewrite the CLI in Go. You'd do that by adding your own custom
> > noun/verbs to the client or replacing existing commands. For example, at
> > Twitter we have our own version of 'aurora update start' that plugs into
> an
> > internal Deploy Orchestration Service and we have a whole other command
> for
> > federated deploys. I can show you how we do that.
> >
> > The first thing I thought of though was - this seems like a perfect
> > starting point to get serious about a HTTP+JSON API for Aurora. Having
> that
> > would make it trivial to do a CLI in any language you want, and the
> Python
> > CLI would really only be there to do Pystachio evaluation.
> > >> CLI integrations is not useful for a lot of different reasons. We need
> > API's from other orchestration points from different components and hence
> > we would package it in a go library. Uber, I believe has taken the same
> > route (info from this mesoscon)
> > We are not writing a replacement cli tool. As noted admin operations
> would
> > heavily leverage current aurora cli.
> > I think REST work has been postponed and attempted few times in past. We
> > cannot wait for that.
> >
>
>
> This is the dev list for contributions to the project, so I assumed we were
> discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
> has Go bindings so you're all set for whatever custom tooling you're
> building. Please use the users list for feedback on that.


While I'm very thankful for your input on the matter, my original text very
clearly said:
"1. Create a Golang library that works as an *alternative* to
Aurora's Python client and Pystachio."

I believe every single item I have listed has the potential to become a
useful contribution to this project.
Therefore, I feel that it is prudent continue the discussion here (unless
others feel similarly).

-Renan

>
> >
> >
> > >
> > > 2. Create support for using multiple executors within a single Aurora
> > > scheduler instance. As of right now, only a single executor can exist
> for
> > > each Aurora scheduler instance. The idea is to allow the Aurora
> scheduler
> > > to support multiple executors such that they can be used alongside one
> > > another, providing flexibility for running jobs.
> > >
> >
> > Just for my own understanding - what problems are you solving with your
> > custom executor? Sorry if you've explained this to the lists before.
> >
> > I ask because I'd really like to see Aurora stop treating the executor
> > config as an opaque blob and being more opinionated. The
> Thermos/Scheduler
> > abstraction really hinders what type of user experience (UI) we can
> build,
> > and other frameworks do a much better job by being more opinionated and
> > pulling executor data into the scheduler UI.
> > >> We probably don't need to debate on this point. Executor is a first
> > class mesos citizen and time is right for aurora to have good support for
> > it.In our case, we have kubernetes like pods modeled through
> docker-compose
> > and the executor manages that. Scheduler UI main features should not get
> > bogged down or be held back for a particular executor. That feels
> > incredibly restrictive. If a special UI mode for thermos executor is
> > created, that should be fine. We do have to differentiate the scheduler
> and
> > thermos and aurora team has done a great job of not hard coupling the
> two.
>
>
> > >
> > > 3. I'd like to add support for some first class Mesos task related
> > fields.
> > > These would be optional and/or have defaults so that the regular Aurora
> > CLI
> > > does not break. One of the examples I'm interested in integrating is
> the
> > > Mesos Fetcher so that resources can be downloaded into the sandbox that
> > the
> > > custom executor may need. (The executor path will never be exposed as
> > this
> > > will be defined serverside and be static).
> >
> >
> > Wouldn't the mesos-fetcher call just be another process to be passed to
> > your executor? I have to admit I don't know enough about how the Mesos
> > fetcher works.
> > >> Apache Mesos - Fetcher
>
>
>
> > We may add other first class fields.
> >
> >
> > >
> > > If anyone has any feedback on these, I'd be very glad to hear it.
> > >
> > > That's it for me for now, thanks!
> > >
> > > -Renan
> > >
> >
> >
> >
> >
> > |
> > |
> > |
> > |   |    |
> >
> >    |
> >
> >   |
> > |
> > |   |
> > Apache Mesos - Fetcher
> >    |   |
> >
> >   |
> >
> >   |
> >
> >
> >
>

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[David McLaughlin <da...@dmclaughlin.com>]

On Thu, Jun 9, 2016 at 5:44 PM, meghdoot bhattacharya <
meghdoot_b@yahoo.com.invalid> wrote:

> Comments inline
>
>       From: David McLaughlin <dm...@apache.org>
>  To: dev@aurora.apache.org
>  Sent: Thursday, June 9, 2016 5:13 PM
>  Subject: Re: Golang Aurora lib, multiple executor support, integrate
> mesos task related fields
>
> On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
> wrote:
>
> > Hello all,
> >
> > I'd like to (re-)introduce myself. My name's Renan DelValle and I've had
> > the pleasure of being part of the Aurora community for the last year or
> so.
> >
> > Last year I worked to allow Aurora to utilize custom executors. With the
> > help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature
> became
> > a reality and made into Aurora late last year. (Also got to show an early
> > beta at MesosCon!)
> >
> > For this year's summer, I have some new goals which I invite anyone to
> > provide input on.
> >
> > 1. Create a Golang library that works as an alternative to Aurora's
> Python
> > client and Pystachio. The initial goal of this library is to support the
> > most critical job related Thrift API's. (Admin operations can continue to
> > be done from the Aurora CLI). Since we support custom executors, we need
> a
> > way that's not so tied to thermos (as Pystachio is) to send jobs
> > configurations to Aurora (and by extension the custom executor).
> >
>
>
> You can easily add support for a different configuration format without
> having to rewrite the CLI in Go. You'd do that by adding your own custom
> noun/verbs to the client or replacing existing commands. For example, at
> Twitter we have our own version of 'aurora update start' that plugs into an
> internal Deploy Orchestration Service and we have a whole other command for
> federated deploys. I can show you how we do that.
>
> The first thing I thought of though was - this seems like a perfect
> starting point to get serious about a HTTP+JSON API for Aurora. Having that
> would make it trivial to do a CLI in any language you want, and the Python
> CLI would really only be there to do Pystachio evaluation.
> >> CLI integrations is not useful for a lot of different reasons. We need
> API's from other orchestration points from different components and hence
> we would package it in a go library. Uber, I believe has taken the same
> route (info from this mesoscon)
> We are not writing a replacement cli tool. As noted admin operations would
> heavily leverage current aurora cli.
> I think REST work has been postponed and attempted few times in past. We
> cannot wait for that.
>


This is the dev list for contributions to the project, so I assumed we were
discussing adding a Go CLI to Apache Aurora. Our API is Thrift and Thrift
has Go bindings so you're all set for whatever custom tooling you're
building. Please use the users list for feedback on that.



>
>
> >
> > 2. Create support for using multiple executors within a single Aurora
> > scheduler instance. As of right now, only a single executor can exist for
> > each Aurora scheduler instance. The idea is to allow the Aurora scheduler
> > to support multiple executors such that they can be used alongside one
> > another, providing flexibility for running jobs.
> >
>
> Just for my own understanding - what problems are you solving with your
> custom executor? Sorry if you've explained this to the lists before.
>
> I ask because I'd really like to see Aurora stop treating the executor
> config as an opaque blob and being more opinionated. The Thermos/Scheduler
> abstraction really hinders what type of user experience (UI) we can build,
> and other frameworks do a much better job by being more opinionated and
> pulling executor data into the scheduler UI.
> >> We probably don't need to debate on this point. Executor is a first
> class mesos citizen and time is right for aurora to have good support for
> it.In our case, we have kubernetes like pods modeled through docker-compose
> and the executor manages that. Scheduler UI main features should not get
> bogged down or be held back for a particular executor. That feels
> incredibly restrictive. If a special UI mode for thermos executor is
> created, that should be fine. We do have to differentiate the scheduler and
> thermos and aurora team has done a great job of not hard coupling the two.


> >
> > 3. I'd like to add support for some first class Mesos task related
> fields.
> > These would be optional and/or have defaults so that the regular Aurora
> CLI
> > does not break. One of the examples I'm interested in integrating is the
> > Mesos Fetcher so that resources can be downloaded into the sandbox that
> the
> > custom executor may need. (The executor path will never be exposed as
> this
> > will be defined serverside and be static).
>
>
> Wouldn't the mesos-fetcher call just be another process to be passed to
> your executor? I have to admit I don't know enough about how the Mesos
> fetcher works.
> >> Apache Mesos - Fetcher



> We may add other first class fields.
>
>
> >
> > If anyone has any feedback on these, I'd be very glad to hear it.
> >
> > That's it for me for now, thanks!
> >
> > -Renan
> >
>
>
>
>
> |
> |
> |
> |   |    |
>
>    |
>
>   |
> |
> |   |
> Apache Mesos - Fetcher
>    |   |
>
>   |
>
>   |
>
>
>

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[meghdoot bhattacharya <me...@yahoo.com.INVALID>]

Comments inline

      From: David McLaughlin <dm...@apache.org>
 To: dev@aurora.apache.org 
 Sent: Thursday, June 9, 2016 5:13 PM
 Subject: Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields
   
On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
wrote:

> Hello all,
>
> I'd like to (re-)introduce myself. My name's Renan DelValle and I've had
> the pleasure of being part of the Aurora community for the last year or so.
>
> Last year I worked to allow Aurora to utilize custom executors. With the
> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature became
> a reality and made into Aurora late last year. (Also got to show an early
> beta at MesosCon!)
>
> For this year's summer, I have some new goals which I invite anyone to
> provide input on.
>
> 1. Create a Golang library that works as an alternative to Aurora's Python
> client and Pystachio. The initial goal of this library is to support the
> most critical job related Thrift API's. (Admin operations can continue to
> be done from the Aurora CLI). Since we support custom executors, we need a
> way that's not so tied to thermos (as Pystachio is) to send jobs
> configurations to Aurora (and by extension the custom executor).
>


You can easily add support for a different configuration format without
having to rewrite the CLI in Go. You'd do that by adding your own custom
noun/verbs to the client or replacing existing commands. For example, at
Twitter we have our own version of 'aurora update start' that plugs into an
internal Deploy Orchestration Service and we have a whole other command for
federated deploys. I can show you how we do that.

The first thing I thought of though was - this seems like a perfect
starting point to get serious about a HTTP+JSON API for Aurora. Having that
would make it trivial to do a CLI in any language you want, and the Python
CLI would really only be there to do Pystachio evaluation.
>> CLI integrations is not useful for a lot of different reasons. We need API's from other orchestration points from different components and hence we would package it in a go library. Uber, I believe has taken the same route (info from this mesoscon)
We are not writing a replacement cli tool. As noted admin operations would heavily leverage current aurora cli.
I think REST work has been postponed and attempted few times in past. We cannot wait for that.


>
> 2. Create support for using multiple executors within a single Aurora
> scheduler instance. As of right now, only a single executor can exist for
> each Aurora scheduler instance. The idea is to allow the Aurora scheduler
> to support multiple executors such that they can be used alongside one
> another, providing flexibility for running jobs.
>

Just for my own understanding - what problems are you solving with your
custom executor? Sorry if you've explained this to the lists before.

I ask because I'd really like to see Aurora stop treating the executor
config as an opaque blob and being more opinionated. The Thermos/Scheduler
abstraction really hinders what type of user experience (UI) we can build,
and other frameworks do a much better job by being more opinionated and
pulling executor data into the scheduler UI.
>> We probably don't need to debate on this point. Executor is a first class mesos citizen and time is right for aurora to have good support for it.In our case, we have kubernetes like pods modeled through docker-compose and the executor manages that. Scheduler UI main features should not get bogged down or be held back for a particular executor. That feels incredibly restrictive. If a special UI mode for thermos executor is created, that should be fine. We do have to differentiate the scheduler and thermos and aurora team has done a great job of not hard coupling the two.


>
> 3. I'd like to add support for some first class Mesos task related fields.
> These would be optional and/or have defaults so that the regular Aurora CLI
> does not break. One of the examples I'm interested in integrating is the
> Mesos Fetcher so that resources can be downloaded into the sandbox that the
> custom executor may need. (The executor path will never be exposed as this
> will be defined serverside and be static).


Wouldn't the mesos-fetcher call just be another process to be passed to
your executor? I have to admit I don't know enough about how the Mesos
fetcher works.
>> Apache Mesos - Fetcher

We may add other first class fields.


>
> If anyone has any feedback on these, I'd be very glad to hear it.
>
> That's it for me for now, thanks!
>
> -Renan
>



  
|  
|   
|   
|   |    |

   |

  |
|  
|   |  
Apache Mesos - Fetcher
   |   |

  |

  |

 
  

Re: Golang Aurora lib, multiple executor support, integrate mesos task related fields

[David McLaughlin <dm...@apache.org>]

On Thu, Jun 9, 2016 at 2:21 PM, Renan DelValle <rd...@binghamton.edu>
wrote:

> Hello all,
>
> I'd like to (re-)introduce myself. My name's Renan DelValle and I've had
> the pleasure of being part of the Aurora community for the last year or so.
>
> Last year I worked to allow Aurora to utilize custom executors. With the
> help from Bill Farner, Kevin Sweeney, and Joshua Cohen, that feature became
> a reality and made into Aurora late last year. (Also got to show an early
> beta at MesosCon!)
>
> For this year's summer, I have some new goals which I invite anyone to
> provide input on.
>
> 1. Create a Golang library that works as an alternative to Aurora's Python
> client and Pystachio. The initial goal of this library is to support the
> most critical job related Thrift API's. (Admin operations can continue to
> be done from the Aurora CLI). Since we support custom executors, we need a
> way that's not so tied to thermos (as Pystachio is) to send jobs
> configurations to Aurora (and by extension the custom executor).
>


You can easily add support for a different configuration format without
having to rewrite the CLI in Go. You'd do that by adding your own custom
noun/verbs to the client or replacing existing commands. For example, at
Twitter we have our own version of 'aurora update start' that plugs into an
internal Deploy Orchestration Service and we have a whole other command for
federated deploys. I can show you how we do that.

The first thing I thought of though was - this seems like a perfect
starting point to get serious about a HTTP+JSON API for Aurora. Having that
would make it trivial to do a CLI in any language you want, and the Python
CLI would really only be there to do Pystachio evaluation.


>
> 2. Create support for using multiple executors within a single Aurora
> scheduler instance. As of right now, only a single executor can exist for
> each Aurora scheduler instance. The idea is to allow the Aurora scheduler
> to support multiple executors such that they can be used alongside one
> another, providing flexibility for running jobs.
>

Just for my own understanding - what problems are you solving with your
custom executor? Sorry if you've explained this to the lists before.

I ask because I'd really like to see Aurora stop treating the executor
config as an opaque blob and being more opinionated. The Thermos/Scheduler
abstraction really hinders what type of user experience (UI) we can build,
and other frameworks do a much better job by being more opinionated and
pulling executor data into the scheduler UI.


>
> 3. I'd like to add support for some first class Mesos task related fields.
> These would be optional and/or have defaults so that the regular Aurora CLI
> does not break. One of the examples I'm interested in integrating is the
> Mesos Fetcher so that resources can be downloaded into the sandbox that the
> custom executor may need. (The executor path will never be exposed as this
> will be defined serverside and be static).


Wouldn't the mesos-fetcher call just be another process to be passed to
your executor? I have to admit I don't know enough about how the Mesos
fetcher works.


>
> If anyone has any feedback on these, I'd be very glad to hear it.
>
> That's it for me for now, thanks!
>
> -Renan
>