You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by Santosh Marella <sm...@maprtech.com> on 2015/08/27 23:24:50 UTC

Re: Flex API

(Reviving this thread)

We've discussed several great points in the thread (PUT vs POST, need for
GET, JSON payload vs parameters in URL, declarative interface etc).
Just to get us going, I think we should focus on a couple of things that
will be useful for Myriad users, while leaving them flexible enough to be
evolved in the future.

What I heard from several folks (some of it brought up again at MesosCon)
about the flex up/down APIs is this:
- flexup doesn't support launching NMs on specific set of hosts. This is
especially needed to launch NMs on same set of nodes that have HDFS
DataNode running.
- flexdown lacks an option to shut down NMs with a specific profile. Today,
we bring down ANY arbitrary NM.
- flexdown lacks an option to shutdown NMs running on specific hosts.

I captured my thoughts in a document here:
https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t

@Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
finalize the APIs. Just let me know.

Thanks,
Santosh

On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:

> Seems like POST is a winner with people.
>
> Another thing to consider is how we want the REST interface to be vs what
> we want the UI to do. The UI could support flexup/flexdown like it is while
> the REST interface is just a declarative state like Adam suggested. The UI
> would just be responsible for translating the request into the new state.
>
> Tomorrow I'll try to put together another swagger doc with some of the
> suggested options.
>
>
> On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> <yufeldman@yahoo.com.invalid
> > wrote:
>
> > I think we are at the point to list all the options we want "flex" API to
> > support.
> > 1. Do we continue supporting flexup/down or just "flex" with additional
> > "preposition" like up/down:https://hostname:port/flex/up(down)
> > 2. I think we should switch to POST and may be maintain PUT for legacy
> (if
> > even needed to keep it). We are not DB after all and not storing any
> > retrievable info here :)
> > 3. We need to add status (GET) to see the status - though I think we have
> > one
> > 4. Define JSON payload to support different cases    a. providing
> > different profiles together: [{profile:"big",
> > instances:2},{profile:"medium",instances:6}]    b. provide what state we
> > want Myriad to be in: "I want 10 medium instances" and then Myriad will
> do
> > whatever isnecessary to transition to that state,
> adding/removing/resizing
> > NMs"    c. flex/down particular instance IDs    d. flex up/down preferred
> > hosts, delays, others
> > 5. How all this fits into FineGrain Scaling? With it we would do
> automatic
> > flex up/down. And the less knobs admin will have to turn the easier it is
> > for admin and the end users.
> >
> >       From: Adam Bordelon <ad...@mesosphere.io>
> >  To: dev@myriad.incubator.apache.org
> >  Sent: Sunday, June 14, 2015 2:54 PM
> >  Subject: Re: Flex API
> >
> > (In addition,) I'd also like to see a more declarative interface. Instead
> > of "add two more instances", the user(s) could just specify the desired
> > state of "I want 10 medium instances" and then Myriad will do whatever is
> > necessary to transition to that state, adding/removing/resizing NMs as
> > necessary.
> >
> >
> >
> > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> > wochandarena@maprtech.com
> > > wrote:
> >
> > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com> wrote:
> > >
> > > > What verb to use when outside of database land can be argued. I would
> > > vote
> > > > for POST over PUT just because I tend to default to POST. PUT was
> there
> > > > when I showed up, so I left it.
> > >
> > >
> > > Last time I agonized about PUT vs POST the most logical distinction I
> > found
> > > was that PUT should be used for idempotent operations, while POST for
> > > non-idempotent (like we have here with flex-up, since instance-ids are
> > > generated).
> > >
> > > Since the api doesn't wait until the
> > > > instances are created to return, we can't really return the instance
> > IDs
> > > we
> > > > created.
> > > >
> > >
> > > That seems OK to me.
> > >
> > >
> > > > The GET would just return some status?
> > > >
> > >
> > > Yeah, I was thinking that this would be needed for a future GUI where
> we
> > > list all instances with parameters and status for each (profile,
> current
> > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to each so
> > > users can multi-select and hit 'delete' to wipe them away (like
> flex-down
> > > does now).
> > >
> > > The PATCH is interesting
> > > >
> > >
> > > Yeah, I started to write PUT but to REST geeks PUT implies you always
> > have
> > > to rewrite the complete object when making changes.  PATCH allows more
> > > flexible modifications.
> > >
> > > The DELETE makes sense to me.
> > > >
> > > > Your use of instances vs instance is interesting. Perhaps we want to
> > > > support POST [{profile:"big", instances:2},{profile:"medium",
> > > instances:6}]
> > > >
> > >
> > > Yeah, that'd be cool!
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> > > > wochandarena@maprtech.com
> > > > > wrote:
> > > >
> > > > > Any reason we need to make it action-based (flex-up/flex-down)
> rather
> > > > than
> > > > > instance outcome-based?  The way i've seen similar things modeled
> in
> > > > other
> > > > > REST APIs is similar to below.  This may also fit better into REST
> > > client
> > > > > frameworks.
> > > > >
> > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2,
> > > "delay" :
> > > > > 2, "preferredHosts" : ["host1", "host2"]}
> > > > > Read - GET /instances/ or GET /instances/<instanceid>
> > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" :
> "big"}
> > > > > Delete - DELETE /instances/<instanceid>
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <kl...@gmail.com>
> > wrote:
> > > > >
> > > > > > That is always an issue with parameters in the URL, and there's
> > > nothing
> > > > > > worse than parameters in the URL and having to put JSON in the
> body
> > > > > because
> > > > > > we want to extend it later. Currently flexup takes a JSON blob in
> > the
> > > > > body
> > > > > > which we could add more to like you showed. I guess the question
> is
> > > if
> > > > we
> > > > > > want to support a quick and dirty flexup api via URL parameters.
> I
> > > > > wouldn't
> > > > > > go any further than these parameters, but all this would do at
> this
> > > > point
> > > > > > is make testing with a curl command easier. Getting rid of it for
> > > > flexup
> > > > > is
> > > > > > fine with me.
> > > > > >
> > > > > > I do like the flexdown/instance/{instance-id} though. We should
> > > > probably
> > > > > do
> > > > > > one with flexdown/instance with a json array of instance-ids too
> > > > though.
> > > > > >
> > > > > > Also with the parameter verification patch I just submitted, if
> you
> > > ask
> > > > > to
> > > > > > flexdown 10 instances and you only have 5 running it will just
> log
> > a
> > > > > > warning and kill all 5. Not sure if that is the desired behavior.
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> > > > smarella@maprtech.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think rather than having an API defined as
> > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to define
> > it
> > > > just
> > > > > > as
> > > > > > > "/cluster/flexup" and have parameters such as "profile:
> > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason is,
> > if
> > > > the
> > > > > > > params are added into the API endpoint, it becomes less
> flexible
> > to
> > > > > > evolve.
> > > > > > > For e.g. /cluster/flexup currently can take just "profile" and
> > > > > > "instances"
> > > > > > > in it's payload, but in the future can optionally take
> parameters
> > > > such
> > > > > as
> > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Santosh
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <kl...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I thought that might happen. I created a gist here:
> > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> > > > > > > >
> > > > > > > > And just in case....
> > > > > > > >
> > > > > > > > swagger: '2.0'
> > > > > > > > info:
> > > > > > > >  version: 0.0.1
> > > > > > > >  title: Myriad API
> > > > > > > >  description: |
> > > > > > > >    Myriad API description
> > > > > > > > basePath: /api
> > > > > > > > schemes:
> > > > > > > >  - http
> > > > > > > > consumes:
> > > > > > > >  - application/json
> > > > > > > >  - text/plain
> > > > > > > > produces:
> > > > > > > >  - application/json
> > > > > > > >  - text/plain
> > > > > > > > paths:
> > > > > > > >  /cluster/flexup:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance
> > > > > > > >          in: body
> > > > > > > >          description: Instance profile and quantity to flex
> up
> > > > > > > >          schema:
> > > > > > > >            $ref: '#/definitions/FlexUp'
> > > > > > > >          required: true
> > > > > > > >      description: Original Flexup API
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexup/profile/{profile-name}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >        - name: instances
> > > > > > > >          in: path
> > > > > > > >          type: integer
> > > > > > > >          description: Number of instances
> > > > > > > >      description: Flexup a multiple instances of a specified
> > > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance
> > > > > > > >          in: body
> > > > > > > >          description: Number of instances to flex down
> > > > > > > >          schema:
> > > > > > > >            $ref: '#/definitions/FlexDown'
> > > > > > > >          required: true
> > > > > > > >      description: Original Flexdown API
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >        - name: instances
> > > > > > > >          in: path
> > > > > > > >          type: integer
> > > > > > > >          description: Number of instances
> > > > > > > >      description: Flexdown a multiple instances of a
> specified
> > > > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance-id
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > > definitions:
> > > > > > > >  FlexUp:
> > > > > > > >    properties:
> > > > > > > >      profile:
> > > > > > > >        type: string
> > > > > > > >      instances:
> > > > > > > >        type: integer
> > > > > > > >        format: int32
> > > > > > > >  FlexDown:
> > > > > > > >    properties:
> > > > > > > >      instances:
> > > > > > > >        type: integer
> > > > > > > >        format: int32
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> > > > > > smarella@maprtech.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jim,
> > > > > > > > >
> > > > > > > > >  Did you attach a file? I think the apache mailing list
> > > > swallowed
> > > > > > the
> > > > > > > > > attachment. Can you please send us the document contents in
> > > plain
> > > > > > text?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Santosh
> > > > > > > > >
> > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> > klucar@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This is in regards to
> > > > https://github.com/mesos/myriad/issues/89
> > > > > > > > > >
> > > > > > > > > > I created a swagger document (attached) of the current
> flex
> > > > APIs
> > > > > > and
> > > > > > > > what
> > > > > > > > > > I propose for the new flex apis. If people like swagger,
> I
> > > can
> > > > > > > document
> > > > > > > > > the
> > > > > > > > > > rest of the API, including the correct response codes.
> > Either
> > > > > way I
> > > > > > > > think
> > > > > > > > > > we should discuss what the API should be going forward.
> > > > > > > > > >
> > > > > > > > > > Paste the file contents into http://editor.swagger.io/
> and
> > > > > you'll
> > > > > > > get
> > > > > > > > > > nice HTML to browse.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>

Re: Flex API

Posted by Santosh Marella <sm...@maprtech.com>.
Thanks Jim. A separate PR would be better. Let me know if you have any
questions on the API. The API doc on PR 129 is a good starting point.


Santosh

On Thu, Sep 24, 2015 at 10:42 AM, Jim Klucar <kl...@gmail.com> wrote:

> Santosh,
>
> I can do the UI work. Should I make it a separate PR or want me to push you
> a patch on the side?
>
> On Wed, Sep 23, 2015 at 6:58 PM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
> > Addressed review comments from Yuliya and updated the PR:
> > https://github.com/mesos/myriad/pull/129
> >
> > @Jim - Your UI expertise is much sought to support "constraints"
> parameter
> > in both flexup/down and "profile" in flexdown.
> > The API doc is updated as part of the PR:
> >
> >
> https://github.com/mesos/myriad/pull/129/files#diff-96cc378dcdef180641176501f42183c7
> >
> > Thanks,
> > Santosh
> >
> > On Mon, Sep 14, 2015 at 10:54 AM, Santosh Marella <smarella@maprtech.com
> >
> > wrote:
> >
> > > The proposal is to have "constraints" parameter supported for both
> > > flexup/flexdown.
> > > So, one could flexdown NMs on specific hosts, for e..g by providing a
> > > constraint
> > > around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE "
> > > specific.host.com"]]}.
> > >
> > > Santosh
> > >
> > > On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <ad...@mesosphere.io>
> > wrote:
> > >
> > >> +1, but EQUALS might be an easier starting point than LIKE. Your
> choice.
> > >> What about flexing down specific hosts? Is that out of scope for now?
> > >>
> > >> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <
> smarella@maprtech.com>
> > >> wrote:
> > >>
> > >> > Ping! Checking if there is anymore feedback on the APIs for
> > flexup/down.
> > >> >
> > >> > Here is a textual summary of what's being proposed:
> > >> > 1. Add a new “constraints” parameter in both flexup and flexdown
> APIs
> > >> (just
> > >> > like how Marathon does).
> > >> > Initially, start with support for LIKE operator in "constraints"
> that
> > >> > operates on “hostname” and “mesos slave attribute
> > >> > <http://mesos.apache.org/documentation/attributes-resources/>”
> fields
> > >> > using
> > >> > regex.
> > >> > 2. Add a new “profile” parameter in the flexdown API.
> > >> >
> > >> > #1 would give admins a better control over where the NMs should be
> > >> launched
> > >> > or which specific NMs should be shut down.
> > >> > #2 helps with shutting down multiple NMs of same "profile" in one
> > shot.
> > >> >
> > >> > If the above sounds reasonable to start with, I'd like to start
> > working
> > >> on
> > >> > the code changes.
> > >> >
> > >> > Thanks,
> > >> > Santosh
> > >> >
> > >> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <
> > smarella@maprtech.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > > One additional thing I'd like is the ability to flexup a nmnode
> on
> > >> all
> > >> > > > agents with a certain attribute as opposed to a fixed number.
> > >> > >
> > >> > > Certainly! Marathon places tasks on agents sharing a common
> > attribute.
> > >> > > Myriad could/should do the same for placing NMs. If each agent
> > running
> > >> > > HDFS data node shares an attribute such as "dfs", then NMs could
> > >> always
> > >> > > be launched on the "dfs" nodes. This improves data locality for
> YARN
> > >> jobs
> > >> > > dramatically.
> > >> > >
> > >> > > Santosh
> > >> > >
> > >> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <
> > >> dbjohnson1978@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> One additional thing I'd like is the ability to flexup a nmnode
> on
> > >> all
> > >> > >> agents with a certain attribute as opposed to a fixed number.
> > >> > >>
> > >> > >
> > >> > >
> > >> > >>
> > >> > >> Also, when I get back from vacation I plan on scoping mesos-1739
> > >> > (dynamic
> > >> > >> attributes) which would allow for tighter integration with the
> hdfs
> > >> > >> framework. An alternative would be to get hostnames from the
> > >> namenode,
> > >> > >> though not as seemless.
> > >> > >> (Reviving this thread)
> > >> > >>
> > >> > >> We've discussed several great points in the thread (PUT vs POST,
> > need
> > >> > for
> > >> > >> GET, JSON payload vs parameters in URL, declarative interface
> etc).
> > >> > >> Just to get us going, I think we should focus on a couple of
> things
> > >> that
> > >> > >> will be useful for Myriad users, while leaving them flexible
> enough
> > >> to
> > >> > be
> > >> > >> evolved in the future.
> > >> > >>
> > >> > >> What I heard from several folks (some of it brought up again at
> > >> > MesosCon)
> > >> > >> about the flex up/down APIs is this:
> > >> > >> - flexup doesn't support launching NMs on specific set of hosts.
> > >> This is
> > >> > >> especially needed to launch NMs on same set of nodes that have
> HDFS
> > >> > >> DataNode running.
> > >> > >> - flexdown lacks an option to shut down NMs with a specific
> > profile.
> > >> > >> Today,
> > >> > >> we bring down ANY arbitrary NM.
> > >> > >> - flexdown lacks an option to shutdown NMs running on specific
> > hosts.
> > >> > >>
> > >> > >> I captured my thoughts in a document here:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
> > >> > >>
> > >> > >> @Jim: Happy to collaborate at one single place (Swagger/Google
> Doc)
> > >> to
> > >> > >> finalize the APIs. Just let me know.
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Santosh
> > >> > >>
> > >> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com>
> > >> wrote:
> > >> > >>
> > >> > >> > Seems like POST is a winner with people.
> > >> > >> >
> > >> > >> > Another thing to consider is how we want the REST interface to
> be
> > >> vs
> > >> > >> what
> > >> > >> > we want the UI to do. The UI could support flexup/flexdown like
> > it
> > >> is
> > >> > >> while
> > >> > >> > the REST interface is just a declarative state like Adam
> > suggested.
> > >> > The
> > >> > >> UI
> > >> > >> > would just be responsible for translating the request into the
> > new
> > >> > >> state.
> > >> > >> >
> > >> > >> > Tomorrow I'll try to put together another swagger doc with some
> > of
> > >> the
> > >> > >> > suggested options.
> > >> > >> >
> > >> > >> >
> > >> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> > >> > >> > <yufeldman@yahoo.com.invalid
> > >> > >> > > wrote:
> > >> > >> >
> > >> > >> > > I think we are at the point to list all the options we want
> > >> "flex"
> > >> > API
> > >> > >> to
> > >> > >> > > support.
> > >> > >> > > 1. Do we continue supporting flexup/down or just "flex" with
> > >> > >> additional
> > >> > >> > > "preposition" like up/down:https://hostname:port
> /flex/up(down)
> > >> > >> > > 2. I think we should switch to POST and may be maintain PUT
> for
> > >> > legacy
> > >> > >> > (if
> > >> > >> > > even needed to keep it). We are not DB after all and not
> > storing
> > >> any
> > >> > >> > > retrievable info here :)
> > >> > >> > > 3. We need to add status (GET) to see the status - though I
> > >> think we
> > >> > >> have
> > >> > >> > > one
> > >> > >> > > 4. Define JSON payload to support different cases    a.
> > providing
> > >> > >> > > different profiles together: [{profile:"big",
> > >> > >> > > instances:2},{profile:"medium",instances:6}]    b. provide
> what
> > >> > state
> > >> > >> we
> > >> > >> > > want Myriad to be in: "I want 10 medium instances" and then
> > >> Myriad
> > >> > >> will
> > >> > >> > do
> > >> > >> > > whatever isnecessary to transition to that state,
> > >> > >> > adding/removing/resizing
> > >> > >> > > NMs"    c. flex/down particular instance IDs    d. flex
> up/down
> > >> > >> preferred
> > >> > >> > > hosts, delays, others
> > >> > >> > > 5. How all this fits into FineGrain Scaling? With it we would
> > do
> > >> > >> > automatic
> > >> > >> > > flex up/down. And the less knobs admin will have to turn the
> > >> easier
> > >> > it
> > >> > >> is
> > >> > >> > > for admin and the end users.
> > >> > >> > >
> > >> > >> > >       From: Adam Bordelon <ad...@mesosphere.io>
> > >> > >> > >  To: dev@myriad.incubator.apache.org
> > >> > >> > >  Sent: Sunday, June 14, 2015 2:54 PM
> > >> > >> > >  Subject: Re: Flex API
> > >> > >> > >
> > >> > >> > > (In addition,) I'd also like to see a more declarative
> > interface.
> > >> > >> Instead
> > >> > >> > > of "add two more instances", the user(s) could just specify
> the
> > >> > >> desired
> > >> > >> > > state of "I want 10 medium instances" and then Myriad will do
> > >> > whatever
> > >> > >> is
> > >> > >> > > necessary to transition to that state,
> adding/removing/resizing
> > >> NMs
> > >> > as
> > >> > >> > > necessary.
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> > >> > >> > > wochandarena@maprtech.com
> > >> > >> > > > wrote:
> > >> > >> > >
> > >> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <
> > klucar@gmail.com>
> > >> > >> wrote:
> > >> > >> > > >
> > >> > >> > > > > What verb to use when outside of database land can be
> > >> argued. I
> > >> > >> would
> > >> > >> > > > vote
> > >> > >> > > > > for POST over PUT just because I tend to default to POST.
> > PUT
> > >> > was
> > >> > >> > there
> > >> > >> > > > > when I showed up, so I left it.
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > Last time I agonized about PUT vs POST the most logical
> > >> > distinction
> > >> > >> I
> > >> > >> > > found
> > >> > >> > > > was that PUT should be used for idempotent operations,
> while
> > >> POST
> > >> > >> for
> > >> > >> > > > non-idempotent (like we have here with flex-up, since
> > >> instance-ids
> > >> > >> are
> > >> > >> > > > generated).
> > >> > >> > > >
> > >> > >> > > > Since the api doesn't wait until the
> > >> > >> > > > > instances are created to return, we can't really return
> the
> > >> > >> instance
> > >> > >> > > IDs
> > >> > >> > > > we
> > >> > >> > > > > created.
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > That seems OK to me.
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > > The GET would just return some status?
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > Yeah, I was thinking that this would be needed for a future
> > GUI
> > >> > >> where
> > >> > >> > we
> > >> > >> > > > list all instances with parameters and status for each
> > >> (profile,
> > >> > >> > current
> > >> > >> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next
> > to
> > >> > each
> > >> > >> so
> > >> > >> > > > users can multi-select and hit 'delete' to wipe them away
> > (like
> > >> > >> > flex-down
> > >> > >> > > > does now).
> > >> > >> > > >
> > >> > >> > > > The PATCH is interesting
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies
> > you
> > >> > >> always
> > >> > >> > > have
> > >> > >> > > > to rewrite the complete object when making changes.  PATCH
> > >> allows
> > >> > >> more
> > >> > >> > > > flexible modifications.
> > >> > >> > > >
> > >> > >> > > > The DELETE makes sense to me.
> > >> > >> > > > >
> > >> > >> > > > > Your use of instances vs instance is interesting. Perhaps
> > we
> > >> > want
> > >> > >> to
> > >> > >> > > > > support POST [{profile:"big",
> > instances:2},{profile:"medium",
> > >> > >> > > > instances:6}]
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > > > Yeah, that'd be cool!
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> > >> > >> > > > > wochandarena@maprtech.com
> > >> > >> > > > > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > > Any reason we need to make it action-based
> > >> (flex-up/flex-down)
> > >> > >> > rather
> > >> > >> > > > > than
> > >> > >> > > > > > instance outcome-based?  The way i've seen similar
> things
> > >> > >> modeled
> > >> > >> > in
> > >> > >> > > > > other
> > >> > >> > > > > > REST APIs is similar to below.  This may also fit
> better
> > >> into
> > >> > >> REST
> > >> > >> > > > client
> > >> > >> > > > > > frameworks.
> > >> > >> > > > > >
> > >> > >> > > > > > Create - POST /instances/ {"profile" : "zero",
> > "instances"
> > >> :
> > >> > 2,
> > >> > >> > > > "delay" :
> > >> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
> > >> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
> > >> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid>
> > >> {"profile" :
> > >> > >> > "big"}
> > >> > >> > > > > > Delete - DELETE /instances/<instanceid>
> > >> > >> > > > > >
> > >> > >> > > > > > Thoughts?
> > >> > >> > > > > >
> > >> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <
> > >> klucar@gmail.com
> > >> > >
> > >> > >> > > wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > That is always an issue with parameters in the URL,
> and
> > >> > >> there's
> > >> > >> > > > nothing
> > >> > >> > > > > > > worse than parameters in the URL and having to put
> JSON
> > >> in
> > >> > the
> > >> > >> > body
> > >> > >> > > > > > because
> > >> > >> > > > > > > we want to extend it later. Currently flexup takes a
> > JSON
> > >> > blob
> > >> > >> in
> > >> > >> > > the
> > >> > >> > > > > > body
> > >> > >> > > > > > > which we could add more to like you showed. I guess
> the
> > >> > >> question
> > >> > >> > is
> > >> > >> > > > if
> > >> > >> > > > > we
> > >> > >> > > > > > > want to support a quick and dirty flexup api via URL
> > >> > >> parameters.
> > >> > >> > I
> > >> > >> > > > > > wouldn't
> > >> > >> > > > > > > go any further than these parameters, but all this
> > would
> > >> do
> > >> > at
> > >> > >> > this
> > >> > >> > > > > point
> > >> > >> > > > > > > is make testing with a curl command easier. Getting
> rid
> > >> of
> > >> > it
> > >> > >> for
> > >> > >> > > > > flexup
> > >> > >> > > > > > is
> > >> > >> > > > > > > fine with me.
> > >> > >> > > > > > >
> > >> > >> > > > > > > I do like the flexdown/instance/{instance-id} though.
> > We
> > >> > >> should
> > >> > >> > > > > probably
> > >> > >> > > > > > do
> > >> > >> > > > > > > one with flexdown/instance with a json array of
> > >> instance-ids
> > >> > >> too
> > >> > >> > > > > though.
> > >> > >> > > > > > >
> > >> > >> > > > > > > Also with the parameter verification patch I just
> > >> submitted,
> > >> > >> if
> > >> > >> > you
> > >> > >> > > > ask
> > >> > >> > > > > > to
> > >> > >> > > > > > > flexdown 10 instances and you only have 5 running it
> > will
> > >> > just
> > >> > >> > log
> > >> > >> > > a
> > >> > >> > > > > > > warning and kill all 5. Not sure if that is the
> desired
> > >> > >> behavior.
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> > >> > >> > > > > smarella@maprtech.com>
> > >> > >> > > > > > > wrote:
> > >> > >> > > > > > >
> > >> > >> > > > > > > > I think rather than having an API defined as
> > >> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's
> better
> > >> to
> > >> > >> define
> > >> > >> > > it
> > >> > >> > > > > just
> > >> > >> > > > > > > as
> > >> > >> > > > > > > > "/cluster/flexup" and have parameters such as
> > "profile:
> > >> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload.
> The
> > >> > reason
> > >> > >> is,
> > >> > >> > > if
> > >> > >> > > > > the
> > >> > >> > > > > > > > params are added into the API endpoint, it becomes
> > less
> > >> > >> > flexible
> > >> > >> > > to
> > >> > >> > > > > > > evolve.
> > >> > >> > > > > > > > For e.g. /cluster/flexup currently can take just
> > >> "profile"
> > >> > >> and
> > >> > >> > > > > > > "instances"
> > >> > >> > > > > > > > in it's payload, but in the future can optionally
> > take
> > >> > >> > parameters
> > >> > >> > > > > such
> > >> > >> > > > > > as
> > >> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> > >> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > Thoughts?
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > Santosh
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
> > >> > >> klucar@gmail.com>
> > >> > >> > > > > wrote:
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > > I thought that might happen. I created a gist
> here:
> > >> > >> > > > > > > > >
> > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > > > And just in case....
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > > > swagger: '2.0'
> > >> > >> > > > > > > > > info:
> > >> > >> > > > > > > > >  version: 0.0.1
> > >> > >> > > > > > > > >  title: Myriad API
> > >> > >> > > > > > > > >  description: |
> > >> > >> > > > > > > > >    Myriad API description
> > >> > >> > > > > > > > > basePath: /api
> > >> > >> > > > > > > > > schemes:
> > >> > >> > > > > > > > >  - http
> > >> > >> > > > > > > > > consumes:
> > >> > >> > > > > > > > >  - application/json
> > >> > >> > > > > > > > >  - text/plain
> > >> > >> > > > > > > > > produces:
> > >> > >> > > > > > > > >  - application/json
> > >> > >> > > > > > > > >  - text/plain
> > >> > >> > > > > > > > > paths:
> > >> > >> > > > > > > > >  /cluster/flexup:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: instance
> > >> > >> > > > > > > > >          in: body
> > >> > >> > > > > > > > >          description: Instance profile and
> quantity
> > >> to
> > >> > >> flex
> > >> > >> > up
> > >> > >> > > > > > > > >          schema:
> > >> > >> > > > > > > > >            $ref: '#/definitions/FlexUp'
> > >> > >> > > > > > > > >          required: true
> > >> > >> > > > > > > > >      description: Original Flexup API
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: profile-name
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: string
> > >> > >> > > > > > > > >          description: Instance profile name
> > >> > >> > > > > > > > >      description: Flexup a single instance of a
> > >> > specified
> > >> > >> > > > profile
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >
> > /cluster/flexup/profile/{profile-name}/{instances}:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: profile-name
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: string
> > >> > >> > > > > > > > >          description: Instance profile name
> > >> > >> > > > > > > > >        - name: instances
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: integer
> > >> > >> > > > > > > > >          description: Number of instances
> > >> > >> > > > > > > > >      description: Flexup a multiple instances of
> a
> > >> > >> specified
> > >> > >> > > > > profile
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >  /cluster/flexdown:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: instance
> > >> > >> > > > > > > > >          in: body
> > >> > >> > > > > > > > >          description: Number of instances to flex
> > >> down
> > >> > >> > > > > > > > >          schema:
> > >> > >> > > > > > > > >            $ref: '#/definitions/FlexDown'
> > >> > >> > > > > > > > >          required: true
> > >> > >> > > > > > > > >      description: Original Flexdown API
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: profile-name
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: string
> > >> > >> > > > > > > > >          description: Instance profile name
> > >> > >> > > > > > > > >      description: Flexup a single instance of a
> > >> > specified
> > >> > >> > > > profile
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >
> > >> /cluster/flexdown/profile/{profile-name}/{instances}:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: profile-name
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: string
> > >> > >> > > > > > > > >          description: Instance profile name
> > >> > >> > > > > > > > >        - name: instances
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: integer
> > >> > >> > > > > > > > >          description: Number of instances
> > >> > >> > > > > > > > >      description: Flexdown a multiple instances
> of
> > a
> > >> > >> > specified
> > >> > >> > > > > > profile
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> > >> > >> > > > > > > > >    put:
> > >> > >> > > > > > > > >      parameters:
> > >> > >> > > > > > > > >        - name: instance-id
> > >> > >> > > > > > > > >          in: path
> > >> > >> > > > > > > > >          type: string
> > >> > >> > > > > > > > >          description: Instance profile name
> > >> > >> > > > > > > > >      description: Flexup a single instance of a
> > >> > specified
> > >> > >> > > > profile
> > >> > >> > > > > > > > >      responses:
> > >> > >> > > > > > > > >        200:
> > >> > >> > > > > > > > >          description: OK
> > >> > >> > > > > > > > > definitions:
> > >> > >> > > > > > > > >  FlexUp:
> > >> > >> > > > > > > > >    properties:
> > >> > >> > > > > > > > >      profile:
> > >> > >> > > > > > > > >        type: string
> > >> > >> > > > > > > > >      instances:
> > >> > >> > > > > > > > >        type: integer
> > >> > >> > > > > > > > >        format: int32
> > >> > >> > > > > > > > >  FlexDown:
> > >> > >> > > > > > > > >    properties:
> > >> > >> > > > > > > > >      instances:
> > >> > >> > > > > > > > >        type: integer
> > >> > >> > > > > > > > >        format: int32
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella
> <
> > >> > >> > > > > > > smarella@maprtech.com>
> > >> > >> > > > > > > > > wrote:
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > > > > Hi Jim,
> > >> > >> > > > > > > > > >
> > >> > >> > > > > > > > > >  Did you attach a file? I think the apache
> > mailing
> > >> > list
> > >> > >> > > > > swallowed
> > >> > >> > > > > > > the
> > >> > >> > > > > > > > > > attachment. Can you please send us the document
> > >> > contents
> > >> > >> in
> > >> > >> > > > plain
> > >> > >> > > > > > > text?
> > >> > >> > > > > > > > > >
> > >> > >> > > > > > > > > > Thanks,
> > >> > >> > > > > > > > > > Santosh
> > >> > >> > > > > > > > > >
> > >> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> > >> > >> > > klucar@gmail.com>
> > >> > >> > > > > > > wrote:
> > >> > >> > > > > > > > > >
> > >> > >> > > > > > > > > > > This is in regards to
> > >> > >> > > > > https://github.com/mesos/myriad/issues/89
> > >> > >> > > > > > > > > > >
> > >> > >> > > > > > > > > > > I created a swagger document (attached) of
> the
> > >> > current
> > >> > >> > flex
> > >> > >> > > > > APIs
> > >> > >> > > > > > > and
> > >> > >> > > > > > > > > what
> > >> > >> > > > > > > > > > > I propose for the new flex apis. If people
> like
> > >> > >> swagger,
> > >> > >> > I
> > >> > >> > > > can
> > >> > >> > > > > > > > document
> > >> > >> > > > > > > > > > the
> > >> > >> > > > > > > > > > > rest of the API, including the correct
> response
> > >> > codes.
> > >> > >> > > Either
> > >> > >> > > > > > way I
> > >> > >> > > > > > > > > think
> > >> > >> > > > > > > > > > > we should discuss what the API should be
> going
> > >> > >> forward.
> > >> > >> > > > > > > > > > >
> > >> > >> > > > > > > > > > > Paste the file contents into
> > >> > >> http://editor.swagger.io/
> > >> > >> > and
> > >> > >> > > > > > you'll
> > >> > >> > > > > > > > get
> > >> > >> > > > > > > > > > > nice HTML to browse.
> > >> > >> > > > > > > > > > >
> > >> > >> > > > > > > > > > >
> > >> > >> > > > > > > > > > >
> > >> > >> > > > > > > > > >
> > >> > >> > > > > > > > >
> > >> > >> > > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Flex API

Posted by Jim Klucar <kl...@gmail.com>.
Santosh,

I can do the UI work. Should I make it a separate PR or want me to push you
a patch on the side?

On Wed, Sep 23, 2015 at 6:58 PM, Santosh Marella <sm...@maprtech.com>
wrote:

> Addressed review comments from Yuliya and updated the PR:
> https://github.com/mesos/myriad/pull/129
>
> @Jim - Your UI expertise is much sought to support "constraints" parameter
> in both flexup/down and "profile" in flexdown.
> The API doc is updated as part of the PR:
>
> https://github.com/mesos/myriad/pull/129/files#diff-96cc378dcdef180641176501f42183c7
>
> Thanks,
> Santosh
>
> On Mon, Sep 14, 2015 at 10:54 AM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
> > The proposal is to have "constraints" parameter supported for both
> > flexup/flexdown.
> > So, one could flexdown NMs on specific hosts, for e..g by providing a
> > constraint
> > around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE "
> > specific.host.com"]]}.
> >
> > Santosh
> >
> > On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <ad...@mesosphere.io>
> wrote:
> >
> >> +1, but EQUALS might be an easier starting point than LIKE. Your choice.
> >> What about flexing down specific hosts? Is that out of scope for now?
> >>
> >> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <sm...@maprtech.com>
> >> wrote:
> >>
> >> > Ping! Checking if there is anymore feedback on the APIs for
> flexup/down.
> >> >
> >> > Here is a textual summary of what's being proposed:
> >> > 1. Add a new “constraints” parameter in both flexup and flexdown APIs
> >> (just
> >> > like how Marathon does).
> >> > Initially, start with support for LIKE operator in "constraints" that
> >> > operates on “hostname” and “mesos slave attribute
> >> > <http://mesos.apache.org/documentation/attributes-resources/>” fields
> >> > using
> >> > regex.
> >> > 2. Add a new “profile” parameter in the flexdown API.
> >> >
> >> > #1 would give admins a better control over where the NMs should be
> >> launched
> >> > or which specific NMs should be shut down.
> >> > #2 helps with shutting down multiple NMs of same "profile" in one
> shot.
> >> >
> >> > If the above sounds reasonable to start with, I'd like to start
> working
> >> on
> >> > the code changes.
> >> >
> >> > Thanks,
> >> > Santosh
> >> >
> >> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <
> smarella@maprtech.com
> >> >
> >> > wrote:
> >> >
> >> > > > One additional thing I'd like is the ability to flexup a nmnode on
> >> all
> >> > > > agents with a certain attribute as opposed to a fixed number.
> >> > >
> >> > > Certainly! Marathon places tasks on agents sharing a common
> attribute.
> >> > > Myriad could/should do the same for placing NMs. If each agent
> running
> >> > > HDFS data node shares an attribute such as "dfs", then NMs could
> >> always
> >> > > be launched on the "dfs" nodes. This improves data locality for YARN
> >> jobs
> >> > > dramatically.
> >> > >
> >> > > Santosh
> >> > >
> >> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <
> >> dbjohnson1978@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> One additional thing I'd like is the ability to flexup a nmnode on
> >> all
> >> > >> agents with a certain attribute as opposed to a fixed number.
> >> > >>
> >> > >
> >> > >
> >> > >>
> >> > >> Also, when I get back from vacation I plan on scoping mesos-1739
> >> > (dynamic
> >> > >> attributes) which would allow for tighter integration with the hdfs
> >> > >> framework. An alternative would be to get hostnames from the
> >> namenode,
> >> > >> though not as seemless.
> >> > >> (Reviving this thread)
> >> > >>
> >> > >> We've discussed several great points in the thread (PUT vs POST,
> need
> >> > for
> >> > >> GET, JSON payload vs parameters in URL, declarative interface etc).
> >> > >> Just to get us going, I think we should focus on a couple of things
> >> that
> >> > >> will be useful for Myriad users, while leaving them flexible enough
> >> to
> >> > be
> >> > >> evolved in the future.
> >> > >>
> >> > >> What I heard from several folks (some of it brought up again at
> >> > MesosCon)
> >> > >> about the flex up/down APIs is this:
> >> > >> - flexup doesn't support launching NMs on specific set of hosts.
> >> This is
> >> > >> especially needed to launch NMs on same set of nodes that have HDFS
> >> > >> DataNode running.
> >> > >> - flexdown lacks an option to shut down NMs with a specific
> profile.
> >> > >> Today,
> >> > >> we bring down ANY arbitrary NM.
> >> > >> - flexdown lacks an option to shutdown NMs running on specific
> hosts.
> >> > >>
> >> > >> I captured my thoughts in a document here:
> >> > >>
> >> > >>
> >> >
> >>
> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
> >> > >>
> >> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc)
> >> to
> >> > >> finalize the APIs. Just let me know.
> >> > >>
> >> > >> Thanks,
> >> > >> Santosh
> >> > >>
> >> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com>
> >> wrote:
> >> > >>
> >> > >> > Seems like POST is a winner with people.
> >> > >> >
> >> > >> > Another thing to consider is how we want the REST interface to be
> >> vs
> >> > >> what
> >> > >> > we want the UI to do. The UI could support flexup/flexdown like
> it
> >> is
> >> > >> while
> >> > >> > the REST interface is just a declarative state like Adam
> suggested.
> >> > The
> >> > >> UI
> >> > >> > would just be responsible for translating the request into the
> new
> >> > >> state.
> >> > >> >
> >> > >> > Tomorrow I'll try to put together another swagger doc with some
> of
> >> the
> >> > >> > suggested options.
> >> > >> >
> >> > >> >
> >> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> >> > >> > <yufeldman@yahoo.com.invalid
> >> > >> > > wrote:
> >> > >> >
> >> > >> > > I think we are at the point to list all the options we want
> >> "flex"
> >> > API
> >> > >> to
> >> > >> > > support.
> >> > >> > > 1. Do we continue supporting flexup/down or just "flex" with
> >> > >> additional
> >> > >> > > "preposition" like up/down:https://hostname:port/flex/up(down)
> >> > >> > > 2. I think we should switch to POST and may be maintain PUT for
> >> > legacy
> >> > >> > (if
> >> > >> > > even needed to keep it). We are not DB after all and not
> storing
> >> any
> >> > >> > > retrievable info here :)
> >> > >> > > 3. We need to add status (GET) to see the status - though I
> >> think we
> >> > >> have
> >> > >> > > one
> >> > >> > > 4. Define JSON payload to support different cases    a.
> providing
> >> > >> > > different profiles together: [{profile:"big",
> >> > >> > > instances:2},{profile:"medium",instances:6}]    b. provide what
> >> > state
> >> > >> we
> >> > >> > > want Myriad to be in: "I want 10 medium instances" and then
> >> Myriad
> >> > >> will
> >> > >> > do
> >> > >> > > whatever isnecessary to transition to that state,
> >> > >> > adding/removing/resizing
> >> > >> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
> >> > >> preferred
> >> > >> > > hosts, delays, others
> >> > >> > > 5. How all this fits into FineGrain Scaling? With it we would
> do
> >> > >> > automatic
> >> > >> > > flex up/down. And the less knobs admin will have to turn the
> >> easier
> >> > it
> >> > >> is
> >> > >> > > for admin and the end users.
> >> > >> > >
> >> > >> > >       From: Adam Bordelon <ad...@mesosphere.io>
> >> > >> > >  To: dev@myriad.incubator.apache.org
> >> > >> > >  Sent: Sunday, June 14, 2015 2:54 PM
> >> > >> > >  Subject: Re: Flex API
> >> > >> > >
> >> > >> > > (In addition,) I'd also like to see a more declarative
> interface.
> >> > >> Instead
> >> > >> > > of "add two more instances", the user(s) could just specify the
> >> > >> desired
> >> > >> > > state of "I want 10 medium instances" and then Myriad will do
> >> > whatever
> >> > >> is
> >> > >> > > necessary to transition to that state, adding/removing/resizing
> >> NMs
> >> > as
> >> > >> > > necessary.
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> >> > >> > > wochandarena@maprtech.com
> >> > >> > > > wrote:
> >> > >> > >
> >> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <
> klucar@gmail.com>
> >> > >> wrote:
> >> > >> > > >
> >> > >> > > > > What verb to use when outside of database land can be
> >> argued. I
> >> > >> would
> >> > >> > > > vote
> >> > >> > > > > for POST over PUT just because I tend to default to POST.
> PUT
> >> > was
> >> > >> > there
> >> > >> > > > > when I showed up, so I left it.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > Last time I agonized about PUT vs POST the most logical
> >> > distinction
> >> > >> I
> >> > >> > > found
> >> > >> > > > was that PUT should be used for idempotent operations, while
> >> POST
> >> > >> for
> >> > >> > > > non-idempotent (like we have here with flex-up, since
> >> instance-ids
> >> > >> are
> >> > >> > > > generated).
> >> > >> > > >
> >> > >> > > > Since the api doesn't wait until the
> >> > >> > > > > instances are created to return, we can't really return the
> >> > >> instance
> >> > >> > > IDs
> >> > >> > > > we
> >> > >> > > > > created.
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > That seems OK to me.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > > The GET would just return some status?
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > Yeah, I was thinking that this would be needed for a future
> GUI
> >> > >> where
> >> > >> > we
> >> > >> > > > list all instances with parameters and status for each
> >> (profile,
> >> > >> > current
> >> > >> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next
> to
> >> > each
> >> > >> so
> >> > >> > > > users can multi-select and hit 'delete' to wipe them away
> (like
> >> > >> > flex-down
> >> > >> > > > does now).
> >> > >> > > >
> >> > >> > > > The PATCH is interesting
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies
> you
> >> > >> always
> >> > >> > > have
> >> > >> > > > to rewrite the complete object when making changes.  PATCH
> >> allows
> >> > >> more
> >> > >> > > > flexible modifications.
> >> > >> > > >
> >> > >> > > > The DELETE makes sense to me.
> >> > >> > > > >
> >> > >> > > > > Your use of instances vs instance is interesting. Perhaps
> we
> >> > want
> >> > >> to
> >> > >> > > > > support POST [{profile:"big",
> instances:2},{profile:"medium",
> >> > >> > > > instances:6}]
> >> > >> > > > >
> >> > >> > > >
> >> > >> > > > Yeah, that'd be cool!
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> >> > >> > > > > wochandarena@maprtech.com
> >> > >> > > > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Any reason we need to make it action-based
> >> (flex-up/flex-down)
> >> > >> > rather
> >> > >> > > > > than
> >> > >> > > > > > instance outcome-based?  The way i've seen similar things
> >> > >> modeled
> >> > >> > in
> >> > >> > > > > other
> >> > >> > > > > > REST APIs is similar to below.  This may also fit better
> >> into
> >> > >> REST
> >> > >> > > > client
> >> > >> > > > > > frameworks.
> >> > >> > > > > >
> >> > >> > > > > > Create - POST /instances/ {"profile" : "zero",
> "instances"
> >> :
> >> > 2,
> >> > >> > > > "delay" :
> >> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
> >> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
> >> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid>
> >> {"profile" :
> >> > >> > "big"}
> >> > >> > > > > > Delete - DELETE /instances/<instanceid>
> >> > >> > > > > >
> >> > >> > > > > > Thoughts?
> >> > >> > > > > >
> >> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <
> >> klucar@gmail.com
> >> > >
> >> > >> > > wrote:
> >> > >> > > > > >
> >> > >> > > > > > > That is always an issue with parameters in the URL, and
> >> > >> there's
> >> > >> > > > nothing
> >> > >> > > > > > > worse than parameters in the URL and having to put JSON
> >> in
> >> > the
> >> > >> > body
> >> > >> > > > > > because
> >> > >> > > > > > > we want to extend it later. Currently flexup takes a
> JSON
> >> > blob
> >> > >> in
> >> > >> > > the
> >> > >> > > > > > body
> >> > >> > > > > > > which we could add more to like you showed. I guess the
> >> > >> question
> >> > >> > is
> >> > >> > > > if
> >> > >> > > > > we
> >> > >> > > > > > > want to support a quick and dirty flexup api via URL
> >> > >> parameters.
> >> > >> > I
> >> > >> > > > > > wouldn't
> >> > >> > > > > > > go any further than these parameters, but all this
> would
> >> do
> >> > at
> >> > >> > this
> >> > >> > > > > point
> >> > >> > > > > > > is make testing with a curl command easier. Getting rid
> >> of
> >> > it
> >> > >> for
> >> > >> > > > > flexup
> >> > >> > > > > > is
> >> > >> > > > > > > fine with me.
> >> > >> > > > > > >
> >> > >> > > > > > > I do like the flexdown/instance/{instance-id} though.
> We
> >> > >> should
> >> > >> > > > > probably
> >> > >> > > > > > do
> >> > >> > > > > > > one with flexdown/instance with a json array of
> >> instance-ids
> >> > >> too
> >> > >> > > > > though.
> >> > >> > > > > > >
> >> > >> > > > > > > Also with the parameter verification patch I just
> >> submitted,
> >> > >> if
> >> > >> > you
> >> > >> > > > ask
> >> > >> > > > > > to
> >> > >> > > > > > > flexdown 10 instances and you only have 5 running it
> will
> >> > just
> >> > >> > log
> >> > >> > > a
> >> > >> > > > > > > warning and kill all 5. Not sure if that is the desired
> >> > >> behavior.
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> >> > >> > > > > smarella@maprtech.com>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > >
> >> > >> > > > > > > > I think rather than having an API defined as
> >> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better
> >> to
> >> > >> define
> >> > >> > > it
> >> > >> > > > > just
> >> > >> > > > > > > as
> >> > >> > > > > > > > "/cluster/flexup" and have parameters such as
> "profile:
> >> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The
> >> > reason
> >> > >> is,
> >> > >> > > if
> >> > >> > > > > the
> >> > >> > > > > > > > params are added into the API endpoint, it becomes
> less
> >> > >> > flexible
> >> > >> > > to
> >> > >> > > > > > > evolve.
> >> > >> > > > > > > > For e.g. /cluster/flexup currently can take just
> >> "profile"
> >> > >> and
> >> > >> > > > > > > "instances"
> >> > >> > > > > > > > in it's payload, but in the future can optionally
> take
> >> > >> > parameters
> >> > >> > > > > such
> >> > >> > > > > > as
> >> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> >> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> >> > >> > > > > > > >
> >> > >> > > > > > > > Thoughts?
> >> > >> > > > > > > >
> >> > >> > > > > > > > Santosh
> >> > >> > > > > > > >
> >> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
> >> > >> klucar@gmail.com>
> >> > >> > > > > wrote:
> >> > >> > > > > > > >
> >> > >> > > > > > > > > I thought that might happen. I created a gist here:
> >> > >> > > > > > > > >
> https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> >> > >> > > > > > > > >
> >> > >> > > > > > > > > And just in case....
> >> > >> > > > > > > > >
> >> > >> > > > > > > > > swagger: '2.0'
> >> > >> > > > > > > > > info:
> >> > >> > > > > > > > >  version: 0.0.1
> >> > >> > > > > > > > >  title: Myriad API
> >> > >> > > > > > > > >  description: |
> >> > >> > > > > > > > >    Myriad API description
> >> > >> > > > > > > > > basePath: /api
> >> > >> > > > > > > > > schemes:
> >> > >> > > > > > > > >  - http
> >> > >> > > > > > > > > consumes:
> >> > >> > > > > > > > >  - application/json
> >> > >> > > > > > > > >  - text/plain
> >> > >> > > > > > > > > produces:
> >> > >> > > > > > > > >  - application/json
> >> > >> > > > > > > > >  - text/plain
> >> > >> > > > > > > > > paths:
> >> > >> > > > > > > > >  /cluster/flexup:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: instance
> >> > >> > > > > > > > >          in: body
> >> > >> > > > > > > > >          description: Instance profile and quantity
> >> to
> >> > >> flex
> >> > >> > up
> >> > >> > > > > > > > >          schema:
> >> > >> > > > > > > > >            $ref: '#/definitions/FlexUp'
> >> > >> > > > > > > > >          required: true
> >> > >> > > > > > > > >      description: Original Flexup API
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: profile-name
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: string
> >> > >> > > > > > > > >          description: Instance profile name
> >> > >> > > > > > > > >      description: Flexup a single instance of a
> >> > specified
> >> > >> > > > profile
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >
> /cluster/flexup/profile/{profile-name}/{instances}:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: profile-name
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: string
> >> > >> > > > > > > > >          description: Instance profile name
> >> > >> > > > > > > > >        - name: instances
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: integer
> >> > >> > > > > > > > >          description: Number of instances
> >> > >> > > > > > > > >      description: Flexup a multiple instances of a
> >> > >> specified
> >> > >> > > > > profile
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >  /cluster/flexdown:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: instance
> >> > >> > > > > > > > >          in: body
> >> > >> > > > > > > > >          description: Number of instances to flex
> >> down
> >> > >> > > > > > > > >          schema:
> >> > >> > > > > > > > >            $ref: '#/definitions/FlexDown'
> >> > >> > > > > > > > >          required: true
> >> > >> > > > > > > > >      description: Original Flexdown API
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: profile-name
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: string
> >> > >> > > > > > > > >          description: Instance profile name
> >> > >> > > > > > > > >      description: Flexup a single instance of a
> >> > specified
> >> > >> > > > profile
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >
> >> /cluster/flexdown/profile/{profile-name}/{instances}:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: profile-name
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: string
> >> > >> > > > > > > > >          description: Instance profile name
> >> > >> > > > > > > > >        - name: instances
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: integer
> >> > >> > > > > > > > >          description: Number of instances
> >> > >> > > > > > > > >      description: Flexdown a multiple instances of
> a
> >> > >> > specified
> >> > >> > > > > > profile
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> >> > >> > > > > > > > >    put:
> >> > >> > > > > > > > >      parameters:
> >> > >> > > > > > > > >        - name: instance-id
> >> > >> > > > > > > > >          in: path
> >> > >> > > > > > > > >          type: string
> >> > >> > > > > > > > >          description: Instance profile name
> >> > >> > > > > > > > >      description: Flexup a single instance of a
> >> > specified
> >> > >> > > > profile
> >> > >> > > > > > > > >      responses:
> >> > >> > > > > > > > >        200:
> >> > >> > > > > > > > >          description: OK
> >> > >> > > > > > > > > definitions:
> >> > >> > > > > > > > >  FlexUp:
> >> > >> > > > > > > > >    properties:
> >> > >> > > > > > > > >      profile:
> >> > >> > > > > > > > >        type: string
> >> > >> > > > > > > > >      instances:
> >> > >> > > > > > > > >        type: integer
> >> > >> > > > > > > > >        format: int32
> >> > >> > > > > > > > >  FlexDown:
> >> > >> > > > > > > > >    properties:
> >> > >> > > > > > > > >      instances:
> >> > >> > > > > > > > >        type: integer
> >> > >> > > > > > > > >        format: int32
> >> > >> > > > > > > > >
> >> > >> > > > > > > > >
> >> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> >> > >> > > > > > > smarella@maprtech.com>
> >> > >> > > > > > > > > wrote:
> >> > >> > > > > > > > >
> >> > >> > > > > > > > > > Hi Jim,
> >> > >> > > > > > > > > >
> >> > >> > > > > > > > > >  Did you attach a file? I think the apache
> mailing
> >> > list
> >> > >> > > > > swallowed
> >> > >> > > > > > > the
> >> > >> > > > > > > > > > attachment. Can you please send us the document
> >> > contents
> >> > >> in
> >> > >> > > > plain
> >> > >> > > > > > > text?
> >> > >> > > > > > > > > >
> >> > >> > > > > > > > > > Thanks,
> >> > >> > > > > > > > > > Santosh
> >> > >> > > > > > > > > >
> >> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> >> > >> > > klucar@gmail.com>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > > > > >
> >> > >> > > > > > > > > > > This is in regards to
> >> > >> > > > > https://github.com/mesos/myriad/issues/89
> >> > >> > > > > > > > > > >
> >> > >> > > > > > > > > > > I created a swagger document (attached) of the
> >> > current
> >> > >> > flex
> >> > >> > > > > APIs
> >> > >> > > > > > > and
> >> > >> > > > > > > > > what
> >> > >> > > > > > > > > > > I propose for the new flex apis. If people like
> >> > >> swagger,
> >> > >> > I
> >> > >> > > > can
> >> > >> > > > > > > > document
> >> > >> > > > > > > > > > the
> >> > >> > > > > > > > > > > rest of the API, including the correct response
> >> > codes.
> >> > >> > > Either
> >> > >> > > > > > way I
> >> > >> > > > > > > > > think
> >> > >> > > > > > > > > > > we should discuss what the API should be going
> >> > >> forward.
> >> > >> > > > > > > > > > >
> >> > >> > > > > > > > > > > Paste the file contents into
> >> > >> http://editor.swagger.io/
> >> > >> > and
> >> > >> > > > > > you'll
> >> > >> > > > > > > > get
> >> > >> > > > > > > > > > > nice HTML to browse.
> >> > >> > > > > > > > > > >
> >> > >> > > > > > > > > > >
> >> > >> > > > > > > > > > >
> >> > >> > > > > > > > > >
> >> > >> > > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Flex API

Posted by Santosh Marella <sm...@maprtech.com>.
Addressed review comments from Yuliya and updated the PR:
https://github.com/mesos/myriad/pull/129

@Jim - Your UI expertise is much sought to support "constraints" parameter
in both flexup/down and "profile" in flexdown.
The API doc is updated as part of the PR:
https://github.com/mesos/myriad/pull/129/files#diff-96cc378dcdef180641176501f42183c7

Thanks,
Santosh

On Mon, Sep 14, 2015 at 10:54 AM, Santosh Marella <sm...@maprtech.com>
wrote:

> The proposal is to have "constraints" parameter supported for both
> flexup/flexdown.
> So, one could flexdown NMs on specific hosts, for e..g by providing a
> constraint
> around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE "
> specific.host.com"]]}.
>
> Santosh
>
> On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <ad...@mesosphere.io> wrote:
>
>> +1, but EQUALS might be an easier starting point than LIKE. Your choice.
>> What about flexing down specific hosts? Is that out of scope for now?
>>
>> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <sm...@maprtech.com>
>> wrote:
>>
>> > Ping! Checking if there is anymore feedback on the APIs for flexup/down.
>> >
>> > Here is a textual summary of what's being proposed:
>> > 1. Add a new “constraints” parameter in both flexup and flexdown APIs
>> (just
>> > like how Marathon does).
>> > Initially, start with support for LIKE operator in "constraints" that
>> > operates on “hostname” and “mesos slave attribute
>> > <http://mesos.apache.org/documentation/attributes-resources/>” fields
>> > using
>> > regex.
>> > 2. Add a new “profile” parameter in the flexdown API.
>> >
>> > #1 would give admins a better control over where the NMs should be
>> launched
>> > or which specific NMs should be shut down.
>> > #2 helps with shutting down multiple NMs of same "profile" in one shot.
>> >
>> > If the above sounds reasonable to start with, I'd like to start working
>> on
>> > the code changes.
>> >
>> > Thanks,
>> > Santosh
>> >
>> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <smarella@maprtech.com
>> >
>> > wrote:
>> >
>> > > > One additional thing I'd like is the ability to flexup a nmnode on
>> all
>> > > > agents with a certain attribute as opposed to a fixed number.
>> > >
>> > > Certainly! Marathon places tasks on agents sharing a common attribute.
>> > > Myriad could/should do the same for placing NMs. If each agent running
>> > > HDFS data node shares an attribute such as "dfs", then NMs could
>> always
>> > > be launched on the "dfs" nodes. This improves data locality for YARN
>> jobs
>> > > dramatically.
>> > >
>> > > Santosh
>> > >
>> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <
>> dbjohnson1978@gmail.com>
>> > > wrote:
>> > >
>> > >> One additional thing I'd like is the ability to flexup a nmnode on
>> all
>> > >> agents with a certain attribute as opposed to a fixed number.
>> > >>
>> > >
>> > >
>> > >>
>> > >> Also, when I get back from vacation I plan on scoping mesos-1739
>> > (dynamic
>> > >> attributes) which would allow for tighter integration with the hdfs
>> > >> framework. An alternative would be to get hostnames from the
>> namenode,
>> > >> though not as seemless.
>> > >> (Reviving this thread)
>> > >>
>> > >> We've discussed several great points in the thread (PUT vs POST, need
>> > for
>> > >> GET, JSON payload vs parameters in URL, declarative interface etc).
>> > >> Just to get us going, I think we should focus on a couple of things
>> that
>> > >> will be useful for Myriad users, while leaving them flexible enough
>> to
>> > be
>> > >> evolved in the future.
>> > >>
>> > >> What I heard from several folks (some of it brought up again at
>> > MesosCon)
>> > >> about the flex up/down APIs is this:
>> > >> - flexup doesn't support launching NMs on specific set of hosts.
>> This is
>> > >> especially needed to launch NMs on same set of nodes that have HDFS
>> > >> DataNode running.
>> > >> - flexdown lacks an option to shut down NMs with a specific profile.
>> > >> Today,
>> > >> we bring down ANY arbitrary NM.
>> > >> - flexdown lacks an option to shutdown NMs running on specific hosts.
>> > >>
>> > >> I captured my thoughts in a document here:
>> > >>
>> > >>
>> >
>> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
>> > >>
>> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc)
>> to
>> > >> finalize the APIs. Just let me know.
>> > >>
>> > >> Thanks,
>> > >> Santosh
>> > >>
>> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com>
>> wrote:
>> > >>
>> > >> > Seems like POST is a winner with people.
>> > >> >
>> > >> > Another thing to consider is how we want the REST interface to be
>> vs
>> > >> what
>> > >> > we want the UI to do. The UI could support flexup/flexdown like it
>> is
>> > >> while
>> > >> > the REST interface is just a declarative state like Adam suggested.
>> > The
>> > >> UI
>> > >> > would just be responsible for translating the request into the new
>> > >> state.
>> > >> >
>> > >> > Tomorrow I'll try to put together another swagger doc with some of
>> the
>> > >> > suggested options.
>> > >> >
>> > >> >
>> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
>> > >> > <yufeldman@yahoo.com.invalid
>> > >> > > wrote:
>> > >> >
>> > >> > > I think we are at the point to list all the options we want
>> "flex"
>> > API
>> > >> to
>> > >> > > support.
>> > >> > > 1. Do we continue supporting flexup/down or just "flex" with
>> > >> additional
>> > >> > > "preposition" like up/down:https://hostname:port/flex/up(down)
>> > >> > > 2. I think we should switch to POST and may be maintain PUT for
>> > legacy
>> > >> > (if
>> > >> > > even needed to keep it). We are not DB after all and not storing
>> any
>> > >> > > retrievable info here :)
>> > >> > > 3. We need to add status (GET) to see the status - though I
>> think we
>> > >> have
>> > >> > > one
>> > >> > > 4. Define JSON payload to support different cases    a. providing
>> > >> > > different profiles together: [{profile:"big",
>> > >> > > instances:2},{profile:"medium",instances:6}]    b. provide what
>> > state
>> > >> we
>> > >> > > want Myriad to be in: "I want 10 medium instances" and then
>> Myriad
>> > >> will
>> > >> > do
>> > >> > > whatever isnecessary to transition to that state,
>> > >> > adding/removing/resizing
>> > >> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
>> > >> preferred
>> > >> > > hosts, delays, others
>> > >> > > 5. How all this fits into FineGrain Scaling? With it we would do
>> > >> > automatic
>> > >> > > flex up/down. And the less knobs admin will have to turn the
>> easier
>> > it
>> > >> is
>> > >> > > for admin and the end users.
>> > >> > >
>> > >> > >       From: Adam Bordelon <ad...@mesosphere.io>
>> > >> > >  To: dev@myriad.incubator.apache.org
>> > >> > >  Sent: Sunday, June 14, 2015 2:54 PM
>> > >> > >  Subject: Re: Flex API
>> > >> > >
>> > >> > > (In addition,) I'd also like to see a more declarative interface.
>> > >> Instead
>> > >> > > of "add two more instances", the user(s) could just specify the
>> > >> desired
>> > >> > > state of "I want 10 medium instances" and then Myriad will do
>> > whatever
>> > >> is
>> > >> > > necessary to transition to that state, adding/removing/resizing
>> NMs
>> > as
>> > >> > > necessary.
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
>> > >> > > wochandarena@maprtech.com
>> > >> > > > wrote:
>> > >> > >
>> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com>
>> > >> wrote:
>> > >> > > >
>> > >> > > > > What verb to use when outside of database land can be
>> argued. I
>> > >> would
>> > >> > > > vote
>> > >> > > > > for POST over PUT just because I tend to default to POST. PUT
>> > was
>> > >> > there
>> > >> > > > > when I showed up, so I left it.
>> > >> > > >
>> > >> > > >
>> > >> > > > Last time I agonized about PUT vs POST the most logical
>> > distinction
>> > >> I
>> > >> > > found
>> > >> > > > was that PUT should be used for idempotent operations, while
>> POST
>> > >> for
>> > >> > > > non-idempotent (like we have here with flex-up, since
>> instance-ids
>> > >> are
>> > >> > > > generated).
>> > >> > > >
>> > >> > > > Since the api doesn't wait until the
>> > >> > > > > instances are created to return, we can't really return the
>> > >> instance
>> > >> > > IDs
>> > >> > > > we
>> > >> > > > > created.
>> > >> > > > >
>> > >> > > >
>> > >> > > > That seems OK to me.
>> > >> > > >
>> > >> > > >
>> > >> > > > > The GET would just return some status?
>> > >> > > > >
>> > >> > > >
>> > >> > > > Yeah, I was thinking that this would be needed for a future GUI
>> > >> where
>> > >> > we
>> > >> > > > list all instances with parameters and status for each
>> (profile,
>> > >> > current
>> > >> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to
>> > each
>> > >> so
>> > >> > > > users can multi-select and hit 'delete' to wipe them away (like
>> > >> > flex-down
>> > >> > > > does now).
>> > >> > > >
>> > >> > > > The PATCH is interesting
>> > >> > > > >
>> > >> > > >
>> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you
>> > >> always
>> > >> > > have
>> > >> > > > to rewrite the complete object when making changes.  PATCH
>> allows
>> > >> more
>> > >> > > > flexible modifications.
>> > >> > > >
>> > >> > > > The DELETE makes sense to me.
>> > >> > > > >
>> > >> > > > > Your use of instances vs instance is interesting. Perhaps we
>> > want
>> > >> to
>> > >> > > > > support POST [{profile:"big", instances:2},{profile:"medium",
>> > >> > > > instances:6}]
>> > >> > > > >
>> > >> > > >
>> > >> > > > Yeah, that'd be cool!
>> > >> > > >
>> > >> > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
>> > >> > > > > wochandarena@maprtech.com
>> > >> > > > > > wrote:
>> > >> > > > >
>> > >> > > > > > Any reason we need to make it action-based
>> (flex-up/flex-down)
>> > >> > rather
>> > >> > > > > than
>> > >> > > > > > instance outcome-based?  The way i've seen similar things
>> > >> modeled
>> > >> > in
>> > >> > > > > other
>> > >> > > > > > REST APIs is similar to below.  This may also fit better
>> into
>> > >> REST
>> > >> > > > client
>> > >> > > > > > frameworks.
>> > >> > > > > >
>> > >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances"
>> :
>> > 2,
>> > >> > > > "delay" :
>> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
>> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
>> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid>
>> {"profile" :
>> > >> > "big"}
>> > >> > > > > > Delete - DELETE /instances/<instanceid>
>> > >> > > > > >
>> > >> > > > > > Thoughts?
>> > >> > > > > >
>> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <
>> klucar@gmail.com
>> > >
>> > >> > > wrote:
>> > >> > > > > >
>> > >> > > > > > > That is always an issue with parameters in the URL, and
>> > >> there's
>> > >> > > > nothing
>> > >> > > > > > > worse than parameters in the URL and having to put JSON
>> in
>> > the
>> > >> > body
>> > >> > > > > > because
>> > >> > > > > > > we want to extend it later. Currently flexup takes a JSON
>> > blob
>> > >> in
>> > >> > > the
>> > >> > > > > > body
>> > >> > > > > > > which we could add more to like you showed. I guess the
>> > >> question
>> > >> > is
>> > >> > > > if
>> > >> > > > > we
>> > >> > > > > > > want to support a quick and dirty flexup api via URL
>> > >> parameters.
>> > >> > I
>> > >> > > > > > wouldn't
>> > >> > > > > > > go any further than these parameters, but all this would
>> do
>> > at
>> > >> > this
>> > >> > > > > point
>> > >> > > > > > > is make testing with a curl command easier. Getting rid
>> of
>> > it
>> > >> for
>> > >> > > > > flexup
>> > >> > > > > > is
>> > >> > > > > > > fine with me.
>> > >> > > > > > >
>> > >> > > > > > > I do like the flexdown/instance/{instance-id} though. We
>> > >> should
>> > >> > > > > probably
>> > >> > > > > > do
>> > >> > > > > > > one with flexdown/instance with a json array of
>> instance-ids
>> > >> too
>> > >> > > > > though.
>> > >> > > > > > >
>> > >> > > > > > > Also with the parameter verification patch I just
>> submitted,
>> > >> if
>> > >> > you
>> > >> > > > ask
>> > >> > > > > > to
>> > >> > > > > > > flexdown 10 instances and you only have 5 running it will
>> > just
>> > >> > log
>> > >> > > a
>> > >> > > > > > > warning and kill all 5. Not sure if that is the desired
>> > >> behavior.
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
>> > >> > > > > smarella@maprtech.com>
>> > >> > > > > > > wrote:
>> > >> > > > > > >
>> > >> > > > > > > > I think rather than having an API defined as
>> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better
>> to
>> > >> define
>> > >> > > it
>> > >> > > > > just
>> > >> > > > > > > as
>> > >> > > > > > > > "/cluster/flexup" and have parameters such as "profile:
>> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The
>> > reason
>> > >> is,
>> > >> > > if
>> > >> > > > > the
>> > >> > > > > > > > params are added into the API endpoint, it becomes less
>> > >> > flexible
>> > >> > > to
>> > >> > > > > > > evolve.
>> > >> > > > > > > > For e.g. /cluster/flexup currently can take just
>> "profile"
>> > >> and
>> > >> > > > > > > "instances"
>> > >> > > > > > > > in it's payload, but in the future can optionally take
>> > >> > parameters
>> > >> > > > > such
>> > >> > > > > > as
>> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
>> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
>> > >> > > > > > > >
>> > >> > > > > > > > Thoughts?
>> > >> > > > > > > >
>> > >> > > > > > > > Santosh
>> > >> > > > > > > >
>> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
>> > >> klucar@gmail.com>
>> > >> > > > > wrote:
>> > >> > > > > > > >
>> > >> > > > > > > > > I thought that might happen. I created a gist here:
>> > >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
>> > >> > > > > > > > >
>> > >> > > > > > > > > And just in case....
>> > >> > > > > > > > >
>> > >> > > > > > > > > swagger: '2.0'
>> > >> > > > > > > > > info:
>> > >> > > > > > > > >  version: 0.0.1
>> > >> > > > > > > > >  title: Myriad API
>> > >> > > > > > > > >  description: |
>> > >> > > > > > > > >    Myriad API description
>> > >> > > > > > > > > basePath: /api
>> > >> > > > > > > > > schemes:
>> > >> > > > > > > > >  - http
>> > >> > > > > > > > > consumes:
>> > >> > > > > > > > >  - application/json
>> > >> > > > > > > > >  - text/plain
>> > >> > > > > > > > > produces:
>> > >> > > > > > > > >  - application/json
>> > >> > > > > > > > >  - text/plain
>> > >> > > > > > > > > paths:
>> > >> > > > > > > > >  /cluster/flexup:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: instance
>> > >> > > > > > > > >          in: body
>> > >> > > > > > > > >          description: Instance profile and quantity
>> to
>> > >> flex
>> > >> > up
>> > >> > > > > > > > >          schema:
>> > >> > > > > > > > >            $ref: '#/definitions/FlexUp'
>> > >> > > > > > > > >          required: true
>> > >> > > > > > > > >      description: Original Flexup API
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: profile-name
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: string
>> > >> > > > > > > > >          description: Instance profile name
>> > >> > > > > > > > >      description: Flexup a single instance of a
>> > specified
>> > >> > > > profile
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: profile-name
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: string
>> > >> > > > > > > > >          description: Instance profile name
>> > >> > > > > > > > >        - name: instances
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: integer
>> > >> > > > > > > > >          description: Number of instances
>> > >> > > > > > > > >      description: Flexup a multiple instances of a
>> > >> specified
>> > >> > > > > profile
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >  /cluster/flexdown:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: instance
>> > >> > > > > > > > >          in: body
>> > >> > > > > > > > >          description: Number of instances to flex
>> down
>> > >> > > > > > > > >          schema:
>> > >> > > > > > > > >            $ref: '#/definitions/FlexDown'
>> > >> > > > > > > > >          required: true
>> > >> > > > > > > > >      description: Original Flexdown API
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: profile-name
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: string
>> > >> > > > > > > > >          description: Instance profile name
>> > >> > > > > > > > >      description: Flexup a single instance of a
>> > specified
>> > >> > > > profile
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >
>> /cluster/flexdown/profile/{profile-name}/{instances}:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: profile-name
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: string
>> > >> > > > > > > > >          description: Instance profile name
>> > >> > > > > > > > >        - name: instances
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: integer
>> > >> > > > > > > > >          description: Number of instances
>> > >> > > > > > > > >      description: Flexdown a multiple instances of a
>> > >> > specified
>> > >> > > > > > profile
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
>> > >> > > > > > > > >    put:
>> > >> > > > > > > > >      parameters:
>> > >> > > > > > > > >        - name: instance-id
>> > >> > > > > > > > >          in: path
>> > >> > > > > > > > >          type: string
>> > >> > > > > > > > >          description: Instance profile name
>> > >> > > > > > > > >      description: Flexup a single instance of a
>> > specified
>> > >> > > > profile
>> > >> > > > > > > > >      responses:
>> > >> > > > > > > > >        200:
>> > >> > > > > > > > >          description: OK
>> > >> > > > > > > > > definitions:
>> > >> > > > > > > > >  FlexUp:
>> > >> > > > > > > > >    properties:
>> > >> > > > > > > > >      profile:
>> > >> > > > > > > > >        type: string
>> > >> > > > > > > > >      instances:
>> > >> > > > > > > > >        type: integer
>> > >> > > > > > > > >        format: int32
>> > >> > > > > > > > >  FlexDown:
>> > >> > > > > > > > >    properties:
>> > >> > > > > > > > >      instances:
>> > >> > > > > > > > >        type: integer
>> > >> > > > > > > > >        format: int32
>> > >> > > > > > > > >
>> > >> > > > > > > > >
>> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
>> > >> > > > > > > smarella@maprtech.com>
>> > >> > > > > > > > > wrote:
>> > >> > > > > > > > >
>> > >> > > > > > > > > > Hi Jim,
>> > >> > > > > > > > > >
>> > >> > > > > > > > > >  Did you attach a file? I think the apache mailing
>> > list
>> > >> > > > > swallowed
>> > >> > > > > > > the
>> > >> > > > > > > > > > attachment. Can you please send us the document
>> > contents
>> > >> in
>> > >> > > > plain
>> > >> > > > > > > text?
>> > >> > > > > > > > > >
>> > >> > > > > > > > > > Thanks,
>> > >> > > > > > > > > > Santosh
>> > >> > > > > > > > > >
>> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
>> > >> > > klucar@gmail.com>
>> > >> > > > > > > wrote:
>> > >> > > > > > > > > >
>> > >> > > > > > > > > > > This is in regards to
>> > >> > > > > https://github.com/mesos/myriad/issues/89
>> > >> > > > > > > > > > >
>> > >> > > > > > > > > > > I created a swagger document (attached) of the
>> > current
>> > >> > flex
>> > >> > > > > APIs
>> > >> > > > > > > and
>> > >> > > > > > > > > what
>> > >> > > > > > > > > > > I propose for the new flex apis. If people like
>> > >> swagger,
>> > >> > I
>> > >> > > > can
>> > >> > > > > > > > document
>> > >> > > > > > > > > > the
>> > >> > > > > > > > > > > rest of the API, including the correct response
>> > codes.
>> > >> > > Either
>> > >> > > > > > way I
>> > >> > > > > > > > > think
>> > >> > > > > > > > > > > we should discuss what the API should be going
>> > >> forward.
>> > >> > > > > > > > > > >
>> > >> > > > > > > > > > > Paste the file contents into
>> > >> http://editor.swagger.io/
>> > >> > and
>> > >> > > > > > you'll
>> > >> > > > > > > > get
>> > >> > > > > > > > > > > nice HTML to browse.
>> > >> > > > > > > > > > >
>> > >> > > > > > > > > > >
>> > >> > > > > > > > > > >
>> > >> > > > > > > > > >
>> > >> > > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: Flex API

Posted by Santosh Marella <sm...@maprtech.com>.
The proposal is to have "constraints" parameter supported for both
flexup/flexdown.
So, one could flexdown NMs on specific hosts, for e..g by providing a
constraint
around "hostname" (/api/flexdown {"constraints": [["hostname" LIKE "
specific.host.com"]]}.

Santosh

On Mon, Sep 14, 2015 at 2:31 AM, Adam Bordelon <ad...@mesosphere.io> wrote:

> +1, but EQUALS might be an easier starting point than LIKE. Your choice.
> What about flexing down specific hosts? Is that out of scope for now?
>
> On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
> > Ping! Checking if there is anymore feedback on the APIs for flexup/down.
> >
> > Here is a textual summary of what's being proposed:
> > 1. Add a new “constraints” parameter in both flexup and flexdown APIs
> (just
> > like how Marathon does).
> > Initially, start with support for LIKE operator in "constraints" that
> > operates on “hostname” and “mesos slave attribute
> > <http://mesos.apache.org/documentation/attributes-resources/>” fields
> > using
> > regex.
> > 2. Add a new “profile” parameter in the flexdown API.
> >
> > #1 would give admins a better control over where the NMs should be
> launched
> > or which specific NMs should be shut down.
> > #2 helps with shutting down multiple NMs of same "profile" in one shot.
> >
> > If the above sounds reasonable to start with, I'd like to start working
> on
> > the code changes.
> >
> > Thanks,
> > Santosh
> >
> > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <sm...@maprtech.com>
> > wrote:
> >
> > > > One additional thing I'd like is the ability to flexup a nmnode on
> all
> > > > agents with a certain attribute as opposed to a fixed number.
> > >
> > > Certainly! Marathon places tasks on agents sharing a common attribute.
> > > Myriad could/should do the same for placing NMs. If each agent running
> > > HDFS data node shares an attribute such as "dfs", then NMs could always
> > > be launched on the "dfs" nodes. This improves data locality for YARN
> jobs
> > > dramatically.
> > >
> > > Santosh
> > >
> > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <
> dbjohnson1978@gmail.com>
> > > wrote:
> > >
> > >> One additional thing I'd like is the ability to flexup a nmnode on all
> > >> agents with a certain attribute as opposed to a fixed number.
> > >>
> > >
> > >
> > >>
> > >> Also, when I get back from vacation I plan on scoping mesos-1739
> > (dynamic
> > >> attributes) which would allow for tighter integration with the hdfs
> > >> framework. An alternative would be to get hostnames from the namenode,
> > >> though not as seemless.
> > >> (Reviving this thread)
> > >>
> > >> We've discussed several great points in the thread (PUT vs POST, need
> > for
> > >> GET, JSON payload vs parameters in URL, declarative interface etc).
> > >> Just to get us going, I think we should focus on a couple of things
> that
> > >> will be useful for Myriad users, while leaving them flexible enough to
> > be
> > >> evolved in the future.
> > >>
> > >> What I heard from several folks (some of it brought up again at
> > MesosCon)
> > >> about the flex up/down APIs is this:
> > >> - flexup doesn't support launching NMs on specific set of hosts. This
> is
> > >> especially needed to launch NMs on same set of nodes that have HDFS
> > >> DataNode running.
> > >> - flexdown lacks an option to shut down NMs with a specific profile.
> > >> Today,
> > >> we bring down ANY arbitrary NM.
> > >> - flexdown lacks an option to shutdown NMs running on specific hosts.
> > >>
> > >> I captured my thoughts in a document here:
> > >>
> > >>
> >
> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
> > >>
> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
> > >> finalize the APIs. Just let me know.
> > >>
> > >> Thanks,
> > >> Santosh
> > >>
> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:
> > >>
> > >> > Seems like POST is a winner with people.
> > >> >
> > >> > Another thing to consider is how we want the REST interface to be vs
> > >> what
> > >> > we want the UI to do. The UI could support flexup/flexdown like it
> is
> > >> while
> > >> > the REST interface is just a declarative state like Adam suggested.
> > The
> > >> UI
> > >> > would just be responsible for translating the request into the new
> > >> state.
> > >> >
> > >> > Tomorrow I'll try to put together another swagger doc with some of
> the
> > >> > suggested options.
> > >> >
> > >> >
> > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> > >> > <yufeldman@yahoo.com.invalid
> > >> > > wrote:
> > >> >
> > >> > > I think we are at the point to list all the options we want "flex"
> > API
> > >> to
> > >> > > support.
> > >> > > 1. Do we continue supporting flexup/down or just "flex" with
> > >> additional
> > >> > > "preposition" like up/down:https://hostname:port/flex/up(down)
> > >> > > 2. I think we should switch to POST and may be maintain PUT for
> > legacy
> > >> > (if
> > >> > > even needed to keep it). We are not DB after all and not storing
> any
> > >> > > retrievable info here :)
> > >> > > 3. We need to add status (GET) to see the status - though I think
> we
> > >> have
> > >> > > one
> > >> > > 4. Define JSON payload to support different cases    a. providing
> > >> > > different profiles together: [{profile:"big",
> > >> > > instances:2},{profile:"medium",instances:6}]    b. provide what
> > state
> > >> we
> > >> > > want Myriad to be in: "I want 10 medium instances" and then Myriad
> > >> will
> > >> > do
> > >> > > whatever isnecessary to transition to that state,
> > >> > adding/removing/resizing
> > >> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
> > >> preferred
> > >> > > hosts, delays, others
> > >> > > 5. How all this fits into FineGrain Scaling? With it we would do
> > >> > automatic
> > >> > > flex up/down. And the less knobs admin will have to turn the
> easier
> > it
> > >> is
> > >> > > for admin and the end users.
> > >> > >
> > >> > >       From: Adam Bordelon <ad...@mesosphere.io>
> > >> > >  To: dev@myriad.incubator.apache.org
> > >> > >  Sent: Sunday, June 14, 2015 2:54 PM
> > >> > >  Subject: Re: Flex API
> > >> > >
> > >> > > (In addition,) I'd also like to see a more declarative interface.
> > >> Instead
> > >> > > of "add two more instances", the user(s) could just specify the
> > >> desired
> > >> > > state of "I want 10 medium instances" and then Myriad will do
> > whatever
> > >> is
> > >> > > necessary to transition to that state, adding/removing/resizing
> NMs
> > as
> > >> > > necessary.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> > >> > > wochandarena@maprtech.com
> > >> > > > wrote:
> > >> > >
> > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com>
> > >> wrote:
> > >> > > >
> > >> > > > > What verb to use when outside of database land can be argued.
> I
> > >> would
> > >> > > > vote
> > >> > > > > for POST over PUT just because I tend to default to POST. PUT
> > was
> > >> > there
> > >> > > > > when I showed up, so I left it.
> > >> > > >
> > >> > > >
> > >> > > > Last time I agonized about PUT vs POST the most logical
> > distinction
> > >> I
> > >> > > found
> > >> > > > was that PUT should be used for idempotent operations, while
> POST
> > >> for
> > >> > > > non-idempotent (like we have here with flex-up, since
> instance-ids
> > >> are
> > >> > > > generated).
> > >> > > >
> > >> > > > Since the api doesn't wait until the
> > >> > > > > instances are created to return, we can't really return the
> > >> instance
> > >> > > IDs
> > >> > > > we
> > >> > > > > created.
> > >> > > > >
> > >> > > >
> > >> > > > That seems OK to me.
> > >> > > >
> > >> > > >
> > >> > > > > The GET would just return some status?
> > >> > > > >
> > >> > > >
> > >> > > > Yeah, I was thinking that this would be needed for a future GUI
> > >> where
> > >> > we
> > >> > > > list all instances with parameters and status for each (profile,
> > >> > current
> > >> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to
> > each
> > >> so
> > >> > > > users can multi-select and hit 'delete' to wipe them away (like
> > >> > flex-down
> > >> > > > does now).
> > >> > > >
> > >> > > > The PATCH is interesting
> > >> > > > >
> > >> > > >
> > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you
> > >> always
> > >> > > have
> > >> > > > to rewrite the complete object when making changes.  PATCH
> allows
> > >> more
> > >> > > > flexible modifications.
> > >> > > >
> > >> > > > The DELETE makes sense to me.
> > >> > > > >
> > >> > > > > Your use of instances vs instance is interesting. Perhaps we
> > want
> > >> to
> > >> > > > > support POST [{profile:"big", instances:2},{profile:"medium",
> > >> > > > instances:6}]
> > >> > > > >
> > >> > > >
> > >> > > > Yeah, that'd be cool!
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> > >> > > > > wochandarena@maprtech.com
> > >> > > > > > wrote:
> > >> > > > >
> > >> > > > > > Any reason we need to make it action-based
> (flex-up/flex-down)
> > >> > rather
> > >> > > > > than
> > >> > > > > > instance outcome-based?  The way i've seen similar things
> > >> modeled
> > >> > in
> > >> > > > > other
> > >> > > > > > REST APIs is similar to below.  This may also fit better
> into
> > >> REST
> > >> > > > client
> > >> > > > > > frameworks.
> > >> > > > > >
> > >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" :
> > 2,
> > >> > > > "delay" :
> > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
> > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile"
> :
> > >> > "big"}
> > >> > > > > > Delete - DELETE /instances/<instanceid>
> > >> > > > > >
> > >> > > > > > Thoughts?
> > >> > > > > >
> > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <
> klucar@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > > That is always an issue with parameters in the URL, and
> > >> there's
> > >> > > > nothing
> > >> > > > > > > worse than parameters in the URL and having to put JSON in
> > the
> > >> > body
> > >> > > > > > because
> > >> > > > > > > we want to extend it later. Currently flexup takes a JSON
> > blob
> > >> in
> > >> > > the
> > >> > > > > > body
> > >> > > > > > > which we could add more to like you showed. I guess the
> > >> question
> > >> > is
> > >> > > > if
> > >> > > > > we
> > >> > > > > > > want to support a quick and dirty flexup api via URL
> > >> parameters.
> > >> > I
> > >> > > > > > wouldn't
> > >> > > > > > > go any further than these parameters, but all this would
> do
> > at
> > >> > this
> > >> > > > > point
> > >> > > > > > > is make testing with a curl command easier. Getting rid of
> > it
> > >> for
> > >> > > > > flexup
> > >> > > > > > is
> > >> > > > > > > fine with me.
> > >> > > > > > >
> > >> > > > > > > I do like the flexdown/instance/{instance-id} though. We
> > >> should
> > >> > > > > probably
> > >> > > > > > do
> > >> > > > > > > one with flexdown/instance with a json array of
> instance-ids
> > >> too
> > >> > > > > though.
> > >> > > > > > >
> > >> > > > > > > Also with the parameter verification patch I just
> submitted,
> > >> if
> > >> > you
> > >> > > > ask
> > >> > > > > > to
> > >> > > > > > > flexdown 10 instances and you only have 5 running it will
> > just
> > >> > log
> > >> > > a
> > >> > > > > > > warning and kill all 5. Not sure if that is the desired
> > >> behavior.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> > >> > > > > smarella@maprtech.com>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > I think rather than having an API defined as
> > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to
> > >> define
> > >> > > it
> > >> > > > > just
> > >> > > > > > > as
> > >> > > > > > > > "/cluster/flexup" and have parameters such as "profile:
> > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The
> > reason
> > >> is,
> > >> > > if
> > >> > > > > the
> > >> > > > > > > > params are added into the API endpoint, it becomes less
> > >> > flexible
> > >> > > to
> > >> > > > > > > evolve.
> > >> > > > > > > > For e.g. /cluster/flexup currently can take just
> "profile"
> > >> and
> > >> > > > > > > "instances"
> > >> > > > > > > > in it's payload, but in the future can optionally take
> > >> > parameters
> > >> > > > > such
> > >> > > > > > as
> > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > > Santosh
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
> > >> klucar@gmail.com>
> > >> > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I thought that might happen. I created a gist here:
> > >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> > >> > > > > > > > >
> > >> > > > > > > > > And just in case....
> > >> > > > > > > > >
> > >> > > > > > > > > swagger: '2.0'
> > >> > > > > > > > > info:
> > >> > > > > > > > >  version: 0.0.1
> > >> > > > > > > > >  title: Myriad API
> > >> > > > > > > > >  description: |
> > >> > > > > > > > >    Myriad API description
> > >> > > > > > > > > basePath: /api
> > >> > > > > > > > > schemes:
> > >> > > > > > > > >  - http
> > >> > > > > > > > > consumes:
> > >> > > > > > > > >  - application/json
> > >> > > > > > > > >  - text/plain
> > >> > > > > > > > > produces:
> > >> > > > > > > > >  - application/json
> > >> > > > > > > > >  - text/plain
> > >> > > > > > > > > paths:
> > >> > > > > > > > >  /cluster/flexup:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: instance
> > >> > > > > > > > >          in: body
> > >> > > > > > > > >          description: Instance profile and quantity to
> > >> flex
> > >> > up
> > >> > > > > > > > >          schema:
> > >> > > > > > > > >            $ref: '#/definitions/FlexUp'
> > >> > > > > > > > >          required: true
> > >> > > > > > > > >      description: Original Flexup API
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: profile-name
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: string
> > >> > > > > > > > >          description: Instance profile name
> > >> > > > > > > > >      description: Flexup a single instance of a
> > specified
> > >> > > > profile
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: profile-name
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: string
> > >> > > > > > > > >          description: Instance profile name
> > >> > > > > > > > >        - name: instances
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: integer
> > >> > > > > > > > >          description: Number of instances
> > >> > > > > > > > >      description: Flexup a multiple instances of a
> > >> specified
> > >> > > > > profile
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexdown:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: instance
> > >> > > > > > > > >          in: body
> > >> > > > > > > > >          description: Number of instances to flex down
> > >> > > > > > > > >          schema:
> > >> > > > > > > > >            $ref: '#/definitions/FlexDown'
> > >> > > > > > > > >          required: true
> > >> > > > > > > > >      description: Original Flexdown API
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: profile-name
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: string
> > >> > > > > > > > >          description: Instance profile name
> > >> > > > > > > > >      description: Flexup a single instance of a
> > specified
> > >> > > > profile
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: profile-name
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: string
> > >> > > > > > > > >          description: Instance profile name
> > >> > > > > > > > >        - name: instances
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: integer
> > >> > > > > > > > >          description: Number of instances
> > >> > > > > > > > >      description: Flexdown a multiple instances of a
> > >> > specified
> > >> > > > > > profile
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> > >> > > > > > > > >    put:
> > >> > > > > > > > >      parameters:
> > >> > > > > > > > >        - name: instance-id
> > >> > > > > > > > >          in: path
> > >> > > > > > > > >          type: string
> > >> > > > > > > > >          description: Instance profile name
> > >> > > > > > > > >      description: Flexup a single instance of a
> > specified
> > >> > > > profile
> > >> > > > > > > > >      responses:
> > >> > > > > > > > >        200:
> > >> > > > > > > > >          description: OK
> > >> > > > > > > > > definitions:
> > >> > > > > > > > >  FlexUp:
> > >> > > > > > > > >    properties:
> > >> > > > > > > > >      profile:
> > >> > > > > > > > >        type: string
> > >> > > > > > > > >      instances:
> > >> > > > > > > > >        type: integer
> > >> > > > > > > > >        format: int32
> > >> > > > > > > > >  FlexDown:
> > >> > > > > > > > >    properties:
> > >> > > > > > > > >      instances:
> > >> > > > > > > > >        type: integer
> > >> > > > > > > > >        format: int32
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> > >> > > > > > > smarella@maprtech.com>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi Jim,
> > >> > > > > > > > > >
> > >> > > > > > > > > >  Did you attach a file? I think the apache mailing
> > list
> > >> > > > > swallowed
> > >> > > > > > > the
> > >> > > > > > > > > > attachment. Can you please send us the document
> > contents
> > >> in
> > >> > > > plain
> > >> > > > > > > text?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Santosh
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> > >> > > klucar@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > This is in regards to
> > >> > > > > https://github.com/mesos/myriad/issues/89
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I created a swagger document (attached) of the
> > current
> > >> > flex
> > >> > > > > APIs
> > >> > > > > > > and
> > >> > > > > > > > > what
> > >> > > > > > > > > > > I propose for the new flex apis. If people like
> > >> swagger,
> > >> > I
> > >> > > > can
> > >> > > > > > > > document
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > rest of the API, including the correct response
> > codes.
> > >> > > Either
> > >> > > > > > way I
> > >> > > > > > > > > think
> > >> > > > > > > > > > > we should discuss what the API should be going
> > >> forward.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Paste the file contents into
> > >> http://editor.swagger.io/
> > >> > and
> > >> > > > > > you'll
> > >> > > > > > > > get
> > >> > > > > > > > > > > nice HTML to browse.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Flex API

Posted by Adam Bordelon <ad...@mesosphere.io>.
+1, but EQUALS might be an easier starting point than LIKE. Your choice.
What about flexing down specific hosts? Is that out of scope for now?

On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <sm...@maprtech.com>
wrote:

> Ping! Checking if there is anymore feedback on the APIs for flexup/down.
>
> Here is a textual summary of what's being proposed:
> 1. Add a new “constraints” parameter in both flexup and flexdown APIs (just
> like how Marathon does).
> Initially, start with support for LIKE operator in "constraints" that
> operates on “hostname” and “mesos slave attribute
> <http://mesos.apache.org/documentation/attributes-resources/>” fields
> using
> regex.
> 2. Add a new “profile” parameter in the flexdown API.
>
> #1 would give admins a better control over where the NMs should be launched
> or which specific NMs should be shut down.
> #2 helps with shutting down multiple NMs of same "profile" in one shot.
>
> If the above sounds reasonable to start with, I'd like to start working on
> the code changes.
>
> Thanks,
> Santosh
>
> On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <sm...@maprtech.com>
> wrote:
>
> > > One additional thing I'd like is the ability to flexup a nmnode on all
> > > agents with a certain attribute as opposed to a fixed number.
> >
> > Certainly! Marathon places tasks on agents sharing a common attribute.
> > Myriad could/should do the same for placing NMs. If each agent running
> > HDFS data node shares an attribute such as "dfs", then NMs could always
> > be launched on the "dfs" nodes. This improves data locality for YARN jobs
> > dramatically.
> >
> > Santosh
> >
> > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <db...@gmail.com>
> > wrote:
> >
> >> One additional thing I'd like is the ability to flexup a nmnode on all
> >> agents with a certain attribute as opposed to a fixed number.
> >>
> >
> >
> >>
> >> Also, when I get back from vacation I plan on scoping mesos-1739
> (dynamic
> >> attributes) which would allow for tighter integration with the hdfs
> >> framework. An alternative would be to get hostnames from the namenode,
> >> though not as seemless.
> >> (Reviving this thread)
> >>
> >> We've discussed several great points in the thread (PUT vs POST, need
> for
> >> GET, JSON payload vs parameters in URL, declarative interface etc).
> >> Just to get us going, I think we should focus on a couple of things that
> >> will be useful for Myriad users, while leaving them flexible enough to
> be
> >> evolved in the future.
> >>
> >> What I heard from several folks (some of it brought up again at
> MesosCon)
> >> about the flex up/down APIs is this:
> >> - flexup doesn't support launching NMs on specific set of hosts. This is
> >> especially needed to launch NMs on same set of nodes that have HDFS
> >> DataNode running.
> >> - flexdown lacks an option to shut down NMs with a specific profile.
> >> Today,
> >> we bring down ANY arbitrary NM.
> >> - flexdown lacks an option to shutdown NMs running on specific hosts.
> >>
> >> I captured my thoughts in a document here:
> >>
> >>
> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
> >>
> >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
> >> finalize the APIs. Just let me know.
> >>
> >> Thanks,
> >> Santosh
> >>
> >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:
> >>
> >> > Seems like POST is a winner with people.
> >> >
> >> > Another thing to consider is how we want the REST interface to be vs
> >> what
> >> > we want the UI to do. The UI could support flexup/flexdown like it is
> >> while
> >> > the REST interface is just a declarative state like Adam suggested.
> The
> >> UI
> >> > would just be responsible for translating the request into the new
> >> state.
> >> >
> >> > Tomorrow I'll try to put together another swagger doc with some of the
> >> > suggested options.
> >> >
> >> >
> >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> >> > <yufeldman@yahoo.com.invalid
> >> > > wrote:
> >> >
> >> > > I think we are at the point to list all the options we want "flex"
> API
> >> to
> >> > > support.
> >> > > 1. Do we continue supporting flexup/down or just "flex" with
> >> additional
> >> > > "preposition" like up/down:https://hostname:port/flex/up(down)
> >> > > 2. I think we should switch to POST and may be maintain PUT for
> legacy
> >> > (if
> >> > > even needed to keep it). We are not DB after all and not storing any
> >> > > retrievable info here :)
> >> > > 3. We need to add status (GET) to see the status - though I think we
> >> have
> >> > > one
> >> > > 4. Define JSON payload to support different cases    a. providing
> >> > > different profiles together: [{profile:"big",
> >> > > instances:2},{profile:"medium",instances:6}]    b. provide what
> state
> >> we
> >> > > want Myriad to be in: "I want 10 medium instances" and then Myriad
> >> will
> >> > do
> >> > > whatever isnecessary to transition to that state,
> >> > adding/removing/resizing
> >> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
> >> preferred
> >> > > hosts, delays, others
> >> > > 5. How all this fits into FineGrain Scaling? With it we would do
> >> > automatic
> >> > > flex up/down. And the less knobs admin will have to turn the easier
> it
> >> is
> >> > > for admin and the end users.
> >> > >
> >> > >       From: Adam Bordelon <ad...@mesosphere.io>
> >> > >  To: dev@myriad.incubator.apache.org
> >> > >  Sent: Sunday, June 14, 2015 2:54 PM
> >> > >  Subject: Re: Flex API
> >> > >
> >> > > (In addition,) I'd also like to see a more declarative interface.
> >> Instead
> >> > > of "add two more instances", the user(s) could just specify the
> >> desired
> >> > > state of "I want 10 medium instances" and then Myriad will do
> whatever
> >> is
> >> > > necessary to transition to that state, adding/removing/resizing NMs
> as
> >> > > necessary.
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> >> > > wochandarena@maprtech.com
> >> > > > wrote:
> >> > >
> >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com>
> >> wrote:
> >> > > >
> >> > > > > What verb to use when outside of database land can be argued. I
> >> would
> >> > > > vote
> >> > > > > for POST over PUT just because I tend to default to POST. PUT
> was
> >> > there
> >> > > > > when I showed up, so I left it.
> >> > > >
> >> > > >
> >> > > > Last time I agonized about PUT vs POST the most logical
> distinction
> >> I
> >> > > found
> >> > > > was that PUT should be used for idempotent operations, while POST
> >> for
> >> > > > non-idempotent (like we have here with flex-up, since instance-ids
> >> are
> >> > > > generated).
> >> > > >
> >> > > > Since the api doesn't wait until the
> >> > > > > instances are created to return, we can't really return the
> >> instance
> >> > > IDs
> >> > > > we
> >> > > > > created.
> >> > > > >
> >> > > >
> >> > > > That seems OK to me.
> >> > > >
> >> > > >
> >> > > > > The GET would just return some status?
> >> > > > >
> >> > > >
> >> > > > Yeah, I was thinking that this would be needed for a future GUI
> >> where
> >> > we
> >> > > > list all instances with parameters and status for each (profile,
> >> > current
> >> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to
> each
> >> so
> >> > > > users can multi-select and hit 'delete' to wipe them away (like
> >> > flex-down
> >> > > > does now).
> >> > > >
> >> > > > The PATCH is interesting
> >> > > > >
> >> > > >
> >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you
> >> always
> >> > > have
> >> > > > to rewrite the complete object when making changes.  PATCH allows
> >> more
> >> > > > flexible modifications.
> >> > > >
> >> > > > The DELETE makes sense to me.
> >> > > > >
> >> > > > > Your use of instances vs instance is interesting. Perhaps we
> want
> >> to
> >> > > > > support POST [{profile:"big", instances:2},{profile:"medium",
> >> > > > instances:6}]
> >> > > > >
> >> > > >
> >> > > > Yeah, that'd be cool!
> >> > > >
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> >> > > > > wochandarena@maprtech.com
> >> > > > > > wrote:
> >> > > > >
> >> > > > > > Any reason we need to make it action-based (flex-up/flex-down)
> >> > rather
> >> > > > > than
> >> > > > > > instance outcome-based?  The way i've seen similar things
> >> modeled
> >> > in
> >> > > > > other
> >> > > > > > REST APIs is similar to below.  This may also fit better into
> >> REST
> >> > > > client
> >> > > > > > frameworks.
> >> > > > > >
> >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" :
> 2,
> >> > > > "delay" :
> >> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
> >> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
> >> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" :
> >> > "big"}
> >> > > > > > Delete - DELETE /instances/<instanceid>
> >> > > > > >
> >> > > > > > Thoughts?
> >> > > > > >
> >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <klucar@gmail.com
> >
> >> > > wrote:
> >> > > > > >
> >> > > > > > > That is always an issue with parameters in the URL, and
> >> there's
> >> > > > nothing
> >> > > > > > > worse than parameters in the URL and having to put JSON in
> the
> >> > body
> >> > > > > > because
> >> > > > > > > we want to extend it later. Currently flexup takes a JSON
> blob
> >> in
> >> > > the
> >> > > > > > body
> >> > > > > > > which we could add more to like you showed. I guess the
> >> question
> >> > is
> >> > > > if
> >> > > > > we
> >> > > > > > > want to support a quick and dirty flexup api via URL
> >> parameters.
> >> > I
> >> > > > > > wouldn't
> >> > > > > > > go any further than these parameters, but all this would do
> at
> >> > this
> >> > > > > point
> >> > > > > > > is make testing with a curl command easier. Getting rid of
> it
> >> for
> >> > > > > flexup
> >> > > > > > is
> >> > > > > > > fine with me.
> >> > > > > > >
> >> > > > > > > I do like the flexdown/instance/{instance-id} though. We
> >> should
> >> > > > > probably
> >> > > > > > do
> >> > > > > > > one with flexdown/instance with a json array of instance-ids
> >> too
> >> > > > > though.
> >> > > > > > >
> >> > > > > > > Also with the parameter verification patch I just submitted,
> >> if
> >> > you
> >> > > > ask
> >> > > > > > to
> >> > > > > > > flexdown 10 instances and you only have 5 running it will
> just
> >> > log
> >> > > a
> >> > > > > > > warning and kill all 5. Not sure if that is the desired
> >> behavior.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> >> > > > > smarella@maprtech.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > I think rather than having an API defined as
> >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to
> >> define
> >> > > it
> >> > > > > just
> >> > > > > > > as
> >> > > > > > > > "/cluster/flexup" and have parameters such as "profile:
> >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The
> reason
> >> is,
> >> > > if
> >> > > > > the
> >> > > > > > > > params are added into the API endpoint, it becomes less
> >> > flexible
> >> > > to
> >> > > > > > > evolve.
> >> > > > > > > > For e.g. /cluster/flexup currently can take just "profile"
> >> and
> >> > > > > > > "instances"
> >> > > > > > > > in it's payload, but in the future can optionally take
> >> > parameters
> >> > > > > such
> >> > > > > > as
> >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> >> > > > > > > >
> >> > > > > > > > Thoughts?
> >> > > > > > > >
> >> > > > > > > > Santosh
> >> > > > > > > >
> >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
> >> klucar@gmail.com>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > I thought that might happen. I created a gist here:
> >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> >> > > > > > > > >
> >> > > > > > > > > And just in case....
> >> > > > > > > > >
> >> > > > > > > > > swagger: '2.0'
> >> > > > > > > > > info:
> >> > > > > > > > >  version: 0.0.1
> >> > > > > > > > >  title: Myriad API
> >> > > > > > > > >  description: |
> >> > > > > > > > >    Myriad API description
> >> > > > > > > > > basePath: /api
> >> > > > > > > > > schemes:
> >> > > > > > > > >  - http
> >> > > > > > > > > consumes:
> >> > > > > > > > >  - application/json
> >> > > > > > > > >  - text/plain
> >> > > > > > > > > produces:
> >> > > > > > > > >  - application/json
> >> > > > > > > > >  - text/plain
> >> > > > > > > > > paths:
> >> > > > > > > > >  /cluster/flexup:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: instance
> >> > > > > > > > >          in: body
> >> > > > > > > > >          description: Instance profile and quantity to
> >> flex
> >> > up
> >> > > > > > > > >          schema:
> >> > > > > > > > >            $ref: '#/definitions/FlexUp'
> >> > > > > > > > >          required: true
> >> > > > > > > > >      description: Original Flexup API
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: profile-name
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: string
> >> > > > > > > > >          description: Instance profile name
> >> > > > > > > > >      description: Flexup a single instance of a
> specified
> >> > > > profile
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: profile-name
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: string
> >> > > > > > > > >          description: Instance profile name
> >> > > > > > > > >        - name: instances
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: integer
> >> > > > > > > > >          description: Number of instances
> >> > > > > > > > >      description: Flexup a multiple instances of a
> >> specified
> >> > > > > profile
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexdown:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: instance
> >> > > > > > > > >          in: body
> >> > > > > > > > >          description: Number of instances to flex down
> >> > > > > > > > >          schema:
> >> > > > > > > > >            $ref: '#/definitions/FlexDown'
> >> > > > > > > > >          required: true
> >> > > > > > > > >      description: Original Flexdown API
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: profile-name
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: string
> >> > > > > > > > >          description: Instance profile name
> >> > > > > > > > >      description: Flexup a single instance of a
> specified
> >> > > > profile
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: profile-name
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: string
> >> > > > > > > > >          description: Instance profile name
> >> > > > > > > > >        - name: instances
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: integer
> >> > > > > > > > >          description: Number of instances
> >> > > > > > > > >      description: Flexdown a multiple instances of a
> >> > specified
> >> > > > > > profile
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> >> > > > > > > > >    put:
> >> > > > > > > > >      parameters:
> >> > > > > > > > >        - name: instance-id
> >> > > > > > > > >          in: path
> >> > > > > > > > >          type: string
> >> > > > > > > > >          description: Instance profile name
> >> > > > > > > > >      description: Flexup a single instance of a
> specified
> >> > > > profile
> >> > > > > > > > >      responses:
> >> > > > > > > > >        200:
> >> > > > > > > > >          description: OK
> >> > > > > > > > > definitions:
> >> > > > > > > > >  FlexUp:
> >> > > > > > > > >    properties:
> >> > > > > > > > >      profile:
> >> > > > > > > > >        type: string
> >> > > > > > > > >      instances:
> >> > > > > > > > >        type: integer
> >> > > > > > > > >        format: int32
> >> > > > > > > > >  FlexDown:
> >> > > > > > > > >    properties:
> >> > > > > > > > >      instances:
> >> > > > > > > > >        type: integer
> >> > > > > > > > >        format: int32
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> >> > > > > > > smarella@maprtech.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Jim,
> >> > > > > > > > > >
> >> > > > > > > > > >  Did you attach a file? I think the apache mailing
> list
> >> > > > > swallowed
> >> > > > > > > the
> >> > > > > > > > > > attachment. Can you please send us the document
> contents
> >> in
> >> > > > plain
> >> > > > > > > text?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Santosh
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> >> > > klucar@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > This is in regards to
> >> > > > > https://github.com/mesos/myriad/issues/89
> >> > > > > > > > > > >
> >> > > > > > > > > > > I created a swagger document (attached) of the
> current
> >> > flex
> >> > > > > APIs
> >> > > > > > > and
> >> > > > > > > > > what
> >> > > > > > > > > > > I propose for the new flex apis. If people like
> >> swagger,
> >> > I
> >> > > > can
> >> > > > > > > > document
> >> > > > > > > > > > the
> >> > > > > > > > > > > rest of the API, including the correct response
> codes.
> >> > > Either
> >> > > > > > way I
> >> > > > > > > > > think
> >> > > > > > > > > > > we should discuss what the API should be going
> >> forward.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Paste the file contents into
> >> http://editor.swagger.io/
> >> > and
> >> > > > > > you'll
> >> > > > > > > > get
> >> > > > > > > > > > > nice HTML to browse.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Flex API

Posted by Santosh Marella <sm...@maprtech.com>.
Ping! Checking if there is anymore feedback on the APIs for flexup/down.

Here is a textual summary of what's being proposed:
1. Add a new “constraints” parameter in both flexup and flexdown APIs (just
like how Marathon does).
Initially, start with support for LIKE operator in "constraints" that
operates on “hostname” and “mesos slave attribute
<http://mesos.apache.org/documentation/attributes-resources/>” fields using
regex.
2. Add a new “profile” parameter in the flexdown API.

#1 would give admins a better control over where the NMs should be launched
or which specific NMs should be shut down.
#2 helps with shutting down multiple NMs of same "profile" in one shot.

If the above sounds reasonable to start with, I'd like to start working on
the code changes.

Thanks,
Santosh

On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <sm...@maprtech.com>
wrote:

> > One additional thing I'd like is the ability to flexup a nmnode on all
> > agents with a certain attribute as opposed to a fixed number.
>
> Certainly! Marathon places tasks on agents sharing a common attribute.
> Myriad could/should do the same for placing NMs. If each agent running
> HDFS data node shares an attribute such as "dfs", then NMs could always
> be launched on the "dfs" nodes. This improves data locality for YARN jobs
> dramatically.
>
> Santosh
>
> On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <db...@gmail.com>
> wrote:
>
>> One additional thing I'd like is the ability to flexup a nmnode on all
>> agents with a certain attribute as opposed to a fixed number.
>>
>
>
>>
>> Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic
>> attributes) which would allow for tighter integration with the hdfs
>> framework. An alternative would be to get hostnames from the namenode,
>> though not as seemless.
>> (Reviving this thread)
>>
>> We've discussed several great points in the thread (PUT vs POST, need for
>> GET, JSON payload vs parameters in URL, declarative interface etc).
>> Just to get us going, I think we should focus on a couple of things that
>> will be useful for Myriad users, while leaving them flexible enough to be
>> evolved in the future.
>>
>> What I heard from several folks (some of it brought up again at MesosCon)
>> about the flex up/down APIs is this:
>> - flexup doesn't support launching NMs on specific set of hosts. This is
>> especially needed to launch NMs on same set of nodes that have HDFS
>> DataNode running.
>> - flexdown lacks an option to shut down NMs with a specific profile.
>> Today,
>> we bring down ANY arbitrary NM.
>> - flexdown lacks an option to shutdown NMs running on specific hosts.
>>
>> I captured my thoughts in a document here:
>>
>> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
>>
>> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
>> finalize the APIs. Just let me know.
>>
>> Thanks,
>> Santosh
>>
>> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:
>>
>> > Seems like POST is a winner with people.
>> >
>> > Another thing to consider is how we want the REST interface to be vs
>> what
>> > we want the UI to do. The UI could support flexup/flexdown like it is
>> while
>> > the REST interface is just a declarative state like Adam suggested. The
>> UI
>> > would just be responsible for translating the request into the new
>> state.
>> >
>> > Tomorrow I'll try to put together another swagger doc with some of the
>> > suggested options.
>> >
>> >
>> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
>> > <yufeldman@yahoo.com.invalid
>> > > wrote:
>> >
>> > > I think we are at the point to list all the options we want "flex" API
>> to
>> > > support.
>> > > 1. Do we continue supporting flexup/down or just "flex" with
>> additional
>> > > "preposition" like up/down:https://hostname:port/flex/up(down)
>> > > 2. I think we should switch to POST and may be maintain PUT for legacy
>> > (if
>> > > even needed to keep it). We are not DB after all and not storing any
>> > > retrievable info here :)
>> > > 3. We need to add status (GET) to see the status - though I think we
>> have
>> > > one
>> > > 4. Define JSON payload to support different cases    a. providing
>> > > different profiles together: [{profile:"big",
>> > > instances:2},{profile:"medium",instances:6}]    b. provide what state
>> we
>> > > want Myriad to be in: "I want 10 medium instances" and then Myriad
>> will
>> > do
>> > > whatever isnecessary to transition to that state,
>> > adding/removing/resizing
>> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
>> preferred
>> > > hosts, delays, others
>> > > 5. How all this fits into FineGrain Scaling? With it we would do
>> > automatic
>> > > flex up/down. And the less knobs admin will have to turn the easier it
>> is
>> > > for admin and the end users.
>> > >
>> > >       From: Adam Bordelon <ad...@mesosphere.io>
>> > >  To: dev@myriad.incubator.apache.org
>> > >  Sent: Sunday, June 14, 2015 2:54 PM
>> > >  Subject: Re: Flex API
>> > >
>> > > (In addition,) I'd also like to see a more declarative interface.
>> Instead
>> > > of "add two more instances", the user(s) could just specify the
>> desired
>> > > state of "I want 10 medium instances" and then Myriad will do whatever
>> is
>> > > necessary to transition to that state, adding/removing/resizing NMs as
>> > > necessary.
>> > >
>> > >
>> > >
>> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
>> > > wochandarena@maprtech.com
>> > > > wrote:
>> > >
>> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com>
>> wrote:
>> > > >
>> > > > > What verb to use when outside of database land can be argued. I
>> would
>> > > > vote
>> > > > > for POST over PUT just because I tend to default to POST. PUT was
>> > there
>> > > > > when I showed up, so I left it.
>> > > >
>> > > >
>> > > > Last time I agonized about PUT vs POST the most logical distinction
>> I
>> > > found
>> > > > was that PUT should be used for idempotent operations, while POST
>> for
>> > > > non-idempotent (like we have here with flex-up, since instance-ids
>> are
>> > > > generated).
>> > > >
>> > > > Since the api doesn't wait until the
>> > > > > instances are created to return, we can't really return the
>> instance
>> > > IDs
>> > > > we
>> > > > > created.
>> > > > >
>> > > >
>> > > > That seems OK to me.
>> > > >
>> > > >
>> > > > > The GET would just return some status?
>> > > > >
>> > > >
>> > > > Yeah, I was thinking that this would be needed for a future GUI
>> where
>> > we
>> > > > list all instances with parameters and status for each (profile,
>> > current
>> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to each
>> so
>> > > > users can multi-select and hit 'delete' to wipe them away (like
>> > flex-down
>> > > > does now).
>> > > >
>> > > > The PATCH is interesting
>> > > > >
>> > > >
>> > > > Yeah, I started to write PUT but to REST geeks PUT implies you
>> always
>> > > have
>> > > > to rewrite the complete object when making changes.  PATCH allows
>> more
>> > > > flexible modifications.
>> > > >
>> > > > The DELETE makes sense to me.
>> > > > >
>> > > > > Your use of instances vs instance is interesting. Perhaps we want
>> to
>> > > > > support POST [{profile:"big", instances:2},{profile:"medium",
>> > > > instances:6}]
>> > > > >
>> > > >
>> > > > Yeah, that'd be cool!
>> > > >
>> > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
>> > > > > wochandarena@maprtech.com
>> > > > > > wrote:
>> > > > >
>> > > > > > Any reason we need to make it action-based (flex-up/flex-down)
>> > rather
>> > > > > than
>> > > > > > instance outcome-based?  The way i've seen similar things
>> modeled
>> > in
>> > > > > other
>> > > > > > REST APIs is similar to below.  This may also fit better into
>> REST
>> > > > client
>> > > > > > frameworks.
>> > > > > >
>> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2,
>> > > > "delay" :
>> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
>> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
>> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" :
>> > "big"}
>> > > > > > Delete - DELETE /instances/<instanceid>
>> > > > > >
>> > > > > > Thoughts?
>> > > > > >
>> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <kl...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > That is always an issue with parameters in the URL, and
>> there's
>> > > > nothing
>> > > > > > > worse than parameters in the URL and having to put JSON in the
>> > body
>> > > > > > because
>> > > > > > > we want to extend it later. Currently flexup takes a JSON blob
>> in
>> > > the
>> > > > > > body
>> > > > > > > which we could add more to like you showed. I guess the
>> question
>> > is
>> > > > if
>> > > > > we
>> > > > > > > want to support a quick and dirty flexup api via URL
>> parameters.
>> > I
>> > > > > > wouldn't
>> > > > > > > go any further than these parameters, but all this would do at
>> > this
>> > > > > point
>> > > > > > > is make testing with a curl command easier. Getting rid of it
>> for
>> > > > > flexup
>> > > > > > is
>> > > > > > > fine with me.
>> > > > > > >
>> > > > > > > I do like the flexdown/instance/{instance-id} though. We
>> should
>> > > > > probably
>> > > > > > do
>> > > > > > > one with flexdown/instance with a json array of instance-ids
>> too
>> > > > > though.
>> > > > > > >
>> > > > > > > Also with the parameter verification patch I just submitted,
>> if
>> > you
>> > > > ask
>> > > > > > to
>> > > > > > > flexdown 10 instances and you only have 5 running it will just
>> > log
>> > > a
>> > > > > > > warning and kill all 5. Not sure if that is the desired
>> behavior.
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
>> > > > > smarella@maprtech.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > I think rather than having an API defined as
>> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to
>> define
>> > > it
>> > > > > just
>> > > > > > > as
>> > > > > > > > "/cluster/flexup" and have parameters such as "profile:
>> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason
>> is,
>> > > if
>> > > > > the
>> > > > > > > > params are added into the API endpoint, it becomes less
>> > flexible
>> > > to
>> > > > > > > evolve.
>> > > > > > > > For e.g. /cluster/flexup currently can take just "profile"
>> and
>> > > > > > > "instances"
>> > > > > > > > in it's payload, but in the future can optionally take
>> > parameters
>> > > > > such
>> > > > > > as
>> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
>> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >
>> > > > > > > > Santosh
>> > > > > > > >
>> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
>> klucar@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I thought that might happen. I created a gist here:
>> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
>> > > > > > > > >
>> > > > > > > > > And just in case....
>> > > > > > > > >
>> > > > > > > > > swagger: '2.0'
>> > > > > > > > > info:
>> > > > > > > > >  version: 0.0.1
>> > > > > > > > >  title: Myriad API
>> > > > > > > > >  description: |
>> > > > > > > > >    Myriad API description
>> > > > > > > > > basePath: /api
>> > > > > > > > > schemes:
>> > > > > > > > >  - http
>> > > > > > > > > consumes:
>> > > > > > > > >  - application/json
>> > > > > > > > >  - text/plain
>> > > > > > > > > produces:
>> > > > > > > > >  - application/json
>> > > > > > > > >  - text/plain
>> > > > > > > > > paths:
>> > > > > > > > >  /cluster/flexup:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: instance
>> > > > > > > > >          in: body
>> > > > > > > > >          description: Instance profile and quantity to
>> flex
>> > up
>> > > > > > > > >          schema:
>> > > > > > > > >            $ref: '#/definitions/FlexUp'
>> > > > > > > > >          required: true
>> > > > > > > > >      description: Original Flexup API
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: profile-name
>> > > > > > > > >          in: path
>> > > > > > > > >          type: string
>> > > > > > > > >          description: Instance profile name
>> > > > > > > > >      description: Flexup a single instance of a specified
>> > > > profile
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: profile-name
>> > > > > > > > >          in: path
>> > > > > > > > >          type: string
>> > > > > > > > >          description: Instance profile name
>> > > > > > > > >        - name: instances
>> > > > > > > > >          in: path
>> > > > > > > > >          type: integer
>> > > > > > > > >          description: Number of instances
>> > > > > > > > >      description: Flexup a multiple instances of a
>> specified
>> > > > > profile
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexdown:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: instance
>> > > > > > > > >          in: body
>> > > > > > > > >          description: Number of instances to flex down
>> > > > > > > > >          schema:
>> > > > > > > > >            $ref: '#/definitions/FlexDown'
>> > > > > > > > >          required: true
>> > > > > > > > >      description: Original Flexdown API
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: profile-name
>> > > > > > > > >          in: path
>> > > > > > > > >          type: string
>> > > > > > > > >          description: Instance profile name
>> > > > > > > > >      description: Flexup a single instance of a specified
>> > > > profile
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: profile-name
>> > > > > > > > >          in: path
>> > > > > > > > >          type: string
>> > > > > > > > >          description: Instance profile name
>> > > > > > > > >        - name: instances
>> > > > > > > > >          in: path
>> > > > > > > > >          type: integer
>> > > > > > > > >          description: Number of instances
>> > > > > > > > >      description: Flexdown a multiple instances of a
>> > specified
>> > > > > > profile
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
>> > > > > > > > >    put:
>> > > > > > > > >      parameters:
>> > > > > > > > >        - name: instance-id
>> > > > > > > > >          in: path
>> > > > > > > > >          type: string
>> > > > > > > > >          description: Instance profile name
>> > > > > > > > >      description: Flexup a single instance of a specified
>> > > > profile
>> > > > > > > > >      responses:
>> > > > > > > > >        200:
>> > > > > > > > >          description: OK
>> > > > > > > > > definitions:
>> > > > > > > > >  FlexUp:
>> > > > > > > > >    properties:
>> > > > > > > > >      profile:
>> > > > > > > > >        type: string
>> > > > > > > > >      instances:
>> > > > > > > > >        type: integer
>> > > > > > > > >        format: int32
>> > > > > > > > >  FlexDown:
>> > > > > > > > >    properties:
>> > > > > > > > >      instances:
>> > > > > > > > >        type: integer
>> > > > > > > > >        format: int32
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
>> > > > > > > smarella@maprtech.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Jim,
>> > > > > > > > > >
>> > > > > > > > > >  Did you attach a file? I think the apache mailing list
>> > > > > swallowed
>> > > > > > > the
>> > > > > > > > > > attachment. Can you please send us the document contents
>> in
>> > > > plain
>> > > > > > > text?
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Santosh
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
>> > > klucar@gmail.com>
>> > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > This is in regards to
>> > > > > https://github.com/mesos/myriad/issues/89
>> > > > > > > > > > >
>> > > > > > > > > > > I created a swagger document (attached) of the current
>> > flex
>> > > > > APIs
>> > > > > > > and
>> > > > > > > > > what
>> > > > > > > > > > > I propose for the new flex apis. If people like
>> swagger,
>> > I
>> > > > can
>> > > > > > > > document
>> > > > > > > > > > the
>> > > > > > > > > > > rest of the API, including the correct response codes.
>> > > Either
>> > > > > > way I
>> > > > > > > > > think
>> > > > > > > > > > > we should discuss what the API should be going
>> forward.
>> > > > > > > > > > >
>> > > > > > > > > > > Paste the file contents into
>> http://editor.swagger.io/
>> > and
>> > > > > > you'll
>> > > > > > > > get
>> > > > > > > > > > > nice HTML to browse.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: Flex API

Posted by Santosh Marella <sm...@maprtech.com>.
> One additional thing I'd like is the ability to flexup a nmnode on all
> agents with a certain attribute as opposed to a fixed number.

Certainly! Marathon places tasks on agents sharing a common attribute.
Myriad could/should do the same for placing NMs. If each agent running
HDFS data node shares an attribute such as "dfs", then NMs could always
be launched on the "dfs" nodes. This improves data locality for YARN jobs
dramatically.

Santosh

On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <db...@gmail.com>
wrote:

> One additional thing I'd like is the ability to flexup a nmnode on all
> agents with a certain attribute as opposed to a fixed number.
>


>
> Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic
> attributes) which would allow for tighter integration with the hdfs
> framework. An alternative would be to get hostnames from the namenode,
> though not as seemless.
> (Reviving this thread)
>
> We've discussed several great points in the thread (PUT vs POST, need for
> GET, JSON payload vs parameters in URL, declarative interface etc).
> Just to get us going, I think we should focus on a couple of things that
> will be useful for Myriad users, while leaving them flexible enough to be
> evolved in the future.
>
> What I heard from several folks (some of it brought up again at MesosCon)
> about the flex up/down APIs is this:
> - flexup doesn't support launching NMs on specific set of hosts. This is
> especially needed to launch NMs on same set of nodes that have HDFS
> DataNode running.
> - flexdown lacks an option to shut down NMs with a specific profile. Today,
> we bring down ANY arbitrary NM.
> - flexdown lacks an option to shutdown NMs running on specific hosts.
>
> I captured my thoughts in a document here:
>
> https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t
>
> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
> finalize the APIs. Just let me know.
>
> Thanks,
> Santosh
>
> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:
>
> > Seems like POST is a winner with people.
> >
> > Another thing to consider is how we want the REST interface to be vs what
> > we want the UI to do. The UI could support flexup/flexdown like it is
> while
> > the REST interface is just a declarative state like Adam suggested. The
> UI
> > would just be responsible for translating the request into the new state.
> >
> > Tomorrow I'll try to put together another swagger doc with some of the
> > suggested options.
> >
> >
> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> > <yufeldman@yahoo.com.invalid
> > > wrote:
> >
> > > I think we are at the point to list all the options we want "flex" API
> to
> > > support.
> > > 1. Do we continue supporting flexup/down or just "flex" with additional
> > > "preposition" like up/down:https://hostname:port/flex/up(down)
> > > 2. I think we should switch to POST and may be maintain PUT for legacy
> > (if
> > > even needed to keep it). We are not DB after all and not storing any
> > > retrievable info here :)
> > > 3. We need to add status (GET) to see the status - though I think we
> have
> > > one
> > > 4. Define JSON payload to support different cases    a. providing
> > > different profiles together: [{profile:"big",
> > > instances:2},{profile:"medium",instances:6}]    b. provide what state
> we
> > > want Myriad to be in: "I want 10 medium instances" and then Myriad will
> > do
> > > whatever isnecessary to transition to that state,
> > adding/removing/resizing
> > > NMs"    c. flex/down particular instance IDs    d. flex up/down
> preferred
> > > hosts, delays, others
> > > 5. How all this fits into FineGrain Scaling? With it we would do
> > automatic
> > > flex up/down. And the less knobs admin will have to turn the easier it
> is
> > > for admin and the end users.
> > >
> > >       From: Adam Bordelon <ad...@mesosphere.io>
> > >  To: dev@myriad.incubator.apache.org
> > >  Sent: Sunday, June 14, 2015 2:54 PM
> > >  Subject: Re: Flex API
> > >
> > > (In addition,) I'd also like to see a more declarative interface.
> Instead
> > > of "add two more instances", the user(s) could just specify the desired
> > > state of "I want 10 medium instances" and then Myriad will do whatever
> is
> > > necessary to transition to that state, adding/removing/resizing NMs as
> > > necessary.
> > >
> > >
> > >
> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> > > wochandarena@maprtech.com
> > > > wrote:
> > >
> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com>
> wrote:
> > > >
> > > > > What verb to use when outside of database land can be argued. I
> would
> > > > vote
> > > > > for POST over PUT just because I tend to default to POST. PUT was
> > there
> > > > > when I showed up, so I left it.
> > > >
> > > >
> > > > Last time I agonized about PUT vs POST the most logical distinction I
> > > found
> > > > was that PUT should be used for idempotent operations, while POST for
> > > > non-idempotent (like we have here with flex-up, since instance-ids
> are
> > > > generated).
> > > >
> > > > Since the api doesn't wait until the
> > > > > instances are created to return, we can't really return the
> instance
> > > IDs
> > > > we
> > > > > created.
> > > > >
> > > >
> > > > That seems OK to me.
> > > >
> > > >
> > > > > The GET would just return some status?
> > > > >
> > > >
> > > > Yeah, I was thinking that this would be needed for a future GUI where
> > we
> > > > list all instances with parameters and status for each (profile,
> > current
> > > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to each
> so
> > > > users can multi-select and hit 'delete' to wipe them away (like
> > flex-down
> > > > does now).
> > > >
> > > > The PATCH is interesting
> > > > >
> > > >
> > > > Yeah, I started to write PUT but to REST geeks PUT implies you always
> > > have
> > > > to rewrite the complete object when making changes.  PATCH allows
> more
> > > > flexible modifications.
> > > >
> > > > The DELETE makes sense to me.
> > > > >
> > > > > Your use of instances vs instance is interesting. Perhaps we want
> to
> > > > > support POST [{profile:"big", instances:2},{profile:"medium",
> > > > instances:6}]
> > > > >
> > > >
> > > > Yeah, that'd be cool!
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> > > > > wochandarena@maprtech.com
> > > > > > wrote:
> > > > >
> > > > > > Any reason we need to make it action-based (flex-up/flex-down)
> > rather
> > > > > than
> > > > > > instance outcome-based?  The way i've seen similar things modeled
> > in
> > > > > other
> > > > > > REST APIs is similar to below.  This may also fit better into
> REST
> > > > client
> > > > > > frameworks.
> > > > > >
> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2,
> > > > "delay" :
> > > > > > 2, "preferredHosts" : ["host1", "host2"]}
> > > > > > Read - GET /instances/ or GET /instances/<instanceid>
> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" :
> > "big"}
> > > > > > Delete - DELETE /instances/<instanceid>
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <kl...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > That is always an issue with parameters in the URL, and there's
> > > > nothing
> > > > > > > worse than parameters in the URL and having to put JSON in the
> > body
> > > > > > because
> > > > > > > we want to extend it later. Currently flexup takes a JSON blob
> in
> > > the
> > > > > > body
> > > > > > > which we could add more to like you showed. I guess the
> question
> > is
> > > > if
> > > > > we
> > > > > > > want to support a quick and dirty flexup api via URL
> parameters.
> > I
> > > > > > wouldn't
> > > > > > > go any further than these parameters, but all this would do at
> > this
> > > > > point
> > > > > > > is make testing with a curl command easier. Getting rid of it
> for
> > > > > flexup
> > > > > > is
> > > > > > > fine with me.
> > > > > > >
> > > > > > > I do like the flexdown/instance/{instance-id} though. We should
> > > > > probably
> > > > > > do
> > > > > > > one with flexdown/instance with a json array of instance-ids
> too
> > > > > though.
> > > > > > >
> > > > > > > Also with the parameter verification patch I just submitted, if
> > you
> > > > ask
> > > > > > to
> > > > > > > flexdown 10 instances and you only have 5 running it will just
> > log
> > > a
> > > > > > > warning and kill all 5. Not sure if that is the desired
> behavior.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> > > > > smarella@maprtech.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think rather than having an API defined as
> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to
> define
> > > it
> > > > > just
> > > > > > > as
> > > > > > > > "/cluster/flexup" and have parameters such as "profile:
> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason
> is,
> > > if
> > > > > the
> > > > > > > > params are added into the API endpoint, it becomes less
> > flexible
> > > to
> > > > > > > evolve.
> > > > > > > > For e.g. /cluster/flexup currently can take just "profile"
> and
> > > > > > > "instances"
> > > > > > > > in it's payload, but in the future can optionally take
> > parameters
> > > > > such
> > > > > > as
> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Santosh
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <
> klucar@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I thought that might happen. I created a gist here:
> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> > > > > > > > >
> > > > > > > > > And just in case....
> > > > > > > > >
> > > > > > > > > swagger: '2.0'
> > > > > > > > > info:
> > > > > > > > >  version: 0.0.1
> > > > > > > > >  title: Myriad API
> > > > > > > > >  description: |
> > > > > > > > >    Myriad API description
> > > > > > > > > basePath: /api
> > > > > > > > > schemes:
> > > > > > > > >  - http
> > > > > > > > > consumes:
> > > > > > > > >  - application/json
> > > > > > > > >  - text/plain
> > > > > > > > > produces:
> > > > > > > > >  - application/json
> > > > > > > > >  - text/plain
> > > > > > > > > paths:
> > > > > > > > >  /cluster/flexup:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: instance
> > > > > > > > >          in: body
> > > > > > > > >          description: Instance profile and quantity to flex
> > up
> > > > > > > > >          schema:
> > > > > > > > >            $ref: '#/definitions/FlexUp'
> > > > > > > > >          required: true
> > > > > > > > >      description: Original Flexup API
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexup/profile/{profile-name}:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: profile-name
> > > > > > > > >          in: path
> > > > > > > > >          type: string
> > > > > > > > >          description: Instance profile name
> > > > > > > > >      description: Flexup a single instance of a specified
> > > > profile
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: profile-name
> > > > > > > > >          in: path
> > > > > > > > >          type: string
> > > > > > > > >          description: Instance profile name
> > > > > > > > >        - name: instances
> > > > > > > > >          in: path
> > > > > > > > >          type: integer
> > > > > > > > >          description: Number of instances
> > > > > > > > >      description: Flexup a multiple instances of a
> specified
> > > > > profile
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexdown:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: instance
> > > > > > > > >          in: body
> > > > > > > > >          description: Number of instances to flex down
> > > > > > > > >          schema:
> > > > > > > > >            $ref: '#/definitions/FlexDown'
> > > > > > > > >          required: true
> > > > > > > > >      description: Original Flexdown API
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: profile-name
> > > > > > > > >          in: path
> > > > > > > > >          type: string
> > > > > > > > >          description: Instance profile name
> > > > > > > > >      description: Flexup a single instance of a specified
> > > > profile
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: profile-name
> > > > > > > > >          in: path
> > > > > > > > >          type: string
> > > > > > > > >          description: Instance profile name
> > > > > > > > >        - name: instances
> > > > > > > > >          in: path
> > > > > > > > >          type: integer
> > > > > > > > >          description: Number of instances
> > > > > > > > >      description: Flexdown a multiple instances of a
> > specified
> > > > > > profile
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> > > > > > > > >    put:
> > > > > > > > >      parameters:
> > > > > > > > >        - name: instance-id
> > > > > > > > >          in: path
> > > > > > > > >          type: string
> > > > > > > > >          description: Instance profile name
> > > > > > > > >      description: Flexup a single instance of a specified
> > > > profile
> > > > > > > > >      responses:
> > > > > > > > >        200:
> > > > > > > > >          description: OK
> > > > > > > > > definitions:
> > > > > > > > >  FlexUp:
> > > > > > > > >    properties:
> > > > > > > > >      profile:
> > > > > > > > >        type: string
> > > > > > > > >      instances:
> > > > > > > > >        type: integer
> > > > > > > > >        format: int32
> > > > > > > > >  FlexDown:
> > > > > > > > >    properties:
> > > > > > > > >      instances:
> > > > > > > > >        type: integer
> > > > > > > > >        format: int32
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> > > > > > > smarella@maprtech.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jim,
> > > > > > > > > >
> > > > > > > > > >  Did you attach a file? I think the apache mailing list
> > > > > swallowed
> > > > > > > the
> > > > > > > > > > attachment. Can you please send us the document contents
> in
> > > > plain
> > > > > > > text?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Santosh
> > > > > > > > > >
> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> > > klucar@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > This is in regards to
> > > > > https://github.com/mesos/myriad/issues/89
> > > > > > > > > > >
> > > > > > > > > > > I created a swagger document (attached) of the current
> > flex
> > > > > APIs
> > > > > > > and
> > > > > > > > > what
> > > > > > > > > > > I propose for the new flex apis. If people like
> swagger,
> > I
> > > > can
> > > > > > > > document
> > > > > > > > > > the
> > > > > > > > > > > rest of the API, including the correct response codes.
> > > Either
> > > > > > way I
> > > > > > > > > think
> > > > > > > > > > > we should discuss what the API should be going forward.
> > > > > > > > > > >
> > > > > > > > > > > Paste the file contents into http://editor.swagger.io/
> > and
> > > > > > you'll
> > > > > > > > get
> > > > > > > > > > > nice HTML to browse.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>

Re: Flex API

Posted by Darin Johnson <db...@gmail.com>.
One additional thing I'd like is the ability to flexup a nmnode on all
agents with a certain attribute as opposed to a fixed number.

Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic
attributes) which would allow for tighter integration with the hdfs
framework. An alternative would be to get hostnames from the namenode,
though not as seemless.
(Reviving this thread)

We've discussed several great points in the thread (PUT vs POST, need for
GET, JSON payload vs parameters in URL, declarative interface etc).
Just to get us going, I think we should focus on a couple of things that
will be useful for Myriad users, while leaving them flexible enough to be
evolved in the future.

What I heard from several folks (some of it brought up again at MesosCon)
about the flex up/down APIs is this:
- flexup doesn't support launching NMs on specific set of hosts. This is
especially needed to launch NMs on same set of nodes that have HDFS
DataNode running.
- flexdown lacks an option to shut down NMs with a specific profile. Today,
we bring down ANY arbitrary NM.
- flexdown lacks an option to shutdown NMs running on specific hosts.

I captured my thoughts in a document here:
https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t

@Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
finalize the APIs. Just let me know.

Thanks,
Santosh

On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <kl...@gmail.com> wrote:

> Seems like POST is a winner with people.
>
> Another thing to consider is how we want the REST interface to be vs what
> we want the UI to do. The UI could support flexup/flexdown like it is
while
> the REST interface is just a declarative state like Adam suggested. The UI
> would just be responsible for translating the request into the new state.
>
> Tomorrow I'll try to put together another swagger doc with some of the
> suggested options.
>
>
> On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
> <yufeldman@yahoo.com.invalid
> > wrote:
>
> > I think we are at the point to list all the options we want "flex" API
to
> > support.
> > 1. Do we continue supporting flexup/down or just "flex" with additional
> > "preposition" like up/down:https://hostname:port/flex/up(down)
> > 2. I think we should switch to POST and may be maintain PUT for legacy
> (if
> > even needed to keep it). We are not DB after all and not storing any
> > retrievable info here :)
> > 3. We need to add status (GET) to see the status - though I think we
have
> > one
> > 4. Define JSON payload to support different cases    a. providing
> > different profiles together: [{profile:"big",
> > instances:2},{profile:"medium",instances:6}]    b. provide what state we
> > want Myriad to be in: "I want 10 medium instances" and then Myriad will
> do
> > whatever isnecessary to transition to that state,
> adding/removing/resizing
> > NMs"    c. flex/down particular instance IDs    d. flex up/down
preferred
> > hosts, delays, others
> > 5. How all this fits into FineGrain Scaling? With it we would do
> automatic
> > flex up/down. And the less knobs admin will have to turn the easier it
is
> > for admin and the end users.
> >
> >       From: Adam Bordelon <ad...@mesosphere.io>
> >  To: dev@myriad.incubator.apache.org
> >  Sent: Sunday, June 14, 2015 2:54 PM
> >  Subject: Re: Flex API
> >
> > (In addition,) I'd also like to see a more declarative interface.
Instead
> > of "add two more instances", the user(s) could just specify the desired
> > state of "I want 10 medium instances" and then Myriad will do whatever
is
> > necessary to transition to that state, adding/removing/resizing NMs as
> > necessary.
> >
> >
> >
> > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena <
> > wochandarena@maprtech.com
> > > wrote:
> >
> > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <kl...@gmail.com> wrote:
> > >
> > > > What verb to use when outside of database land can be argued. I
would
> > > vote
> > > > for POST over PUT just because I tend to default to POST. PUT was
> there
> > > > when I showed up, so I left it.
> > >
> > >
> > > Last time I agonized about PUT vs POST the most logical distinction I
> > found
> > > was that PUT should be used for idempotent operations, while POST for
> > > non-idempotent (like we have here with flex-up, since instance-ids are
> > > generated).
> > >
> > > Since the api doesn't wait until the
> > > > instances are created to return, we can't really return the instance
> > IDs
> > > we
> > > > created.
> > > >
> > >
> > > That seems OK to me.
> > >
> > >
> > > > The GET would just return some status?
> > > >
> > >
> > > Yeah, I was thinking that this would be needed for a future GUI where
> we
> > > list all instances with parameters and status for each (profile,
> current
> > > cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to each
so
> > > users can multi-select and hit 'delete' to wipe them away (like
> flex-down
> > > does now).
> > >
> > > The PATCH is interesting
> > > >
> > >
> > > Yeah, I started to write PUT but to REST geeks PUT implies you always
> > have
> > > to rewrite the complete object when making changes.  PATCH allows more
> > > flexible modifications.
> > >
> > > The DELETE makes sense to me.
> > > >
> > > > Your use of instances vs instance is interesting. Perhaps we want to
> > > > support POST [{profile:"big", instances:2},{profile:"medium",
> > > instances:6}]
> > > >
> > >
> > > Yeah, that'd be cool!
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena <
> > > > wochandarena@maprtech.com
> > > > > wrote:
> > > >
> > > > > Any reason we need to make it action-based (flex-up/flex-down)
> rather
> > > > than
> > > > > instance outcome-based?  The way i've seen similar things modeled
> in
> > > > other
> > > > > REST APIs is similar to below.  This may also fit better into REST
> > > client
> > > > > frameworks.
> > > > >
> > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2,
> > > "delay" :
> > > > > 2, "preferredHosts" : ["host1", "host2"]}
> > > > > Read - GET /instances/ or GET /instances/<instanceid>
> > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" :
> "big"}
> > > > > Delete - DELETE /instances/<instanceid>
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <kl...@gmail.com>
> > wrote:
> > > > >
> > > > > > That is always an issue with parameters in the URL, and there's
> > > nothing
> > > > > > worse than parameters in the URL and having to put JSON in the
> body
> > > > > because
> > > > > > we want to extend it later. Currently flexup takes a JSON blob
in
> > the
> > > > > body
> > > > > > which we could add more to like you showed. I guess the question
> is
> > > if
> > > > we
> > > > > > want to support a quick and dirty flexup api via URL parameters.
> I
> > > > > wouldn't
> > > > > > go any further than these parameters, but all this would do at
> this
> > > > point
> > > > > > is make testing with a curl command easier. Getting rid of it
for
> > > > flexup
> > > > > is
> > > > > > fine with me.
> > > > > >
> > > > > > I do like the flexdown/instance/{instance-id} though. We should
> > > > probably
> > > > > do
> > > > > > one with flexdown/instance with a json array of instance-ids too
> > > > though.
> > > > > >
> > > > > > Also with the parameter verification patch I just submitted, if
> you
> > > ask
> > > > > to
> > > > > > flexdown 10 instances and you only have 5 running it will just
> log
> > a
> > > > > > warning and kill all 5. Not sure if that is the desired
behavior.
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella <
> > > > smarella@maprtech.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think rather than having an API defined as
> > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to
define
> > it
> > > > just
> > > > > > as
> > > > > > > "/cluster/flexup" and have parameters such as "profile:
> > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason
is,
> > if
> > > > the
> > > > > > > params are added into the API endpoint, it becomes less
> flexible
> > to
> > > > > > evolve.
> > > > > > > For e.g. /cluster/flexup currently can take just "profile" and
> > > > > > "instances"
> > > > > > > in it's payload, but in the future can optionally take
> parameters
> > > > such
> > > > > as
> > > > > > > "preferredHosts: [host1, host2, host9]", "delay:
> > > > > > > <take_some_sweet_time_before_flexing_up>" etc.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Santosh
> > > > > > >
> > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <kl...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > I thought that might happen. I created a gist here:
> > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e
> > > > > > > >
> > > > > > > > And just in case....
> > > > > > > >
> > > > > > > > swagger: '2.0'
> > > > > > > > info:
> > > > > > > >  version: 0.0.1
> > > > > > > >  title: Myriad API
> > > > > > > >  description: |
> > > > > > > >    Myriad API description
> > > > > > > > basePath: /api
> > > > > > > > schemes:
> > > > > > > >  - http
> > > > > > > > consumes:
> > > > > > > >  - application/json
> > > > > > > >  - text/plain
> > > > > > > > produces:
> > > > > > > >  - application/json
> > > > > > > >  - text/plain
> > > > > > > > paths:
> > > > > > > >  /cluster/flexup:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance
> > > > > > > >          in: body
> > > > > > > >          description: Instance profile and quantity to flex
> up
> > > > > > > >          schema:
> > > > > > > >            $ref: '#/definitions/FlexUp'
> > > > > > > >          required: true
> > > > > > > >      description: Original Flexup API
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexup/profile/{profile-name}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexup/profile/{profile-name}/{instances}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >        - name: instances
> > > > > > > >          in: path
> > > > > > > >          type: integer
> > > > > > > >          description: Number of instances
> > > > > > > >      description: Flexup a multiple instances of a specified
> > > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance
> > > > > > > >          in: body
> > > > > > > >          description: Number of instances to flex down
> > > > > > > >          schema:
> > > > > > > >            $ref: '#/definitions/FlexDown'
> > > > > > > >          required: true
> > > > > > > >      description: Original Flexdown API
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/profile/{profile-name}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/profile/{profile-name}/{instances}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: profile-name
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >        - name: instances
> > > > > > > >          in: path
> > > > > > > >          type: integer
> > > > > > > >          description: Number of instances
> > > > > > > >      description: Flexdown a multiple instances of a
> specified
> > > > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > >  /cluster/flexdown/instance/{instance-id}:
> > > > > > > >    put:
> > > > > > > >      parameters:
> > > > > > > >        - name: instance-id
> > > > > > > >          in: path
> > > > > > > >          type: string
> > > > > > > >          description: Instance profile name
> > > > > > > >      description: Flexup a single instance of a specified
> > > profile
> > > > > > > >      responses:
> > > > > > > >        200:
> > > > > > > >          description: OK
> > > > > > > > definitions:
> > > > > > > >  FlexUp:
> > > > > > > >    properties:
> > > > > > > >      profile:
> > > > > > > >        type: string
> > > > > > > >      instances:
> > > > > > > >        type: integer
> > > > > > > >        format: int32
> > > > > > > >  FlexDown:
> > > > > > > >    properties:
> > > > > > > >      instances:
> > > > > > > >        type: integer
> > > > > > > >        format: int32
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella <
> > > > > > smarella@maprtech.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jim,
> > > > > > > > >
> > > > > > > > >  Did you attach a file? I think the apache mailing list
> > > > swallowed
> > > > > > the
> > > > > > > > > attachment. Can you please send us the document contents
in
> > > plain
> > > > > > text?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Santosh
> > > > > > > > >
> > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar <
> > klucar@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This is in regards to
> > > > https://github.com/mesos/myriad/issues/89
> > > > > > > > > >
> > > > > > > > > > I created a swagger document (attached) of the current
> flex
> > > > APIs
> > > > > > and
> > > > > > > > what
> > > > > > > > > > I propose for the new flex apis. If people like swagger,
> I
> > > can
> > > > > > > document
> > > > > > > > > the
> > > > > > > > > > rest of the API, including the correct response codes.
> > Either
> > > > > way I
> > > > > > > > think
> > > > > > > > > > we should discuss what the API should be going forward.
> > > > > > > > > >
> > > > > > > > > > Paste the file contents into http://editor.swagger.io/
> and
> > > > > you'll
> > > > > > > get
> > > > > > > > > > nice HTML to browse.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>