You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Rodric Rabbah <ro...@gmail.com> on 2017/03/05 16:39:00 UTC
factoring out "runtimes" into deployment configuration
I've been thinking about how we can add more action runtimes more easily -
without changing the backend (API, controller, invoker, CLI). One way I'm
leaning toward and started prototyping to get a better understanding of the
feasibility is to strip all runtime types from the backend (and CLI) and
encode the information about what action runtimes the system supports
through configuration parameters.
This will make it possible to decouple deploying the core components from
managing the action runtime images. An example of encoding the runtime
information in a deployment configuration is
https://github.com/openwhisk/openwhisk/pull/1948.
In support of this general direction (even if we settle on a different
approach), I have increasingly flattened the "Exec" type hierarchy in the
backend[1-3]
[1] https://github.com/openwhisk/openwhisk/pull/1938
[2] https://github.com/openwhisk/openwhisk/pull/1942
[3] https://github.com/openwhisk/openwhisk/pull/1911
A sketch of how to use the runtime manifest can be seen in this WIP commit:
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
I can see how using this we can add new versions of Node.js, python:3, and
other handy images by curating a list of these in a separate process.
Feedback, as always, solicited and appreciated.
-r
Re: factoring out "runtimes" into deployment configuration
Posted by Markus Thömmes <ma...@me.com>.
Good one!
Another vector where this will improve the system in general is the ability to run a minimized set of tests for our core components and thus reducing feedback time.
- mt
Von meinem iPhone gesendet
> Am 05.03.2017 um 17:39 schrieb Rodric Rabbah <ro...@gmail.com>:
>
> I've been thinking about how we can add more action runtimes more easily -
> without changing the backend (API, controller, invoker, CLI). One way I'm
> leaning toward and started prototyping to get a better understanding of the
> feasibility is to strip all runtime types from the backend (and CLI) and
> encode the information about what action runtimes the system supports
> through configuration parameters.
>
> This will make it possible to decouple deploying the core components from
> managing the action runtime images. An example of encoding the runtime
> information in a deployment configuration is
> https://github.com/openwhisk/openwhisk/pull/1948.
>
> In support of this general direction (even if we settle on a different
> approach), I have increasingly flattened the "Exec" type hierarchy in the
> backend[1-3]
>
> [1] https://github.com/openwhisk/openwhisk/pull/1938
> [2] https://github.com/openwhisk/openwhisk/pull/1942
> [3] https://github.com/openwhisk/openwhisk/pull/1911
>
> A sketch of how to use the runtime manifest can be seen in this WIP commit:
> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
> I can see how using this we can add new versions of Node.js, python:3, and
> other handy images by curating a list of these in a separate process.
>
> Feedback, as always, solicited and appreciated.
>
> -r
Re: factoring out "runtimes" into deployment configuration
Posted by Dascalita Dragos <dd...@gmail.com>.
I was thinking we could avoid the zip.
On Thu, Mar 9, 2017 at 4:13 PM Rodric Rabbah <ro...@gmail.com> wrote:
> You can send a zip file to a docker container. So if the image is
> openwhisk/dockerskeleton or openwhisk/swift3action it works as you'd
> expect. In either case there's no docker pull. And this is doable today
> already. What am I missing from your explanation?
>
> -r
>
> On Mar 9, 2017, at 6:44 PM, Dascalita Dragos <dd...@gmail.com> wrote:
>
> >> Isn't that what we call "blackbox" (docker) actions now?
> >
> > IIRC, with "blackbox", developers need to rebuild the image every time,
> > push it to a registry then invoke the action:
> > ```
> > wsk action create --docker my-blackbox container/name
> > ```
> >
> > What I was imagining is a case where developers still deploy "code" but,
> in
> > addition, they get to specify which "container" to use for executing that
> > code. I.e.
> > ```
> > wsk action create my-action ./my-action.js --kind:container/name:tag
> > ```
> > Assuming that `container/name` extends an existing OW container.
> >
> > Now that I'm thinking at this, I'd say that probably a more elegant
> > solution would be for developers to describe the extra libs in the
> > `manifest.yaml` then, at deploy time, a control-plane component of OW
> > builds an optimized container to be used later when invoking the action
> ...
> > not an easier option to implement though.
> >
> >
> >
> >
> > On Thu, Mar 9, 2017 at 2:55 PM Rodric Rabbah <ro...@gmail.com> wrote:
> >
> >>> Extending the thought: what if we allow users to specify a custom
> >> container
> >> name when creating / updating actions, in case users want to ?
> >>
> >> Isn't that what we call "blackbox" (docker) actions now?
> >> But in general - the reason for this work - is along these lines.
> >> Rather than having a small number of fixed images per runtime,
> >> we can support many more.
> >>
> >> A concern however is that we don't want to incur docker pull costs for
> some
> >> set of curated images (in the future, you can image that docker actions
> >> come
> >> with an explicit push operation at create time - but we're not there
> yet).
> >>
> >> So by having an extensible list of images, it makes it easier to curate
> >> action runtimes.
> >>
> >> The scenario you describe otherwise is already supported (we know there
> are
> >> users
> >> who extend our nodejs base image with their own artifacts). This has the
> >> benefit
> >> of pulling in fewer layers.
> >>
> >> -r
> >>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
You can send a zip file to a docker container. So if the image is openwhisk/dockerskeleton or openwhisk/swift3action it works as you'd expect. In either case there's no docker pull. And this is doable today already. What am I missing from your explanation?
-r
On Mar 9, 2017, at 6:44 PM, Dascalita Dragos <dd...@gmail.com> wrote:
>> Isn't that what we call "blackbox" (docker) actions now?
>
> IIRC, with "blackbox", developers need to rebuild the image every time,
> push it to a registry then invoke the action:
> ```
> wsk action create --docker my-blackbox container/name
> ```
>
> What I was imagining is a case where developers still deploy "code" but, in
> addition, they get to specify which "container" to use for executing that
> code. I.e.
> ```
> wsk action create my-action ./my-action.js --kind:container/name:tag
> ```
> Assuming that `container/name` extends an existing OW container.
>
> Now that I'm thinking at this, I'd say that probably a more elegant
> solution would be for developers to describe the extra libs in the
> `manifest.yaml` then, at deploy time, a control-plane component of OW
> builds an optimized container to be used later when invoking the action ...
> not an easier option to implement though.
>
>
>
>
> On Thu, Mar 9, 2017 at 2:55 PM Rodric Rabbah <ro...@gmail.com> wrote:
>
>>> Extending the thought: what if we allow users to specify a custom
>> container
>> name when creating / updating actions, in case users want to ?
>>
>> Isn't that what we call "blackbox" (docker) actions now?
>> But in general - the reason for this work - is along these lines.
>> Rather than having a small number of fixed images per runtime,
>> we can support many more.
>>
>> A concern however is that we don't want to incur docker pull costs for some
>> set of curated images (in the future, you can image that docker actions
>> come
>> with an explicit push operation at create time - but we're not there yet).
>>
>> So by having an extensible list of images, it makes it easier to curate
>> action runtimes.
>>
>> The scenario you describe otherwise is already supported (we know there are
>> users
>> who extend our nodejs base image with their own artifacts). This has the
>> benefit
>> of pulling in fewer layers.
>>
>> -r
>>
Re: factoring out "runtimes" into deployment configuration
Posted by Dascalita Dragos <dd...@gmail.com>.
> Isn't that what we call "blackbox" (docker) actions now?
IIRC, with "blackbox", developers need to rebuild the image every time,
push it to a registry then invoke the action:
```
wsk action create --docker my-blackbox container/name
```
What I was imagining is a case where developers still deploy "code" but, in
addition, they get to specify which "container" to use for executing that
code. I.e.
```
wsk action create my-action ./my-action.js --kind:container/name:tag
```
Assuming that `container/name` extends an existing OW container.
Now that I'm thinking at this, I'd say that probably a more elegant
solution would be for developers to describe the extra libs in the
`manifest.yaml` then, at deploy time, a control-plane component of OW
builds an optimized container to be used later when invoking the action ...
not an easier option to implement though.
On Thu, Mar 9, 2017 at 2:55 PM Rodric Rabbah <ro...@gmail.com> wrote:
> > Extending the thought: what if we allow users to specify a custom
> container
> name when creating / updating actions, in case users want to ?
>
> Isn't that what we call "blackbox" (docker) actions now?
> But in general - the reason for this work - is along these lines.
> Rather than having a small number of fixed images per runtime,
> we can support many more.
>
> A concern however is that we don't want to incur docker pull costs for some
> set of curated images (in the future, you can image that docker actions
> come
> with an explicit push operation at create time - but we're not there yet).
>
> So by having an extensible list of images, it makes it easier to curate
> action runtimes.
>
> The scenario you describe otherwise is already supported (we know there are
> users
> who extend our nodejs base image with their own artifacts). This has the
> benefit
> of pulling in fewer layers.
>
> -r
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
> Extending the thought: what if we allow users to specify a custom
container
name when creating / updating actions, in case users want to ?
Isn't that what we call "blackbox" (docker) actions now?
But in general - the reason for this work - is along these lines.
Rather than having a small number of fixed images per runtime,
we can support many more.
A concern however is that we don't want to incur docker pull costs for some
set of curated images (in the future, you can image that docker actions come
with an explicit push operation at create time - but we're not there yet).
So by having an extensible list of images, it makes it easier to curate
action runtimes.
The scenario you describe otherwise is already supported (we know there are
users
who extend our nodejs base image with their own artifacts). This has the
benefit
of pulling in fewer layers.
-r
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
OK - got it (fwiw that's mostly a CLI tweat to enable it - "2" from earlier
discussions in a longer arc).
-r
On Fri, Mar 10, 2017 at 6:55 AM, Alex Glikson <GL...@il.ibm.com> wrote:
> Yes, zip or code - just like with 'native' actions.
> Think of a case when I am developing a bunch of serverless applications in
> a certain application domain, where certain libraries are a common prereq
> (e.g., opencv). Even if all the dependencies were pure nodejs modules
> (which is not the case for opencv), I would prefer working with my own
> version of the nodejs image and just pass the application code rather than
> pushing large zip files with the same dependencies to all the actions
> (those zip files also have quite restrictive size limit at the moment).
>
> Regards,
> Alex
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 10/03/2017 01:14 PM
> Subject: Re: factoring out "runtimes" into deployment configuration
>
>
>
> Ah are you referring to the zip file with "--docker" where you can't name
> the image in the cli today?
>
> -r
>
> > On Mar 10, 2017, at 6:00 AM, Alex Glikson <GL...@il.ibm.com> wrote:
> >
> > I was referring to custom images, which are based on standard ones but
> > with a certain twist (e.g. additional libraries). They would be
> 'blackbox'
> > in terms of image management, but "native" in terms of action code
> > lifecycle. Today you need to use REST API to create/update such actions.
> > Agree that some optimization around #2 would be important to improve
> > performance for such actions.
> > Fine to have popular/approved images in a manifest for now, but would be
>
> > good to have the design flexible enough to enable future replacement
> with
> > a dynamic mechanism (maybe it already is).
> >
> > Regards,
> > Alex
> >
> >
> ============================================================
> ===============
> > Alex Glikson
> > Senior Researcher, Cloud Platforms, IBM Research - Haifa; IBM Lead for
> > FIWARE
> > Email: glikson@il.ibm.com | Skype: alex.glikson | Phone: +972-4-8281085
> |
> > Mobile: +972-54-6466667
> >
> >
> >
> >
> > From: Rodric Rabbah <ro...@gmail.com>
> > To: dev@openwhisk.apache.org
> > Date: 10/03/2017 12:32 PM
> > Subject: Re: factoring out "runtimes" into deployment
> configuration
> >
> >
> >
> > For 1. The cli now is doing the inference for you, sure we can replace
> > that inference with an image name so instead of --kind python you'd
> write
> > --image openwhisk/python. But...
> >
> > The issue more generally however is if you don't solve 2, making it
> easier
> > for users to use images is going to get them penalized for using
> arbitrary
> > but openwhisk-compatible images. So without addressing how to pull
> images
> > to local docker registries at create time for example, this is a non
> > starter in my opinion.
> >
> > The difference with the openwhisk/* images is that they are already
> > pulled, hence that addresses 2 for a limited set of images. These are
> the
> > popular images, per se. If we're missing one, we add it to the manifest.
> >
> > There is still, I posit, a need to know which are the "popular" or
> > approved images to use. That's what the manifest now centralizes into
> the
> > deployment configuration.
> >
> > -r
> >
> >> On Mar 10, 2017, at 2:01 AM, Alex Glikson <GL...@il.ibm.com> wrote:
> >>
> >> I can think of couple of ways to make what you called "kind-aware
> >> blackbox" actions more developer-friendly:
> >> 1. Support them in the CLI (i.e., supporting both "image" and "code"
> >> arguments). This would make the developer experience straightforard.
> >> 2. Generalize the caching/pooling mechanism to handle *popular* images,
>
> >> rather than a hard-coded set of "native" kind images. This would
> > eliminate
> >> the performance overhead.
> >>
> >> Regards,
> >> Alex
> >>
> >> P.S. Once we do #1 and #2 above, my claim is that we don't need
> >> pre-defined set of "kind" images any more, and just need a flexible
> >> hierarchy of images that anyone can reuse and extend
> >>
> >> Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
> >>
> >>> From: Markus Thömmes <ma...@me.com>
> >>> To: dev@openwhisk.apache.org
> >>> Date: 10/03/2017 12:48 AM
> >>> Subject: Re: factoring out "runtimes" into deployment configuration
> >>>
> >>> What you're referring to is basically a "kind-aware" blackbox
> >>> action, where you get the easyness of injecting your code and the
> >>> flexibility of using whatever you want in the image as long as the
> >>> interface fits.
> >>>
> >>> I generally like the idea and brought it up as well once (i custom
> >>> built a blackbox to do something similar). Note though that the same
> >>> performance implications as with blackbox images apply.
> >>>
> >>> Von meinem iPhone gesendet
> >>>
> >>>> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
> >>>>
> >>>> With the "runtimeManifest" property I'm wondering if we can also
> >> expose the
> >>>> name of the container image instead of having to compute it in the
> >> code ?
> >>>> Extending the thought: what if we allow users to specify a custom
> >> container
> >>>> name when creating / updating actions, in case users want to ?
> >>>>
> >>>> I'm thinking at the use-case when a package with a group of actions
> >>>> requires multiple libraries for each action. To simplify my example
> >> I'm
> >>>> going to refer only to JS actions:
> >>>> - Today: when actions needs extra libs users have 2 options :
> >>>> a) create a zip
> >>>> b) browserify/"compile" JS into a single JS file
> >>>> - What I'm thinking: allow users to create their own "runtime"
> image
> >>>> where they can bundle those dependent libs into. All the actions in
> >> the
> >>>> package would be smaller, they would use less space and would
> probably
> >
> >> flow
> >>>> easier through the system when being invoked. A potential problem in
> >> this
> >>>> case would be whether it's secure to let users push their own images
> >> in the
> >>>> OW nodes, but at the same time allowing them to upload a ZIP is
> >>>> conceptually also a "package"/"container" that can contain anything;
> I
> >>>> guess the security aspect would still remain even with user-defined
> >>>> containers.
> >>>>
> >>>> Dragos
> >>>>
> >>>>
> >>>>
> >>>>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
> >> wrote:
> >>>>>
> >>>>> Hi Rodric,
> >>>>>
> >>>>> I think that?s great.
> >>>>>
> >>>>> Only comment (more to Matt than you):
> >>>>> If the runtimes are dynamic by deployment (and of course over time)
> >> this
> >>>>> probably should be reflected in the packaging format spec [1].
> >>>>> (see also previous comment [2]:)
> >>>>>
> >>>>> - line 252: not sure if the list of runtime names should be in the
> >>>>> normative
> >>>>> section of a spec, because (as you also write) this list will
> >> fluctuate
> >>>>> between
> >>>>> deployments and over time. OTOH there should be established well
> >> known
> >>>>> names for
> >>>>> well-known runtimes. Maybe point to a community-curated, separate
> >> markdown
> >>>>> file
> >>>>> (where new entries get a PR that can be voted upon)?
> >>>>>
> >>>>> Cheers
> >>>>> Michael
> >>>>>
> >>>>> [1]
> >>>>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
> >>> specification/openwhisk_v0.8.pdf
> >>>>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
> >>>>> rodric@gmail.com>> wrote:
> >>>>>
> >>>>> I've been thinking about how we can add more action runtimes more
> >> easily -
> >>>>> without changing the backend (API, controller, invoker, CLI). One
> way
> >
> >> I'm
> >>>>> leaning toward and started prototyping to get a better understanding
>
> >> of the
> >>>>> feasibility is to strip all runtime types from the backend (and CLI)
>
> >> and
> >>>>> encode the information about what action runtimes the system
> supports
> >>>>> through configuration parameters.
> >>>>>
> >>>>> This will make it possible to decouple deploying the core components
>
> >> from
> >>>>> managing the action runtime images. An example of encoding the
> >> runtime
> >>>>> information in a deployment configuration is
> >>>>> https://github.com/openwhisk/openwhisk/pull/1948.
> >>>>>
> >>>>> In support of this general direction (even if we settle on a
> >> different
> >>>>> approach), I have increasingly flattened the "Exec" type hierarchy
> in
> >
> >> the
> >>>>> backend[1-3]
> >>>>>
> >>>>> [1] https://github.com/openwhisk/openwhisk/pull/1938
> >>>>> [2] https://github.com/openwhisk/openwhisk/pull/1942
> >>>>> [3] https://github.com/openwhisk/openwhisk/pull/1911
> >>>>>
> >>>>> A sketch of how to use the runtime manifest can be seen in this WIP
> >> commit:
> >>>>>
> >>>>> https://github.com/rabbah/openwhisk/commit/
> >>> a8890abe51a3e7e6a34f96a46798de660583e36f
> >>>>>
> >>>>> I can see how using this we can add new versions of Node.js,
> >> python:3, and
> >>>>> other handy images by curating a list of these in a separate
> process.
> >>>>>
> >>>>> Feedback, as always, solicited and appreciated.
> >>>>>
> >>>>> -r
> >>>>>
> >>>>>
> >>>
> >>
> >>
> >
> >
> >
> >
> >
>
>
>
>
>
>
Re: Current status of Jenkins setup for OpenWhisk
Posted by Carlos Santana <cs...@gmail.com>.
Welcome back Vicent
- Carlos Santana
@csantanapr
> On Mar 12, 2019, at 11:30 AM, Matt Rutkowski <mr...@us.ibm.com> wrote:
>
> Welcome back Vincent! It is good to have your skill back to help the
> project with multi-node staging!
>
> -Matt
>
>
>
>
>
> From: "Vincent S Hou" <sh...@us.ibm.com>
> To: dev@openwhisk.apache.org
> Date: 03/12/2019 10:01 AM
> Subject: Current status of Jenkins setup for OpenWhisk
>
>
>
> Hi OpenWhiskers,
>
> I have come back from my holiday last week, and officially start to work
> this week. During the past few days, I have made some progress in terms of
> getting the access to openwhisk VMs and Apache Jenkins.
>
> Current state:
> My apache ID has granted root access to the VMs:
> openwhisk-vm1-he-de.apache.org
> openwhisk-vm2-he-de.apache.org
> openwhisk-vm3-he-de.apache.org
>
> I am now able to use the command ssh openwhisk-vm1-he-de.apache.org -l <my
> apache ID> to access the VM via terminal.
>
> My Apache ID has been added into the hudson-jobadmin group:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__whimsy.apache.org_roster_group_hudson-2Djobadmin&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=-e-Ty-ohYS2xvL-9BVb-Z5W25ZsmOFwxplXIE469IFU&s=LznFkcRufXe67pIkVWjsvbfexSapnQJZpVYWZIm_QbM&e=
> , which means I have earned the admin access to Apache Jenkins environment
> at
> https://urldefense.proofpoint.com/v2/url?u=https-3A__builds.apache.org_&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=-e-Ty-ohYS2xvL-9BVb-Z5W25ZsmOFwxplXIE469IFU&s=DlXo7_7di99K3NIwYr-x16N_CGcSPBAmAK92cP5S5fw&e=
> . I should be able to create build or pipeline.
>
> I have granted the user "jenkins" root access within three of these VMs.
>
> Work items to do:
> Next, I will configured the ssh access among three of these VMs, since I
> plan to use one VM as the master, launching all the services except
> invoker, and the other two as designated nodes to launch invoker service.
>
> Travis CI has already tested the case, when all the openwhisk services are
> installed on one server. I plan to use Jenkins to test openwhisk installed
> in a distributed way.
>
> Welcome to chime in and add your comments, if you have any idea about how
> to use these three VMs for OpenWhisk.
>
> Thank you.
>
> Best wishes.
> Vincent Hou (侯胜博)
>
> Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM
> Cloud
>
> Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
> Phone: +1(919)254-7182
> Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United
> States
>
>
>
>
>
>
Re: Current status of Jenkins setup for OpenWhisk
Posted by Matt Rutkowski <mr...@us.ibm.com>.
Welcome back Vincent! It is good to have your skill back to help the
project with multi-node staging!
-Matt
From: "Vincent S Hou" <sh...@us.ibm.com>
To: dev@openwhisk.apache.org
Date: 03/12/2019 10:01 AM
Subject: Current status of Jenkins setup for OpenWhisk
Hi OpenWhiskers,
I have come back from my holiday last week, and officially start to work
this week. During the past few days, I have made some progress in terms of
getting the access to openwhisk VMs and Apache Jenkins.
Current state:
My apache ID has granted root access to the VMs:
openwhisk-vm1-he-de.apache.org
openwhisk-vm2-he-de.apache.org
openwhisk-vm3-he-de.apache.org
I am now able to use the command ssh openwhisk-vm1-he-de.apache.org -l <my
apache ID> to access the VM via terminal.
My Apache ID has been added into the hudson-jobadmin group:
https://urldefense.proofpoint.com/v2/url?u=https-3A__whimsy.apache.org_roster_group_hudson-2Djobadmin&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=-e-Ty-ohYS2xvL-9BVb-Z5W25ZsmOFwxplXIE469IFU&s=LznFkcRufXe67pIkVWjsvbfexSapnQJZpVYWZIm_QbM&e=
, which means I have earned the admin access to Apache Jenkins environment
at
https://urldefense.proofpoint.com/v2/url?u=https-3A__builds.apache.org_&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=6zQLM7Gc0Sv1iwayKOKa4_SFxRIxS478q2gZlAJj4Zw&m=-e-Ty-ohYS2xvL-9BVb-Z5W25ZsmOFwxplXIE469IFU&s=DlXo7_7di99K3NIwYr-x16N_CGcSPBAmAK92cP5S5fw&e=
. I should be able to create build or pipeline.
I have granted the user "jenkins" root access within three of these VMs.
Work items to do:
Next, I will configured the ssh access among three of these VMs, since I
plan to use one VM as the master, launching all the services except
invoker, and the other two as designated nodes to launch invoker service.
Travis CI has already tested the case, when all the openwhisk services are
installed on one server. I plan to use Jenkins to test openwhisk installed
in a distributed way.
Welcome to chime in and add your comments, if you have any idea about how
to use these three VMs for OpenWhisk.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM
Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United
States
Current status of Jenkins setup for OpenWhisk
Posted by Vincent S Hou <sh...@us.ibm.com>.
Hi OpenWhiskers,
I have come back from my holiday last week, and officially start to work this week. During the past few days, I have made some progress in terms of getting the access to openwhisk VMs and Apache Jenkins.
Current state:
My apache ID has granted root access to the VMs:
openwhisk-vm1-he-de.apache.org
openwhisk-vm2-he-de.apache.org
openwhisk-vm3-he-de.apache.org
I am now able to use the command ssh openwhisk-vm1-he-de.apache.org -l <my apache ID> to access the VM via terminal.
My Apache ID has been added into the hudson-jobadmin group: https://whimsy.apache.org/roster/group/hudson-jobadmin, which means I have earned the admin access to Apache Jenkins environment at https://builds.apache.org/. I should be able to create build or pipeline.
I have granted the user "jenkins" root access within three of these VMs.
Work items to do:
Next, I will configured the ssh access among three of these VMs, since I plan to use one VM as the master, launching all the services except invoker, and the other two as designated nodes to launch invoker service.
Travis CI has already tested the case, when all the openwhisk services are installed on one server. I plan to use Jenkins to test openwhisk installed in a distributed way.
Welcome to chime in and add your comments, if you have any idea about how to use these three VMs for OpenWhisk.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
Re: Current status of Jenkins setup for OpenWhisk
Posted by Vincent S Hou <sh...@us.ibm.com>.
Hi folks,
I have improved the way how we utilize the three VMs for Jenkins pipeline.
Instead of using openwhisk1 as the edge service node, running all the services except invoker, we can also schedule openwhisk2 and openwhisk3 to run the edge service as well.
How to implement the distributed model to deploy openwhisk?
The deployment topology is orchestrated as below:
If openwhisk1 runs as the edge service, openwhisk2 runs invoker0 and openwhisk3 runs invoker1.
If openwhisk2 runs as the edge service, openwhisk3 runs invoker0 and openwhisk1 runs invoker1.
If openwhisk3 runs as the edge service, openwhisk1 runs invoker0 and openwhisk2 runs invoker1.
Even three of these deployments are able to run at the same time, instead of one deployment at a time.
For safety reasons, I also added resource lock to each node in Jenkinsfile, so that even if the job is scheduled to a busy node, it will wait there till the previous job finishes.
I am now looking at the test to stop one of the invokers and run the designated tests. If you have any idea on how to do that, welcome to chime in.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
-----Vincent S Hou/Raleigh/IBM wrote: -----
To: dev@openwhisk.apache.org
From: Vincent S Hou/Raleigh/IBM
Date: 03/12/2019 10:59AM
Subject: Current status of Jenkins setup for OpenWhisk
Hi OpenWhiskers,
I have come back from my holiday last week, and officially start to work this week. During the past few days, I have made some progress in terms of getting the access to openwhisk VMs and Apache Jenkins.
Current state:
My apache ID has granted root access to the VMs:
openwhisk-vm1-he-de.apache.org
openwhisk-vm2-he-de.apache.org
openwhisk-vm3-he-de.apache.org
I am now able to use the command ssh openwhisk-vm1-he-de.apache.org -l <my apache ID> to access the VM via terminal.
My Apache ID has been added into the hudson-jobadmin group: https://whimsy.apache.org/roster/group/hudson-jobadmin, which means I have earned the admin access to Apache Jenkins environment at https://builds.apache.org/. I should be able to create build or pipeline.
I have granted the user "jenkins" root access within three of these VMs.
Work items to do:
Next, I will configured the ssh access among three of these VMs, since I plan to use one VM as the master, launching all the services except invoker, and the other two as designated nodes to launch invoker service.
Travis CI has already tested the case, when all the openwhisk services are installed on one server. I plan to use Jenkins to test openwhisk installed in a distributed way.
Welcome to chime in and add your comments, if you have any idea about how to use these three VMs for OpenWhisk.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
Re: Current status of Jenkins setup for OpenWhisk
Posted by Vincent S Hou <sh...@us.ibm.com>.
Hi OpenWhiskers,
I have already made the Jenkins pipeline working.
I resolved the issue about the constantly failing test case by enabling the memory limit of docker on each openwhisk VM.
Three VMs are able to run three instances of OpenWhisk deployment and test at the same time.
Normally the result will turn out passed for our code located in master branch of OpenWhisk. The pipeline is healthy now.
If the pipeline build fails, it could be due to the error of the PR. However, report the issues you find, if you believe it is caused by mistake of the pipeline.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
-----Vincent S Hou/Raleigh/IBM wrote: -----
To: dev@openwhisk.apache.org
From: Vincent S Hou/Raleigh/IBM
Date: 03/26/2019 11:26AM
Subject: Re: Current status of Jenkins setup for OpenWhisk
Hi folks,
I have improved the way how we utilize the three VMs for Jenkins pipeline.
Instead of using openwhisk1 as the edge service node, running all the services except invoker, we can also schedule openwhisk2 and openwhisk3 to run the edge service as well.
How to implement the distributed model to deploy openwhisk?
The deployment topology is orchestrated as below:
If openwhisk1 runs as the edge service, openwhisk2 runs invoker0 and openwhisk3 runs invoker1.
If openwhisk2 runs as the edge service, openwhisk3 runs invoker0 and openwhisk1 runs invoker1.
If openwhisk3 runs as the edge service, openwhisk1 runs invoker0 and openwhisk2 runs invoker1.
Even three of these deployments are able to run at the same time, instead of one deployment at a time.
For safety reasons, I also added resource lock to each node in Jenkinsfile, so that even if the job is scheduled to a busy node, it will wait there till the previous job finishes.
I am now looking at the test to stop one of the invokers and run the designated tests. If you have any idea on how to do that, welcome to chime in.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
-----Vincent S Hou/Raleigh/IBM wrote: -----
To: dev@openwhisk.apache.org
From: Vincent S Hou/Raleigh/IBM
Date: 03/12/2019 10:59AM
Subject: Current status of Jenkins setup for OpenWhisk
Hi OpenWhiskers,
I have come back from my holiday last week, and officially start to work this week. During the past few days, I have made some progress in terms of getting the access to openwhisk VMs and Apache Jenkins.
Current state:
My apache ID has granted root access to the VMs:
openwhisk-vm1-he-de.apache.org
openwhisk-vm2-he-de.apache.org
openwhisk-vm3-he-de.apache.org
I am now able to use the command ssh openwhisk-vm1-he-de.apache.org -l <my apache ID> to access the VM via terminal.
My Apache ID has been added into the hudson-jobadmin group: https://whimsy.apache.org/roster/group/hudson-jobadmin, which means I have earned the admin access to Apache Jenkins environment at https://builds.apache.org/. I should be able to create build or pipeline.
I have granted the user "jenkins" root access within three of these VMs.
Work items to do:
Next, I will configured the ssh access among three of these VMs, since I plan to use one VM as the master, launching all the services except invoker, and the other two as designated nodes to launch invoker service.
Travis CI has already tested the case, when all the openwhisk services are installed on one server. I plan to use Jenkins to test openwhisk installed in a distributed way.
Welcome to chime in and add your comments, if you have any idea about how to use these three VMs for OpenWhisk.
Thank you.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
Yes, zip or code - just like with 'native' actions.
Think of a case when I am developing a bunch of serverless applications in
a certain application domain, where certain libraries are a common prereq
(e.g., opencv). Even if all the dependencies were pure nodejs modules
(which is not the case for opencv), I would prefer working with my own
version of the nodejs image and just pass the application code rather than
pushing large zip files with the same dependencies to all the actions
(those zip files also have quite restrictive size limit at the moment).
Regards,
Alex
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.apache.org
Date: 10/03/2017 01:14 PM
Subject: Re: factoring out "runtimes" into deployment configuration
Ah are you referring to the zip file with "--docker" where you can't name
the image in the cli today?
-r
> On Mar 10, 2017, at 6:00 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> I was referring to custom images, which are based on standard ones but
> with a certain twist (e.g. additional libraries). They would be
'blackbox'
> in terms of image management, but "native" in terms of action code
> lifecycle. Today you need to use REST API to create/update such actions.
> Agree that some optimization around #2 would be important to improve
> performance for such actions.
> Fine to have popular/approved images in a manifest for now, but would be
> good to have the design flexible enough to enable future replacement
with
> a dynamic mechanism (maybe it already is).
>
> Regards,
> Alex
>
>
===========================================================================
> Alex Glikson
> Senior Researcher, Cloud Platforms, IBM Research - Haifa; IBM Lead for
> FIWARE
> Email: glikson@il.ibm.com | Skype: alex.glikson | Phone: +972-4-8281085
|
> Mobile: +972-54-6466667
>
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 10/03/2017 12:32 PM
> Subject: Re: factoring out "runtimes" into deployment
configuration
>
>
>
> For 1. The cli now is doing the inference for you, sure we can replace
> that inference with an image name so instead of --kind python you'd
write
> --image openwhisk/python. But...
>
> The issue more generally however is if you don't solve 2, making it
easier
> for users to use images is going to get them penalized for using
arbitrary
> but openwhisk-compatible images. So without addressing how to pull
images
> to local docker registries at create time for example, this is a non
> starter in my opinion.
>
> The difference with the openwhisk/* images is that they are already
> pulled, hence that addresses 2 for a limited set of images. These are
the
> popular images, per se. If we're missing one, we add it to the manifest.
>
> There is still, I posit, a need to know which are the "popular" or
> approved images to use. That's what the manifest now centralizes into
the
> deployment configuration.
>
> -r
>
>> On Mar 10, 2017, at 2:01 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>>
>> I can think of couple of ways to make what you called "kind-aware
>> blackbox" actions more developer-friendly:
>> 1. Support them in the CLI (i.e., supporting both "image" and "code"
>> arguments). This would make the developer experience straightforard.
>> 2. Generalize the caching/pooling mechanism to handle *popular* images,
>> rather than a hard-coded set of "native" kind images. This would
> eliminate
>> the performance overhead.
>>
>> Regards,
>> Alex
>>
>> P.S. Once we do #1 and #2 above, my claim is that we don't need
>> pre-defined set of "kind" images any more, and just need a flexible
>> hierarchy of images that anyone can reuse and extend
>>
>> Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
>>
>>> From: Markus Thömmes <ma...@me.com>
>>> To: dev@openwhisk.apache.org
>>> Date: 10/03/2017 12:48 AM
>>> Subject: Re: factoring out "runtimes" into deployment configuration
>>>
>>> What you're referring to is basically a "kind-aware" blackbox
>>> action, where you get the easyness of injecting your code and the
>>> flexibility of using whatever you want in the image as long as the
>>> interface fits.
>>>
>>> I generally like the idea and brought it up as well once (i custom
>>> built a blackbox to do something similar). Note though that the same
>>> performance implications as with blackbox images apply.
>>>
>>> Von meinem iPhone gesendet
>>>
>>>> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
>>>>
>>>> With the "runtimeManifest" property I'm wondering if we can also
>> expose the
>>>> name of the container image instead of having to compute it in the
>> code ?
>>>> Extending the thought: what if we allow users to specify a custom
>> container
>>>> name when creating / updating actions, in case users want to ?
>>>>
>>>> I'm thinking at the use-case when a package with a group of actions
>>>> requires multiple libraries for each action. To simplify my example
>> I'm
>>>> going to refer only to JS actions:
>>>> - Today: when actions needs extra libs users have 2 options :
>>>> a) create a zip
>>>> b) browserify/"compile" JS into a single JS file
>>>> - What I'm thinking: allow users to create their own "runtime"
image
>>>> where they can bundle those dependent libs into. All the actions in
>> the
>>>> package would be smaller, they would use less space and would
probably
>
>> flow
>>>> easier through the system when being invoked. A potential problem in
>> this
>>>> case would be whether it's secure to let users push their own images
>> in the
>>>> OW nodes, but at the same time allowing them to upload a ZIP is
>>>> conceptually also a "package"/"container" that can contain anything;
I
>>>> guess the security aspect would still remain even with user-defined
>>>> containers.
>>>>
>>>> Dragos
>>>>
>>>>
>>>>
>>>>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
>> wrote:
>>>>>
>>>>> Hi Rodric,
>>>>>
>>>>> I think that?s great.
>>>>>
>>>>> Only comment (more to Matt than you):
>>>>> If the runtimes are dynamic by deployment (and of course over time)
>> this
>>>>> probably should be reflected in the packaging format spec [1].
>>>>> (see also previous comment [2]:)
>>>>>
>>>>> - line 252: not sure if the list of runtime names should be in the
>>>>> normative
>>>>> section of a spec, because (as you also write) this list will
>> fluctuate
>>>>> between
>>>>> deployments and over time. OTOH there should be established well
>> known
>>>>> names for
>>>>> well-known runtimes. Maybe point to a community-curated, separate
>> markdown
>>>>> file
>>>>> (where new entries get a PR that can be voted upon)?
>>>>>
>>>>> Cheers
>>>>> Michael
>>>>>
>>>>> [1]
>>>>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
>>> specification/openwhisk_v0.8.pdf
>>>>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>>>>>
>>>>>
>>>>>
>>>>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
>>>>> rodric@gmail.com>> wrote:
>>>>>
>>>>> I've been thinking about how we can add more action runtimes more
>> easily -
>>>>> without changing the backend (API, controller, invoker, CLI). One
way
>
>> I'm
>>>>> leaning toward and started prototyping to get a better understanding
>> of the
>>>>> feasibility is to strip all runtime types from the backend (and CLI)
>> and
>>>>> encode the information about what action runtimes the system
supports
>>>>> through configuration parameters.
>>>>>
>>>>> This will make it possible to decouple deploying the core components
>> from
>>>>> managing the action runtime images. An example of encoding the
>> runtime
>>>>> information in a deployment configuration is
>>>>> https://github.com/openwhisk/openwhisk/pull/1948.
>>>>>
>>>>> In support of this general direction (even if we settle on a
>> different
>>>>> approach), I have increasingly flattened the "Exec" type hierarchy
in
>
>> the
>>>>> backend[1-3]
>>>>>
>>>>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>>>>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>>>>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>>>>
>>>>> A sketch of how to use the runtime manifest can be seen in this WIP
>> commit:
>>>>>
>>>>> https://github.com/rabbah/openwhisk/commit/
>>> a8890abe51a3e7e6a34f96a46798de660583e36f
>>>>>
>>>>> I can see how using this we can add new versions of Node.js,
>> python:3, and
>>>>> other handy images by curating a list of these in a separate
process.
>>>>>
>>>>> Feedback, as always, solicited and appreciated.
>>>>>
>>>>> -r
>>>>>
>>>>>
>>>
>>
>>
>
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
Ah are you referring to the zip file with "--docker" where you can't name the image in the cli today?
-r
> On Mar 10, 2017, at 6:00 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> I was referring to custom images, which are based on standard ones but
> with a certain twist (e.g. additional libraries). They would be 'blackbox'
> in terms of image management, but "native" in terms of action code
> lifecycle. Today you need to use REST API to create/update such actions.
> Agree that some optimization around #2 would be important to improve
> performance for such actions.
> Fine to have popular/approved images in a manifest for now, but would be
> good to have the design flexible enough to enable future replacement with
> a dynamic mechanism (maybe it already is).
>
> Regards,
> Alex
>
> ===========================================================================
> Alex Glikson
> Senior Researcher, Cloud Platforms, IBM Research - Haifa; IBM Lead for
> FIWARE
> Email: glikson@il.ibm.com | Skype: alex.glikson | Phone: +972-4-8281085 |
> Mobile: +972-54-6466667
>
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 10/03/2017 12:32 PM
> Subject: Re: factoring out "runtimes" into deployment configuration
>
>
>
> For 1. The cli now is doing the inference for you, sure we can replace
> that inference with an image name so instead of --kind python you'd write
> --image openwhisk/python. But...
>
> The issue more generally however is if you don't solve 2, making it easier
> for users to use images is going to get them penalized for using arbitrary
> but openwhisk-compatible images. So without addressing how to pull images
> to local docker registries at create time for example, this is a non
> starter in my opinion.
>
> The difference with the openwhisk/* images is that they are already
> pulled, hence that addresses 2 for a limited set of images. These are the
> popular images, per se. If we're missing one, we add it to the manifest.
>
> There is still, I posit, a need to know which are the "popular" or
> approved images to use. That's what the manifest now centralizes into the
> deployment configuration.
>
> -r
>
>> On Mar 10, 2017, at 2:01 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>>
>> I can think of couple of ways to make what you called "kind-aware
>> blackbox" actions more developer-friendly:
>> 1. Support them in the CLI (i.e., supporting both "image" and "code"
>> arguments). This would make the developer experience straightforard.
>> 2. Generalize the caching/pooling mechanism to handle *popular* images,
>> rather than a hard-coded set of "native" kind images. This would
> eliminate
>> the performance overhead.
>>
>> Regards,
>> Alex
>>
>> P.S. Once we do #1 and #2 above, my claim is that we don't need
>> pre-defined set of "kind" images any more, and just need a flexible
>> hierarchy of images that anyone can reuse and extend
>>
>> Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
>>
>>> From: Markus Thömmes <ma...@me.com>
>>> To: dev@openwhisk.apache.org
>>> Date: 10/03/2017 12:48 AM
>>> Subject: Re: factoring out "runtimes" into deployment configuration
>>>
>>> What you're referring to is basically a "kind-aware" blackbox
>>> action, where you get the easyness of injecting your code and the
>>> flexibility of using whatever you want in the image as long as the
>>> interface fits.
>>>
>>> I generally like the idea and brought it up as well once (i custom
>>> built a blackbox to do something similar). Note though that the same
>>> performance implications as with blackbox images apply.
>>>
>>> Von meinem iPhone gesendet
>>>
>>>> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
>>>>
>>>> With the "runtimeManifest" property I'm wondering if we can also
>> expose the
>>>> name of the container image instead of having to compute it in the
>> code ?
>>>> Extending the thought: what if we allow users to specify a custom
>> container
>>>> name when creating / updating actions, in case users want to ?
>>>>
>>>> I'm thinking at the use-case when a package with a group of actions
>>>> requires multiple libraries for each action. To simplify my example
>> I'm
>>>> going to refer only to JS actions:
>>>> - Today: when actions needs extra libs users have 2 options :
>>>> a) create a zip
>>>> b) browserify/"compile" JS into a single JS file
>>>> - What I'm thinking: allow users to create their own "runtime" image
>>>> where they can bundle those dependent libs into. All the actions in
>> the
>>>> package would be smaller, they would use less space and would probably
>
>> flow
>>>> easier through the system when being invoked. A potential problem in
>> this
>>>> case would be whether it's secure to let users push their own images
>> in the
>>>> OW nodes, but at the same time allowing them to upload a ZIP is
>>>> conceptually also a "package"/"container" that can contain anything; I
>>>> guess the security aspect would still remain even with user-defined
>>>> containers.
>>>>
>>>> Dragos
>>>>
>>>>
>>>>
>>>>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
>> wrote:
>>>>>
>>>>> Hi Rodric,
>>>>>
>>>>> I think that?s great.
>>>>>
>>>>> Only comment (more to Matt than you):
>>>>> If the runtimes are dynamic by deployment (and of course over time)
>> this
>>>>> probably should be reflected in the packaging format spec [1].
>>>>> (see also previous comment [2]:)
>>>>>
>>>>> - line 252: not sure if the list of runtime names should be in the
>>>>> normative
>>>>> section of a spec, because (as you also write) this list will
>> fluctuate
>>>>> between
>>>>> deployments and over time. OTOH there should be established well
>> known
>>>>> names for
>>>>> well-known runtimes. Maybe point to a community-curated, separate
>> markdown
>>>>> file
>>>>> (where new entries get a PR that can be voted upon)?
>>>>>
>>>>> Cheers
>>>>> Michael
>>>>>
>>>>> [1]
>>>>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
>>> specification/openwhisk_v0.8.pdf
>>>>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>>>>>
>>>>>
>>>>>
>>>>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
>>>>> rodric@gmail.com>> wrote:
>>>>>
>>>>> I've been thinking about how we can add more action runtimes more
>> easily -
>>>>> without changing the backend (API, controller, invoker, CLI). One way
>
>> I'm
>>>>> leaning toward and started prototyping to get a better understanding
>> of the
>>>>> feasibility is to strip all runtime types from the backend (and CLI)
>> and
>>>>> encode the information about what action runtimes the system supports
>>>>> through configuration parameters.
>>>>>
>>>>> This will make it possible to decouple deploying the core components
>> from
>>>>> managing the action runtime images. An example of encoding the
>> runtime
>>>>> information in a deployment configuration is
>>>>> https://github.com/openwhisk/openwhisk/pull/1948.
>>>>>
>>>>> In support of this general direction (even if we settle on a
>> different
>>>>> approach), I have increasingly flattened the "Exec" type hierarchy in
>
>> the
>>>>> backend[1-3]
>>>>>
>>>>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>>>>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>>>>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>>>>
>>>>> A sketch of how to use the runtime manifest can be seen in this WIP
>> commit:
>>>>>
>>>>> https://github.com/rabbah/openwhisk/commit/
>>> a8890abe51a3e7e6a34f96a46798de660583e36f
>>>>>
>>>>> I can see how using this we can add new versions of Node.js,
>> python:3, and
>>>>> other handy images by curating a list of these in a separate process.
>>>>>
>>>>> Feedback, as always, solicited and appreciated.
>>>>>
>>>>> -r
>>>>>
>>>>>
>>>
>>
>>
>
>
>
>
>
Re: Tech Interchange call this Wed. Nov. 07 - add agenda topics here
Posted by Vincent S Hou <sh...@us.ibm.com>.
Hi folks,
Just a reminder: please send me the topic you would like to present on tomorrow's meeting. Thx.
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
-----"Vincent S Hou" <sh...@us.ibm.com> wrote: -----
To: dev@openwhisk.apache.org
From: "Vincent S Hou" <sh...@us.ibm.com>
Date: 11/05/2018 09:41PM
Subject: Tech Interchange call this Wed. Nov. 07 - add agenda topics here
Hi OpenWhiskers,
Please send me the any agenda items you'd like to present at the Tech Interchange call this Wednesday (Nov 07).
Call details:
Web Meeting: Tech Interchange (bi-weekly):
- Day-Time: Wednesdays, 11AM EDT (Eastern US), 5PM CEST (Central Europe),
4PM UTC, 12PM CST (Beijing)
- Zoom: https://zoom.us/my/asfopenwhisk
Thanks,
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
Tech Interchange call this Wed. Nov. 07 - add agenda topics here
Posted by Vincent S Hou <sh...@us.ibm.com>.
Hi OpenWhiskers,
Please send me the any agenda items you'd like to present at the Tech Interchange call this Wednesday (Nov 07).
Call details:
Web Meeting: Tech Interchange (bi-weekly):
- Day-Time: Wednesdays, 11AM EDT (Eastern US), 5PM CEST (Central Europe),
4PM UTC, 12PM CST (Beijing)
- Zoom: https://zoom.us/my/asfopenwhisk
Thanks,
Best wishes.
Vincent Hou (侯胜博)
Advisory Software Engineer, OpenWhisk Contributor, Open Technology, IBM Cloud
Notes ID: Vincent S Hou/Raleigh/IBM, E-mail: shou@us.ibm.com,
Phone: +1(919)254-7182
Address: 4205 S Miami Blvd (Cornwallis Drive), Durham, NC 27703, United States
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
I was referring to custom images, which are based on standard ones but
with a certain twist (e.g. additional libraries). They would be 'blackbox'
in terms of image management, but "native" in terms of action code
lifecycle. Today you need to use REST API to create/update such actions.
Agree that some optimization around #2 would be important to improve
performance for such actions.
Fine to have popular/approved images in a manifest for now, but would be
good to have the design flexible enough to enable future replacement with
a dynamic mechanism (maybe it already is).
Regards,
Alex
===========================================================================
Alex Glikson
Senior Researcher, Cloud Platforms, IBM Research - Haifa; IBM Lead for
FIWARE
Email: glikson@il.ibm.com | Skype: alex.glikson | Phone: +972-4-8281085 |
Mobile: +972-54-6466667
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.apache.org
Date: 10/03/2017 12:32 PM
Subject: Re: factoring out "runtimes" into deployment configuration
For 1. The cli now is doing the inference for you, sure we can replace
that inference with an image name so instead of --kind python you'd write
--image openwhisk/python. But...
The issue more generally however is if you don't solve 2, making it easier
for users to use images is going to get them penalized for using arbitrary
but openwhisk-compatible images. So without addressing how to pull images
to local docker registries at create time for example, this is a non
starter in my opinion.
The difference with the openwhisk/* images is that they are already
pulled, hence that addresses 2 for a limited set of images. These are the
popular images, per se. If we're missing one, we add it to the manifest.
There is still, I posit, a need to know which are the "popular" or
approved images to use. That's what the manifest now centralizes into the
deployment configuration.
-r
> On Mar 10, 2017, at 2:01 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> I can think of couple of ways to make what you called "kind-aware
> blackbox" actions more developer-friendly:
> 1. Support them in the CLI (i.e., supporting both "image" and "code"
> arguments). This would make the developer experience straightforard.
> 2. Generalize the caching/pooling mechanism to handle *popular* images,
> rather than a hard-coded set of "native" kind images. This would
eliminate
> the performance overhead.
>
> Regards,
> Alex
>
> P.S. Once we do #1 and #2 above, my claim is that we don't need
> pre-defined set of "kind" images any more, and just need a flexible
> hierarchy of images that anyone can reuse and extend
>
> Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
>
>> From: Markus Thömmes <ma...@me.com>
>> To: dev@openwhisk.apache.org
>> Date: 10/03/2017 12:48 AM
>> Subject: Re: factoring out "runtimes" into deployment configuration
>>
>> What you're referring to is basically a "kind-aware" blackbox
>> action, where you get the easyness of injecting your code and the
>> flexibility of using whatever you want in the image as long as the
>> interface fits.
>>
>> I generally like the idea and brought it up as well once (i custom
>> built a blackbox to do something similar). Note though that the same
>> performance implications as with blackbox images apply.
>>
>> Von meinem iPhone gesendet
>>
>>> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
>>>
>>> With the "runtimeManifest" property I'm wondering if we can also
> expose the
>>> name of the container image instead of having to compute it in the
> code ?
>>> Extending the thought: what if we allow users to specify a custom
> container
>>> name when creating / updating actions, in case users want to ?
>>>
>>> I'm thinking at the use-case when a package with a group of actions
>>> requires multiple libraries for each action. To simplify my example
> I'm
>>> going to refer only to JS actions:
>>> - Today: when actions needs extra libs users have 2 options :
>>> a) create a zip
>>> b) browserify/"compile" JS into a single JS file
>>> - What I'm thinking: allow users to create their own "runtime" image
>>> where they can bundle those dependent libs into. All the actions in
> the
>>> package would be smaller, they would use less space and would probably
> flow
>>> easier through the system when being invoked. A potential problem in
> this
>>> case would be whether it's secure to let users push their own images
> in the
>>> OW nodes, but at the same time allowing them to upload a ZIP is
>>> conceptually also a "package"/"container" that can contain anything; I
>>> guess the security aspect would still remain even with user-defined
>>> containers.
>>>
>>> Dragos
>>>
>>>
>>>
>>>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
> wrote:
>>>>
>>>> Hi Rodric,
>>>>
>>>> I think that?s great.
>>>>
>>>> Only comment (more to Matt than you):
>>>> If the runtimes are dynamic by deployment (and of course over time)
> this
>>>> probably should be reflected in the packaging format spec [1].
>>>> (see also previous comment [2]:)
>>>>
>>>> - line 252: not sure if the list of runtime names should be in the
>>>> normative
>>>> section of a spec, because (as you also write) this list will
> fluctuate
>>>> between
>>>> deployments and over time. OTOH there should be established well
> known
>>>> names for
>>>> well-known runtimes. Maybe point to a community-curated, separate
> markdown
>>>> file
>>>> (where new entries get a PR that can be voted upon)?
>>>>
>>>> Cheers
>>>> Michael
>>>>
>>>> [1]
>>>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
>> specification/openwhisk_v0.8.pdf
>>>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>>>>
>>>>
>>>>
>>>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
>>>> rodric@gmail.com>> wrote:
>>>>
>>>> I've been thinking about how we can add more action runtimes more
> easily -
>>>> without changing the backend (API, controller, invoker, CLI). One way
> I'm
>>>> leaning toward and started prototyping to get a better understanding
> of the
>>>> feasibility is to strip all runtime types from the backend (and CLI)
> and
>>>> encode the information about what action runtimes the system supports
>>>> through configuration parameters.
>>>>
>>>> This will make it possible to decouple deploying the core components
> from
>>>> managing the action runtime images. An example of encoding the
> runtime
>>>> information in a deployment configuration is
>>>> https://github.com/openwhisk/openwhisk/pull/1948.
>>>>
>>>> In support of this general direction (even if we settle on a
> different
>>>> approach), I have increasingly flattened the "Exec" type hierarchy in
> the
>>>> backend[1-3]
>>>>
>>>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>>>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>>>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>>>
>>>> A sketch of how to use the runtime manifest can be seen in this WIP
> commit:
>>>>
>>>> https://github.com/rabbah/openwhisk/commit/
>> a8890abe51a3e7e6a34f96a46798de660583e36f
>>>>
>>>> I can see how using this we can add new versions of Node.js,
> python:3, and
>>>> other handy images by curating a list of these in a separate process.
>>>>
>>>> Feedback, as always, solicited and appreciated.
>>>>
>>>> -r
>>>>
>>>>
>>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
For 1. The cli now is doing the inference for you, sure we can replace that inference with an image name so instead of --kind python you'd write --image openwhisk/python. But...
The issue more generally however is if you don't solve 2, making it easier for users to use images is going to get them penalized for using arbitrary but openwhisk-compatible images. So without addressing how to pull images to local docker registries at create time for example, this is a non starter in my opinion.
The difference with the openwhisk/* images is that they are already pulled, hence that addresses 2 for a limited set of images. These are the popular images, per se. If we're missing one, we add it to the manifest.
There is still, I posit, a need to know which are the "popular" or approved images to use. That's what the manifest now centralizes into the deployment configuration.
-r
> On Mar 10, 2017, at 2:01 AM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> I can think of couple of ways to make what you called "kind-aware
> blackbox" actions more developer-friendly:
> 1. Support them in the CLI (i.e., supporting both "image" and "code"
> arguments). This would make the developer experience straightforard.
> 2. Generalize the caching/pooling mechanism to handle *popular* images,
> rather than a hard-coded set of "native" kind images. This would eliminate
> the performance overhead.
>
> Regards,
> Alex
>
> P.S. Once we do #1 and #2 above, my claim is that we don't need
> pre-defined set of "kind" images any more, and just need a flexible
> hierarchy of images that anyone can reuse and extend
>
> Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
>
>> From: Markus Thömmes <ma...@me.com>
>> To: dev@openwhisk.apache.org
>> Date: 10/03/2017 12:48 AM
>> Subject: Re: factoring out "runtimes" into deployment configuration
>>
>> What you're referring to is basically a "kind-aware" blackbox
>> action, where you get the easyness of injecting your code and the
>> flexibility of using whatever you want in the image as long as the
>> interface fits.
>>
>> I generally like the idea and brought it up as well once (i custom
>> built a blackbox to do something similar). Note though that the same
>> performance implications as with blackbox images apply.
>>
>> Von meinem iPhone gesendet
>>
>>> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
>>>
>>> With the "runtimeManifest" property I'm wondering if we can also
> expose the
>>> name of the container image instead of having to compute it in the
> code ?
>>> Extending the thought: what if we allow users to specify a custom
> container
>>> name when creating / updating actions, in case users want to ?
>>>
>>> I'm thinking at the use-case when a package with a group of actions
>>> requires multiple libraries for each action. To simplify my example
> I'm
>>> going to refer only to JS actions:
>>> - Today: when actions needs extra libs users have 2 options :
>>> a) create a zip
>>> b) browserify/"compile" JS into a single JS file
>>> - What I'm thinking: allow users to create their own "runtime" image
>>> where they can bundle those dependent libs into. All the actions in
> the
>>> package would be smaller, they would use less space and would probably
> flow
>>> easier through the system when being invoked. A potential problem in
> this
>>> case would be whether it's secure to let users push their own images
> in the
>>> OW nodes, but at the same time allowing them to upload a ZIP is
>>> conceptually also a "package"/"container" that can contain anything; I
>>> guess the security aspect would still remain even with user-defined
>>> containers.
>>>
>>> Dragos
>>>
>>>
>>>
>>>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
> wrote:
>>>>
>>>> Hi Rodric,
>>>>
>>>> I think that?s great.
>>>>
>>>> Only comment (more to Matt than you):
>>>> If the runtimes are dynamic by deployment (and of course over time)
> this
>>>> probably should be reflected in the packaging format spec [1].
>>>> (see also previous comment [2]:)
>>>>
>>>> - line 252: not sure if the list of runtime names should be in the
>>>> normative
>>>> section of a spec, because (as you also write) this list will
> fluctuate
>>>> between
>>>> deployments and over time. OTOH there should be established well
> known
>>>> names for
>>>> well-known runtimes. Maybe point to a community-curated, separate
> markdown
>>>> file
>>>> (where new entries get a PR that can be voted upon)?
>>>>
>>>> Cheers
>>>> Michael
>>>>
>>>> [1]
>>>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
>> specification/openwhisk_v0.8.pdf
>>>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>>>>
>>>>
>>>>
>>>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
>>>> rodric@gmail.com>> wrote:
>>>>
>>>> I've been thinking about how we can add more action runtimes more
> easily -
>>>> without changing the backend (API, controller, invoker, CLI). One way
> I'm
>>>> leaning toward and started prototyping to get a better understanding
> of the
>>>> feasibility is to strip all runtime types from the backend (and CLI)
> and
>>>> encode the information about what action runtimes the system supports
>>>> through configuration parameters.
>>>>
>>>> This will make it possible to decouple deploying the core components
> from
>>>> managing the action runtime images. An example of encoding the
> runtime
>>>> information in a deployment configuration is
>>>> https://github.com/openwhisk/openwhisk/pull/1948.
>>>>
>>>> In support of this general direction (even if we settle on a
> different
>>>> approach), I have increasingly flattened the "Exec" type hierarchy in
> the
>>>> backend[1-3]
>>>>
>>>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>>>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>>>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>>>
>>>> A sketch of how to use the runtime manifest can be seen in this WIP
> commit:
>>>>
>>>> https://github.com/rabbah/openwhisk/commit/
>> a8890abe51a3e7e6a34f96a46798de660583e36f
>>>>
>>>> I can see how using this we can add new versions of Node.js,
> python:3, and
>>>> other handy images by curating a list of these in a separate process.
>>>>
>>>> Feedback, as always, solicited and appreciated.
>>>>
>>>> -r
>>>>
>>>>
>>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
I can think of couple of ways to make what you called "kind-aware
blackbox" actions more developer-friendly:
1. Support them in the CLI (i.e., supporting both "image" and "code"
arguments). This would make the developer experience straightforard.
2. Generalize the caching/pooling mechanism to handle *popular* images,
rather than a hard-coded set of "native" kind images. This would eliminate
the performance overhead.
Regards,
Alex
P.S. Once we do #1 and #2 above, my claim is that we don't need
pre-defined set of "kind" images any more, and just need a flexible
hierarchy of images that anyone can reuse and extend
Markus Thömmes <ma...@me.com> wrote on 10/03/2017 12:48:17 AM:
> From: Markus Thömmes <ma...@me.com>
> To: dev@openwhisk.apache.org
> Date: 10/03/2017 12:48 AM
> Subject: Re: factoring out "runtimes" into deployment configuration
>
> What you're referring to is basically a "kind-aware" blackbox
> action, where you get the easyness of injecting your code and the
> flexibility of using whatever you want in the image as long as the
> interface fits.
>
> I generally like the idea and brought it up as well once (i custom
> built a blackbox to do something similar). Note though that the same
> performance implications as with blackbox images apply.
>
> Von meinem iPhone gesendet
>
> > Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
> >
> > With the "runtimeManifest" property I'm wondering if we can also
expose the
> > name of the container image instead of having to compute it in the
code ?
> > Extending the thought: what if we allow users to specify a custom
container
> > name when creating / updating actions, in case users want to ?
> >
> > I'm thinking at the use-case when a package with a group of actions
> > requires multiple libraries for each action. To simplify my example
I'm
> > going to refer only to JS actions:
> > - Today: when actions needs extra libs users have 2 options :
> > a) create a zip
> > b) browserify/"compile" JS into a single JS file
> > - What I'm thinking: allow users to create their own "runtime" image
> > where they can bundle those dependent libs into. All the actions in
the
> > package would be smaller, they would use less space and would probably
flow
> > easier through the system when being invoked. A potential problem in
this
> > case would be whether it's secure to let users push their own images
in the
> > OW nodes, but at the same time allowing them to upload a ZIP is
> > conceptually also a "package"/"container" that can contain anything; I
> > guess the security aspect would still remain even with user-defined
> > containers.
> >
> > Dragos
> >
> >
> >
> >> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com>
wrote:
> >>
> >> Hi Rodric,
> >>
> >> I think that?s great.
> >>
> >> Only comment (more to Matt than you):
> >> If the runtimes are dynamic by deployment (and of course over time)
this
> >> probably should be reflected in the packaging format spec [1].
> >> (see also previous comment [2]:)
> >>
> >> - line 252: not sure if the list of runtime names should be in the
> >> normative
> >> section of a spec, because (as you also write) this list will
fluctuate
> >> between
> >> deployments and over time. OTOH there should be established well
known
> >> names for
> >> well-known runtimes. Maybe point to a community-curated, separate
markdown
> >> file
> >> (where new entries get a PR that can be voted upon)?
> >>
> >> Cheers
> >> Michael
> >>
> >> [1]
> >> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/
> specification/openwhisk_v0.8.pdf
> >> [2] http://markmail.org/message/pa35wxxl52tvbfxc
> >>
> >>
> >>
> >> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
> >> rodric@gmail.com>> wrote:
> >>
> >> I've been thinking about how we can add more action runtimes more
easily -
> >> without changing the backend (API, controller, invoker, CLI). One way
I'm
> >> leaning toward and started prototyping to get a better understanding
of the
> >> feasibility is to strip all runtime types from the backend (and CLI)
and
> >> encode the information about what action runtimes the system supports
> >> through configuration parameters.
> >>
> >> This will make it possible to decouple deploying the core components
from
> >> managing the action runtime images. An example of encoding the
runtime
> >> information in a deployment configuration is
> >> https://github.com/openwhisk/openwhisk/pull/1948.
> >>
> >> In support of this general direction (even if we settle on a
different
> >> approach), I have increasingly flattened the "Exec" type hierarchy in
the
> >> backend[1-3]
> >>
> >> [1] https://github.com/openwhisk/openwhisk/pull/1938
> >> [2] https://github.com/openwhisk/openwhisk/pull/1942
> >> [3] https://github.com/openwhisk/openwhisk/pull/1911
> >>
> >> A sketch of how to use the runtime manifest can be seen in this WIP
commit:
> >>
> >> https://github.com/rabbah/openwhisk/commit/
> a8890abe51a3e7e6a34f96a46798de660583e36f
> >>
> >> I can see how using this we can add new versions of Node.js,
python:3, and
> >> other handy images by curating a list of these in a separate process.
> >>
> >> Feedback, as always, solicited and appreciated.
> >>
> >> -r
> >>
> >>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Markus Thömmes <ma...@me.com>.
What you're referring to is basically a "kind-aware" blackbox action, where you get the easyness of injecting your code and the flexibility of using whatever you want in the image as long as the interface fits.
I generally like the idea and brought it up as well once (i custom built a blackbox to do something similar). Note though that the same performance implications as with blackbox images apply.
Von meinem iPhone gesendet
> Am 09.03.2017 um 23:43 schrieb Dascalita Dragos <dd...@gmail.com>:
>
> With the "runtimeManifest" property I'm wondering if we can also expose the
> name of the container image instead of having to compute it in the code ?
> Extending the thought: what if we allow users to specify a custom container
> name when creating / updating actions, in case users want to ?
>
> I'm thinking at the use-case when a package with a group of actions
> requires multiple libraries for each action. To simplify my example I'm
> going to refer only to JS actions:
> - Today: when actions needs extra libs users have 2 options :
> a) create a zip
> b) browserify/"compile" JS into a single JS file
> - What I'm thinking: allow users to create their own "runtime" image
> where they can bundle those dependent libs into. All the actions in the
> package would be smaller, they would use less space and would probably flow
> easier through the system when being invoked. A potential problem in this
> case would be whether it's secure to let users push their own images in the
> OW nodes, but at the same time allowing them to upload a ZIP is
> conceptually also a "package"/"container" that can contain anything; I
> guess the security aspect would still remain even with user-defined
> containers.
>
> Dragos
>
>
>
>> On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com> wrote:
>>
>> Hi Rodric,
>>
>> I think that’s great.
>>
>> Only comment (more to Matt than you):
>> If the runtimes are dynamic by deployment (and of course over time) this
>> probably should be reflected in the packaging format spec [1].
>> (see also previous comment [2]:)
>>
>> - line 252: not sure if the list of runtime names should be in the
>> normative
>> section of a spec, because (as you also write) this list will fluctuate
>> between
>> deployments and over time. OTOH there should be established well known
>> names for
>> well-known runtimes. Maybe point to a community-curated, separate markdown
>> file
>> (where new entries get a PR that can be voted upon)?
>>
>> Cheers
>> Michael
>>
>> [1]
>> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/specification/openwhisk_v0.8.pdf
>> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>>
>>
>>
>> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
>> rodric@gmail.com>> wrote:
>>
>> I've been thinking about how we can add more action runtimes more easily -
>> without changing the backend (API, controller, invoker, CLI). One way I'm
>> leaning toward and started prototyping to get a better understanding of the
>> feasibility is to strip all runtime types from the backend (and CLI) and
>> encode the information about what action runtimes the system supports
>> through configuration parameters.
>>
>> This will make it possible to decouple deploying the core components from
>> managing the action runtime images. An example of encoding the runtime
>> information in a deployment configuration is
>> https://github.com/openwhisk/openwhisk/pull/1948.
>>
>> In support of this general direction (even if we settle on a different
>> approach), I have increasingly flattened the "Exec" type hierarchy in the
>> backend[1-3]
>>
>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>
>> A sketch of how to use the runtime manifest can be seen in this WIP commit:
>>
>> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>>
>> I can see how using this we can add new versions of Node.js, python:3, and
>> other handy images by curating a list of these in a separate process.
>>
>> Feedback, as always, solicited and appreciated.
>>
>> -r
>>
>>
Re: factoring out "runtimes" into deployment configuration
Posted by Dascalita Dragos <dd...@gmail.com>.
With the "runtimeManifest" property I'm wondering if we can also expose the
name of the container image instead of having to compute it in the code ?
Extending the thought: what if we allow users to specify a custom container
name when creating / updating actions, in case users want to ?
I'm thinking at the use-case when a package with a group of actions
requires multiple libraries for each action. To simplify my example I'm
going to refer only to JS actions:
- Today: when actions needs extra libs users have 2 options :
a) create a zip
b) browserify/"compile" JS into a single JS file
- What I'm thinking: allow users to create their own "runtime" image
where they can bundle those dependent libs into. All the actions in the
package would be smaller, they would use less space and would probably flow
easier through the system when being invoked. A potential problem in this
case would be whether it's secure to let users push their own images in the
OW nodes, but at the same time allowing them to upload a ZIP is
conceptually also a "package"/"container" that can contain anything; I
guess the security aspect would still remain even with user-defined
containers.
Dragos
On Tue, Mar 7, 2017 at 4:20 AM Michael Marth <mm...@adobe.com> wrote:
> Hi Rodric,
>
> I think that’s great.
>
> Only comment (more to Matt than you):
> If the runtimes are dynamic by deployment (and of course over time) this
> probably should be reflected in the packaging format spec [1].
> (see also previous comment [2]:)
>
> - line 252: not sure if the list of runtime names should be in the
> normative
> section of a spec, because (as you also write) this list will fluctuate
> between
> deployments and over time. OTOH there should be established well known
> names for
> well-known runtimes. Maybe point to a community-curated, separate markdown
> file
> (where new entries get a PR that can be voted upon)?
>
> Cheers
> Michael
>
> [1]
> https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/specification/openwhisk_v0.8.pdf
> [2] http://markmail.org/message/pa35wxxl52tvbfxc
>
>
>
> On 05/03/17 17:39, "Rodric Rabbah" <rodric@gmail.com<mailto:
> rodric@gmail.com>> wrote:
>
> I've been thinking about how we can add more action runtimes more easily -
> without changing the backend (API, controller, invoker, CLI). One way I'm
> leaning toward and started prototyping to get a better understanding of the
> feasibility is to strip all runtime types from the backend (and CLI) and
> encode the information about what action runtimes the system supports
> through configuration parameters.
>
> This will make it possible to decouple deploying the core components from
> managing the action runtime images. An example of encoding the runtime
> information in a deployment configuration is
> https://github.com/openwhisk/openwhisk/pull/1948.
>
> In support of this general direction (even if we settle on a different
> approach), I have increasingly flattened the "Exec" type hierarchy in the
> backend[1-3]
>
> [1] https://github.com/openwhisk/openwhisk/pull/1938
> [2] https://github.com/openwhisk/openwhisk/pull/1942
> [3] https://github.com/openwhisk/openwhisk/pull/1911
>
> A sketch of how to use the runtime manifest can be seen in this WIP commit:
>
> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
> I can see how using this we can add new versions of Node.js, python:3, and
> other handy images by curating a list of these in a separate process.
>
> Feedback, as always, solicited and appreciated.
>
> -r
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Michael Marth <mm...@adobe.com>.
Hi Rodric,
I think that’s great.
Only comment (more to Matt than you):
If the runtimes are dynamic by deployment (and of course over time) this probably should be reflected in the packaging format spec [1].
(see also previous comment [2]:)
- line 252: not sure if the list of runtime names should be in the normative
section of a spec, because (as you also write) this list will fluctuate between
deployments and over time. OTOH there should be established well known names for
well-known runtimes. Maybe point to a community-curated, separate markdown file
(where new entries get a PR that can be voted upon)?
Cheers
Michael
[1] https://github.com/openwhisk/openwhisk-wskdeploy/blob/master/specification/openwhisk_v0.8.pdf
[2] http://markmail.org/message/pa35wxxl52tvbfxc
On 05/03/17 17:39, "Rodric Rabbah" <ro...@gmail.com>> wrote:
I've been thinking about how we can add more action runtimes more easily -
without changing the backend (API, controller, invoker, CLI). One way I'm
leaning toward and started prototyping to get a better understanding of the
feasibility is to strip all runtime types from the backend (and CLI) and
encode the information about what action runtimes the system supports
through configuration parameters.
This will make it possible to decouple deploying the core components from
managing the action runtime images. An example of encoding the runtime
information in a deployment configuration is
https://github.com/openwhisk/openwhisk/pull/1948.
In support of this general direction (even if we settle on a different
approach), I have increasingly flattened the "Exec" type hierarchy in the
backend[1-3]
[1] https://github.com/openwhisk/openwhisk/pull/1938
[2] https://github.com/openwhisk/openwhisk/pull/1942
[3] https://github.com/openwhisk/openwhisk/pull/1911
A sketch of how to use the runtime manifest can be seen in this WIP commit:
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
I can see how using this we can add new versions of Node.js, python:3, and
other handy images by curating a list of these in a separate process.
Feedback, as always, solicited and appreciated.
-r
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
Note further that for actions that aren't intrinsic that making is isomorphic with these proposed changes.
-r
> On Mar 5, 2017, at 2:52 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> Good direction!
>
> How about taking this one step further and removing "kind" from the API
> altogether, leaving just "image", "code" (text or binary) and "main"? We
> could manage the kind-->image mapping on the client side.
>
> Regards,
> Alex
>
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.incubator.apache.org
> Date: 05/03/2017 06:39 PM
> Subject: factoring out "runtimes" into deployment configuration
>
>
>
> I've been thinking about how we can add more action runtimes more easily -
> without changing the backend (API, controller, invoker, CLI). One way I'm
> leaning toward and started prototyping to get a better understanding of
> the
> feasibility is to strip all runtime types from the backend (and CLI) and
> encode the information about what action runtimes the system supports
> through configuration parameters.
>
> This will make it possible to decouple deploying the core components from
> managing the action runtime images. An example of encoding the runtime
> information in a deployment configuration is
> https://github.com/openwhisk/openwhisk/pull/1948.
>
> In support of this general direction (even if we settle on a different
> approach), I have increasingly flattened the "Exec" type hierarchy in the
> backend[1-3]
>
> [1] https://github.com/openwhisk/openwhisk/pull/1938
> [2] https://github.com/openwhisk/openwhisk/pull/1942
> [3] https://github.com/openwhisk/openwhisk/pull/1911
>
> A sketch of how to use the runtime manifest can be seen in this WIP
> commit:
> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
>
> I can see how using this we can add new versions of Node.js, python:3, and
> other handy images by curating a list of these in a separate process.
>
> Feedback, as always, solicited and appreciated.
>
> -r
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
So, maybe we just need two "kinds" at the moment - 'regular' action (with
image and code, plus whatever 'syntactic sugar' we want at the client
side) and 'sequence' (with 'components').
Excluding sequences, what else is controller currently doing with 'kind'
besides selecting the image? Now that 'blackbox' actions can be used to
run 'native' language actions equally (?) well, it feels redundant to have
two args in the API used for the same purpose.
Regards,
Alex
===========================================================================
Alex Glikson
Senior Researcher, Cloud Platforms, IBM Research - Haifa; IBM Lead for
FIWARE
Email: glikson@il.ibm.com | Skype: alex.glikson | Phone: +972-4-8281085 |
Mobile: +972-54-6466667
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.apache.org
Date: 05/03/2017 10:54 PM
Subject: Re: factoring out "runtimes" into deployment configuration
The sequence doesn't run in a container. It's more an continuation style
activation. Its components do, of course. So abstractly it's an action but
there is no code to execute in a container for the sequence itself.
-r
> On Mar 5, 2017, at 3:41 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> Don't (or at least can't) they run in Docker containers too?
>
> Alex
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 05/03/2017 09:59 PM
> Subject: Re: factoring out "runtimes" into deployment
configuration
>
>
>
> I considered that but it would be at odds with actions that have no
images
> - in particular sequences and other combinators in the future.
>
> -r
>
>> On Mar 5, 2017, at 2:52 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>>
>> Good direction!
>>
>> How about taking this one step further and removing "kind" from the API
>> altogether, leaving just "image", "code" (text or binary) and "main"?
We
>
>> could manage the kind-->image mapping on the client side.
>>
>> Regards,
>> Alex
>>
>>
>>
>>
>> From: Rodric Rabbah <ro...@gmail.com>
>> To: dev@openwhisk.incubator.apache.org
>> Date: 05/03/2017 06:39 PM
>> Subject: factoring out "runtimes" into deployment configuration
>>
>>
>>
>> I've been thinking about how we can add more action runtimes more
easily
> -
>> without changing the backend (API, controller, invoker, CLI). One way
> I'm
>> leaning toward and started prototyping to get a better understanding of
>> the
>> feasibility is to strip all runtime types from the backend (and CLI)
and
>> encode the information about what action runtimes the system supports
>> through configuration parameters.
>>
>> This will make it possible to decouple deploying the core components
> from
>> managing the action runtime images. An example of encoding the runtime
>> information in a deployment configuration is
>> https://github.com/openwhisk/openwhisk/pull/1948.
>>
>> In support of this general direction (even if we settle on a different
>> approach), I have increasingly flattened the "Exec" type hierarchy in
> the
>> backend[1-3]
>>
>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>
>> A sketch of how to use the runtime manifest can be seen in this WIP
>> commit:
>>
>
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
>>
>>
>> I can see how using this we can add new versions of Node.js, python:3,
> and
>> other handy images by curating a list of these in a separate process.
>>
>> Feedback, as always, solicited and appreciated.
>>
>> -r
>>
>>
>>
>>
>
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
The sequence doesn't run in a container. It's more an continuation style activation. Its components do, of course. So abstractly it's an action but there is no code to execute in a container for the sequence itself.
-r
> On Mar 5, 2017, at 3:41 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> Don't (or at least can't) they run in Docker containers too?
>
> Alex
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 05/03/2017 09:59 PM
> Subject: Re: factoring out "runtimes" into deployment configuration
>
>
>
> I considered that but it would be at odds with actions that have no images
> - in particular sequences and other combinators in the future.
>
> -r
>
>> On Mar 5, 2017, at 2:52 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>>
>> Good direction!
>>
>> How about taking this one step further and removing "kind" from the API
>> altogether, leaving just "image", "code" (text or binary) and "main"? We
>
>> could manage the kind-->image mapping on the client side.
>>
>> Regards,
>> Alex
>>
>>
>>
>>
>> From: Rodric Rabbah <ro...@gmail.com>
>> To: dev@openwhisk.incubator.apache.org
>> Date: 05/03/2017 06:39 PM
>> Subject: factoring out "runtimes" into deployment configuration
>>
>>
>>
>> I've been thinking about how we can add more action runtimes more easily
> -
>> without changing the backend (API, controller, invoker, CLI). One way
> I'm
>> leaning toward and started prototyping to get a better understanding of
>> the
>> feasibility is to strip all runtime types from the backend (and CLI) and
>> encode the information about what action runtimes the system supports
>> through configuration parameters.
>>
>> This will make it possible to decouple deploying the core components
> from
>> managing the action runtime images. An example of encoding the runtime
>> information in a deployment configuration is
>> https://github.com/openwhisk/openwhisk/pull/1948.
>>
>> In support of this general direction (even if we settle on a different
>> approach), I have increasingly flattened the "Exec" type hierarchy in
> the
>> backend[1-3]
>>
>> [1] https://github.com/openwhisk/openwhisk/pull/1938
>> [2] https://github.com/openwhisk/openwhisk/pull/1942
>> [3] https://github.com/openwhisk/openwhisk/pull/1911
>>
>> A sketch of how to use the runtime manifest can be seen in this WIP
>> commit:
>>
> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
>>
>>
>> I can see how using this we can add new versions of Node.js, python:3,
> and
>> other handy images by curating a list of these in a separate process.
>>
>> Feedback, as always, solicited and appreciated.
>>
>> -r
>>
>>
>>
>>
>
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
Don't (or at least can't) they run in Docker containers too?
Alex
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.apache.org
Date: 05/03/2017 09:59 PM
Subject: Re: factoring out "runtimes" into deployment configuration
I considered that but it would be at odds with actions that have no images
- in particular sequences and other combinators in the future.
-r
> On Mar 5, 2017, at 2:52 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> Good direction!
>
> How about taking this one step further and removing "kind" from the API
> altogether, leaving just "image", "code" (text or binary) and "main"? We
> could manage the kind-->image mapping on the client side.
>
> Regards,
> Alex
>
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.incubator.apache.org
> Date: 05/03/2017 06:39 PM
> Subject: factoring out "runtimes" into deployment configuration
>
>
>
> I've been thinking about how we can add more action runtimes more easily
-
> without changing the backend (API, controller, invoker, CLI). One way
I'm
> leaning toward and started prototyping to get a better understanding of
> the
> feasibility is to strip all runtime types from the backend (and CLI) and
> encode the information about what action runtimes the system supports
> through configuration parameters.
>
> This will make it possible to decouple deploying the core components
from
> managing the action runtime images. An example of encoding the runtime
> information in a deployment configuration is
> https://github.com/openwhisk/openwhisk/pull/1948.
>
> In support of this general direction (even if we settle on a different
> approach), I have increasingly flattened the "Exec" type hierarchy in
the
> backend[1-3]
>
> [1] https://github.com/openwhisk/openwhisk/pull/1938
> [2] https://github.com/openwhisk/openwhisk/pull/1942
> [3] https://github.com/openwhisk/openwhisk/pull/1911
>
> A sketch of how to use the runtime manifest can be seen in this WIP
> commit:
>
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
>
> I can see how using this we can add new versions of Node.js, python:3,
and
> other handy images by curating a list of these in a separate process.
>
> Feedback, as always, solicited and appreciated.
>
> -r
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Rodric Rabbah <ro...@gmail.com>.
I considered that but it would be at odds with actions that have no images - in particular sequences and other combinators in the future.
-r
> On Mar 5, 2017, at 2:52 PM, Alex Glikson <GL...@il.ibm.com> wrote:
>
> Good direction!
>
> How about taking this one step further and removing "kind" from the API
> altogether, leaving just "image", "code" (text or binary) and "main"? We
> could manage the kind-->image mapping on the client side.
>
> Regards,
> Alex
>
>
>
>
> From: Rodric Rabbah <ro...@gmail.com>
> To: dev@openwhisk.incubator.apache.org
> Date: 05/03/2017 06:39 PM
> Subject: factoring out "runtimes" into deployment configuration
>
>
>
> I've been thinking about how we can add more action runtimes more easily -
> without changing the backend (API, controller, invoker, CLI). One way I'm
> leaning toward and started prototyping to get a better understanding of
> the
> feasibility is to strip all runtime types from the backend (and CLI) and
> encode the information about what action runtimes the system supports
> through configuration parameters.
>
> This will make it possible to decouple deploying the core components from
> managing the action runtime images. An example of encoding the runtime
> information in a deployment configuration is
> https://github.com/openwhisk/openwhisk/pull/1948.
>
> In support of this general direction (even if we settle on a different
> approach), I have increasingly flattened the "Exec" type hierarchy in the
> backend[1-3]
>
> [1] https://github.com/openwhisk/openwhisk/pull/1938
> [2] https://github.com/openwhisk/openwhisk/pull/1942
> [3] https://github.com/openwhisk/openwhisk/pull/1911
>
> A sketch of how to use the runtime manifest can be seen in this WIP
> commit:
> https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
>
>
> I can see how using this we can add new versions of Node.js, python:3, and
> other handy images by curating a list of these in a separate process.
>
> Feedback, as always, solicited and appreciated.
>
> -r
>
>
>
>
Re: factoring out "runtimes" into deployment configuration
Posted by Alex Glikson <GL...@il.ibm.com>.
Good direction!
How about taking this one step further and removing "kind" from the API
altogether, leaving just "image", "code" (text or binary) and "main"? We
could manage the kind-->image mapping on the client side.
Regards,
Alex
From: Rodric Rabbah <ro...@gmail.com>
To: dev@openwhisk.incubator.apache.org
Date: 05/03/2017 06:39 PM
Subject: factoring out "runtimes" into deployment configuration
I've been thinking about how we can add more action runtimes more easily -
without changing the backend (API, controller, invoker, CLI). One way I'm
leaning toward and started prototyping to get a better understanding of
the
feasibility is to strip all runtime types from the backend (and CLI) and
encode the information about what action runtimes the system supports
through configuration parameters.
This will make it possible to decouple deploying the core components from
managing the action runtime images. An example of encoding the runtime
information in a deployment configuration is
https://github.com/openwhisk/openwhisk/pull/1948.
In support of this general direction (even if we settle on a different
approach), I have increasingly flattened the "Exec" type hierarchy in the
backend[1-3]
[1] https://github.com/openwhisk/openwhisk/pull/1938
[2] https://github.com/openwhisk/openwhisk/pull/1942
[3] https://github.com/openwhisk/openwhisk/pull/1911
A sketch of how to use the runtime manifest can be seen in this WIP
commit:
https://github.com/rabbah/openwhisk/commit/a8890abe51a3e7e6a34f96a46798de660583e36f
I can see how using this we can add new versions of Node.js, python:3, and
other handy images by curating a list of these in a separate process.
Feedback, as always, solicited and appreciated.
-r