You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <op...@sciabarra.com> on 2018/04/15 06:20:03 UTC
Proposal: offline compilation support for Swift, Go and more...
A brief recap: I implemented support for Golang actions, and adapted support to work also with Swift, using another "high performance" protocol. The support actually works for any binary that follow a certain "protocol", so it could be used for programs written in c, rust, haskell, whatever.
For convenience the proxy also supports zip and sources. Source support works compiling an action on the fly, thus producing a binary when you init a runtime. Source support is language specific, so I have now an image able to compile golang actions and another able to compile swift actions. More can be provided (and it would be interesting to do).
All of this has been already coded and tested, I submitted the PR waiting for review.
Now my proposal. Source support has a problem: slow initialization. Source support is very convenient in development but when the application scale, re-initializing more runtimes using source is slow. So you have a warm-up problem.
This problem can be solved precompiling the images but it was considered a complex problem to solve. Except I realized is not! Having built source support in the runtimes, it is a small step to adapt the runtimes to became also offline compilers!
Most notably my idea is this: assuming you have the sources in folder like ./src, you can generate the executable using something like:
$ docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-golang compile main.go main
Translated:
1. run the runtime as a compiler invoking a 'compile' in the image
2. pass the source and the target folder as volumes to the image
3. run the compilation (compile will expect sources to be in /src and place the final executable in /out)
This way you will be able to generate the binaries offline without having to do a complex setup.
Then you can deploy the binary with
$ wsk action create myaction out/main --native
Processing sources for generating binaries can be done by some processing tools, at some point can be even integrated in wskdeploy or provided by some preprocessing. That is not the biggest problem, once you have support in place on the runtimes.
If you agree on this proposal I can go on implementing AND DOCUMENTING them for the images I am building, the go image and the swift image.
Thoughts?
--
Michele Sciabarra
openwhisk@sciabarra.com
Re: Proposal: offline compilation support for Swift, Go and more...
Posted by Michele Sciabarra <mi...@sciabarra.com>.
Ok I did it,
it works now both for swift and go.
Using the newly created images , if you have a swift action in src/hello.swift you do
docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-swift-v4.1 compile
and you will get the binary in out/main
Also with go actions it works the same: if you have a src/hello.go running
docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-golang-v1.9 compile
and you the the binary in out/main
Basically, there is a "compile" command that defaults reading input in /src and producing output in /out
I implemented it both for swift and go
You can specify mode parameters:
docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-golang-v1.9 compile hello
to use an "Hello" function,
docker run -v $PWD/src:/source -v $PWD/out:/output actionloop-golang-v1.9 compile hello /source /output
if you want to use different directories.
Code is in:
https://github.com/sciabarracom/incubator-openwhisk-runtime-go
https://github.com/sciabarracom/incubator-openwhisk-runtime-swift
I submitted PRs both to go and swift.
--
Michele Sciabarra
michele@sciabarra.com
On Sun, Apr 15, 2018, at 4:30 PM, Carlos Santana wrote:
> Approach and direction sounds good
>
> If you read the docs for Swift on how to create a pre-complied binary
> and zip is the same process we tell user to use the swift runtime image
> and use the compile.sh to produce the zip with a single command running
> inside the image nothing to install
>
> [1]
> https://github.com/apache/incubator-openwhisk/blob/master/docs/actions.md#packaging-an-action-as-a-swift-executable
>
>
> - Carlos Santana
> @csantanapr
>
> > On Apr 15, 2018, at 2:20 AM, Michele Sciabarra <op...@sciabarra.com> wrote:
> >
> > A brief recap: I implemented support for Golang actions, and adapted support to work also with Swift, using another "high performance" protocol. The support actually works for any binary that follow a certain "protocol", so it could be used for programs written in c, rust, haskell, whatever.
> >
> > For convenience the proxy also supports zip and sources. Source support works compiling an action on the fly, thus producing a binary when you init a runtime. Source support is language specific, so I have now an image able to compile golang actions and another able to compile swift actions. More can be provided (and it would be interesting to do).
> >
> > All of this has been already coded and tested, I submitted the PR waiting for review.
> >
> > Now my proposal. Source support has a problem: slow initialization. Source support is very convenient in development but when the application scale, re-initializing more runtimes using source is slow. So you have a warm-up problem.
> >
> > This problem can be solved precompiling the images but it was considered a complex problem to solve. Except I realized is not! Having built source support in the runtimes, it is a small step to adapt the runtimes to became also offline compilers!
> >
> > Most notably my idea is this: assuming you have the sources in folder like ./src, you can generate the executable using something like:
> >
> > $ docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-golang compile main.go main
> >
> > Translated:
> > 1. run the runtime as a compiler invoking a 'compile' in the image
> > 2. pass the source and the target folder as volumes to the image
> > 3. run the compilation (compile will expect sources to be in /src and place the final executable in /out)
> >
> > This way you will be able to generate the binaries offline without having to do a complex setup.
> > Then you can deploy the binary with
> >
> > $ wsk action create myaction out/main --native
> >
> > Processing sources for generating binaries can be done by some processing tools, at some point can be even integrated in wskdeploy or provided by some preprocessing. That is not the biggest problem, once you have support in place on the runtimes.
> >
> > If you agree on this proposal I can go on implementing AND DOCUMENTING them for the images I am building, the go image and the swift image.
> >
> > Thoughts?
> >
> >
> > --
> > Michele Sciabarra
> > openwhisk@sciabarra.com
Re: Proposal: offline compilation support for Swift, Go and more...
Posted by Carlos Santana <cs...@gmail.com>.
Approach and direction sounds good
If you read the docs for Swift on how to create a pre-complied binary and zip is the same process we tell user to use the swift runtime image and use the compile.sh to produce the zip with a single command running inside the image nothing to install
[1] https://github.com/apache/incubator-openwhisk/blob/master/docs/actions.md#packaging-an-action-as-a-swift-executable
- Carlos Santana
@csantanapr
> On Apr 15, 2018, at 2:20 AM, Michele Sciabarra <op...@sciabarra.com> wrote:
>
> A brief recap: I implemented support for Golang actions, and adapted support to work also with Swift, using another "high performance" protocol. The support actually works for any binary that follow a certain "protocol", so it could be used for programs written in c, rust, haskell, whatever.
>
> For convenience the proxy also supports zip and sources. Source support works compiling an action on the fly, thus producing a binary when you init a runtime. Source support is language specific, so I have now an image able to compile golang actions and another able to compile swift actions. More can be provided (and it would be interesting to do).
>
> All of this has been already coded and tested, I submitted the PR waiting for review.
>
> Now my proposal. Source support has a problem: slow initialization. Source support is very convenient in development but when the application scale, re-initializing more runtimes using source is slow. So you have a warm-up problem.
>
> This problem can be solved precompiling the images but it was considered a complex problem to solve. Except I realized is not! Having built source support in the runtimes, it is a small step to adapt the runtimes to became also offline compilers!
>
> Most notably my idea is this: assuming you have the sources in folder like ./src, you can generate the executable using something like:
>
> $ docker run -v $PWD/src:/src -v $PWD/out:/out actionloop-golang compile main.go main
>
> Translated:
> 1. run the runtime as a compiler invoking a 'compile' in the image
> 2. pass the source and the target folder as volumes to the image
> 3. run the compilation (compile will expect sources to be in /src and place the final executable in /out)
>
> This way you will be able to generate the binaries offline without having to do a complex setup.
> Then you can deploy the binary with
>
> $ wsk action create myaction out/main --native
>
> Processing sources for generating binaries can be done by some processing tools, at some point can be even integrated in wskdeploy or provided by some preprocessing. That is not the biggest problem, once you have support in place on the runtimes.
>
> If you agree on this proposal I can go on implementing AND DOCUMENTING them for the images I am building, the go image and the swift image.
>
> Thoughts?
>
>
> --
> Michele Sciabarra
> openwhisk@sciabarra.com