You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <op...@sciabarra.com> on 2018/02/24 18:58:04 UTC
Implementation plan for Go actions
Hello, whiskers
I am at the risk of expanding too much the scope of what I am trying to do, so I want to try to focus and work step by step.
So I am writing down my plan for review.
# Step one: implement a good support for GoLang, with piped standard input and output, and log in stderr.
Because there is not a past support for Go, except the generic docker support, I think I have some freedom here.
The support will then work this way (please review carefully):
## /init
The runtime expects posted in /init the usual
{
"binary": ...
"code": ...
"main": ..
}
where "code" is either a zip file (with binary: true) or a source code (with binary: false or absent).
If it source code, it expects to be a go source.
It places in /action/main/exec.go and compiles it with go build -o /action/exec /action/main/exec.go, getting a /action/exec to execute.
If is is a zip file, it will unzip it in /action. If there is a /action/exec file in the zip it will run it.
If there is not, it will look if there is a "main" parameter, getting the <main-value>.
It will compile a /action/main/<main-value>.go. If the <main-value> is missing, it will default to /action/main/exec.go.
Note the zip can include other source files that will be stored in /action/<folders>. So I will have to use GOPATH=/action to find them.
## /run
The /run behaviour is:
At every request, if no action process is running,it will start one and will pass the first line as an argument and also in standard output.
If the executable does not terminate, it will keep feeding more input in standard input, expecting output in standard output (line by line) and logs in stderr.
The server will have to collect stderr logs and write them in docker stdout, and terminate the output with the markers when an action completes:
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
It the executable DOES terminate, it will execute again at the next /run request
## Step 2: need to implement the same logic for swift actions (we have to keep the existing compilation infrastructure)
Issues to solve here are the compilation and the "warm" startup problem. I have some ideas but... I do better to discuss them once the Go support is finished.
## Step 3: will discuss with the list if it is worth to extend the support to the generic dockerskeleton, that is based on the python proxy.
There are compatibility issues, because the current docker support logs everything in stdout and the last line is the answer, while for using the pipe you need to do separate log.
How does it sound?
Suggested or required changes?
Am I missing something?
Approved?
--
Michele Sciabarra
openwhisk@sciabarra.com
Re: Implementation plan for Go actions
Posted by Michele Sciabarra <op...@sciabarra.com>.
> > Michele
> >
> > Sorry about that, I think I got confused the exec in code with the ability
> > to uploading a string/executable file like a shell/bash script or perl
> > script.
> I know that if binary is false then it treats as a script and if it is
> true as a zip file.
> Problem is with Go actions an executable and a zip looks the same...
>
> >
> > I think we can take a todo after implementing exec inside zip and add
> > improvements.
> >
> > We should be able at the proxy level when binary=true (base64) distinguish
> > the `code` by parsing the first bytes and tell if it's a zip, tgz, or
> > executable related [1]
> Great! It is exactly what I already implemented in the Go runtime.
>
> it is based on this library: https://github.com/h2non/filetype
>
> At the moment I am busy trying to make things robust and write unit tests...
Ok just committed the implementation that is able to distinguish a binary in elf format from a binary in zip format and handle it properly. Including unit tests :)
Here is the code:
https://github.com/sciabarracom/openwhisk-runtime-go/blob/pipe-loop/openwhisk/extractor.go
https://github.com/sciabarracom/openwhisk-runtime-go/blob/pipe-loop/openwhisk/extractor_test.go
Re: Implementation plan for Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
On Tue, Feb 27, 2018, at 7:56 PM, Carlos Santana wrote:
> Michele
>
> Sorry about that, I think I got confused the exec in code with the ability
> to uploading a string/executable file like a shell/bash script or perl
> script.
I know that if binary is false then it treats as a script and if it is true as a zip file.
Problem is with Go actions an executable and a zip looks the same...
>
> I think we can take a todo after implementing exec inside zip and add
> improvements.
>
> We should be able at the proxy level when binary=true (base64) distinguish
> the `code` by parsing the first bytes and tell if it's a zip, tgz, or
> executable related [1]
Great! It is exactly what I already implemented in the Go runtime.
it is based on this library: https://github.com/h2non/filetype
At the moment I am busy trying to make things robust and write unit tests...
> That way support just uploading a exec without the extra step to zip, or
> even support tgz which have come up for other runtimes like nodejs8 [1]
>
> [1] https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/16
> [2] https://github.com/apache/incubator-openwhisk-runtime-nodejs/issues/15
>
> -- Carlos
>
>
> On Tue, Feb 27, 2018 at 8:47 AM Michele Sciabarra <mi...@sciabarra.com>
> wrote:
>
> > On Mon, Feb 26, 2018, at 6:06 PM, Carlos Santana wrote:
> > > Hi Michele
> > >
> > > >There is only one "annoyance" here: I noticed the dockerskeleton always
> > > expects a zip file including an "exec"
> > >
> > > dockerskeleton suppports adding an exec straight, it will take the file
> > do
> > > a chmod +x and executed.
> > >
> > > So not sure what you are referring here as this is possible, maybe I miss
> > > understood
> >
> > I use this script to post:
> > --- init.sh ---
> > #!/bin/bash
> > FILE=${1:?file}
> > echo '{"value":{"binary":true,"code":"'$(base64 $FILE)'"}}' >$FILE.json
> > curl -XPOST http://localhost:${PORT:-8080}/init -d @$FILE.json
> > ----
> >
> > exec is the compilation of this sample:
> > http://jamesthom.as/blog/2016/06/21/serverless-go-actions/
> >
> > I launch the docker skeleton:
> >
> > docker -p 8080:8080 -ti openwhisk/dockerskeleton
> >
> > If I post it as is:
> >
> > ./init.sh exec
> > I get:
> > err File is not a zip file
> >
> > if I zip it:
> >
> > zip exec.zip exec
> > ./init.sh exec.zip
> >
> > it works
> >
> > Where am I wrong?
> >
> >
Re: Implementation plan for Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Michele
Sorry about that, I think I got confused the exec in code with the ability
to uploading a string/executable file like a shell/bash script or perl
script.
I think we can take a todo after implementing exec inside zip and add
improvements.
We should be able at the proxy level when binary=true (base64) distinguish
the `code` by parsing the first bytes and tell if it's a zip, tgz, or
executable related [1]
That way support just uploading a exec without the extra step to zip, or
even support tgz which have come up for other runtimes like nodejs8 [1]
[1] https://github.com/apache/incubator-openwhisk-runtime-nodejs/pull/16
[2] https://github.com/apache/incubator-openwhisk-runtime-nodejs/issues/15
-- Carlos
On Tue, Feb 27, 2018 at 8:47 AM Michele Sciabarra <mi...@sciabarra.com>
wrote:
> On Mon, Feb 26, 2018, at 6:06 PM, Carlos Santana wrote:
> > Hi Michele
> >
> > >There is only one "annoyance" here: I noticed the dockerskeleton always
> > expects a zip file including an "exec"
> >
> > dockerskeleton suppports adding an exec straight, it will take the file
> do
> > a chmod +x and executed.
> >
> > So not sure what you are referring here as this is possible, maybe I miss
> > understood
>
> I use this script to post:
> --- init.sh ---
> #!/bin/bash
> FILE=${1:?file}
> echo '{"value":{"binary":true,"code":"'$(base64 $FILE)'"}}' >$FILE.json
> curl -XPOST http://localhost:${PORT:-8080}/init -d @$FILE.json
> ----
>
> exec is the compilation of this sample:
> http://jamesthom.as/blog/2016/06/21/serverless-go-actions/
>
> I launch the docker skeleton:
>
> docker -p 8080:8080 -ti openwhisk/dockerskeleton
>
> If I post it as is:
>
> ./init.sh exec
> I get:
> err File is not a zip file
>
> if I zip it:
>
> zip exec.zip exec
> ./init.sh exec.zip
>
> it works
>
> Where am I wrong?
>
>
Re: Implementation plan for Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
On Mon, Feb 26, 2018, at 6:06 PM, Carlos Santana wrote:
> Hi Michele
>
> >There is only one "annoyance" here: I noticed the dockerskeleton always
> expects a zip file including an "exec"
>
> dockerskeleton suppports adding an exec straight, it will take the file do
> a chmod +x and executed.
>
> So not sure what you are referring here as this is possible, maybe I miss
> understood
I use this script to post:
--- init.sh ---
#!/bin/bash
FILE=${1:?file}
echo '{"value":{"binary":true,"code":"'$(base64 $FILE)'"}}' >$FILE.json
curl -XPOST http://localhost:${PORT:-8080}/init -d @$FILE.json
----
exec is the compilation of this sample: http://jamesthom.as/blog/2016/06/21/serverless-go-actions/
I launch the docker skeleton:
docker -p 8080:8080 -ti openwhisk/dockerskeleton
If I post it as is:
./init.sh exec
I get:
err File is not a zip file
if I zip it:
zip exec.zip exec
./init.sh exec.zip
it works
Where am I wrong?
Re: Implementation plan for Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Hi Michele
>There is only one "annoyance" here: I noticed the dockerskeleton always
expects a zip file including an "exec"
dockerskeleton suppports adding an exec straight, it will take the file do
a chmod +x and executed.
So not sure what you are referring here as this is possible, maybe I miss
understood
>It is noticeably slow. Go is a bit faster but the problem with Go is
downloading dependencies (that can be a lot...).
This is the major problem for Swift also, SPM (Swift Package Manager) will
take the longest time git cloning entires repos to pull down dependencies
>1. create an apache/incubator-openwhisk-runtime-go repo, so I can fork it
and add my code in order to follow your build infrastructure and code
guidelines. Then I will submit a PR to it.
Yes this was the intention, to create the apache repo, then you can fork it
and submit a PR
>2. ....To make it super easy you may want a simple function somewhere to
import. .... just to add some code to the existing
https://github.com/apache/incubator-openwhisk-client-go.
This is super goes in line with what we do for the javascript library npm,
we use the same package in action code, and also outside client side to the
whisk api.
>3. Yes no issues in changing the wsk cli
super +1
> so please let some real expert on swift give a look at what I did and
fix it.
I'm not an expert :-), but I can help here either myself, or I can reach
out to swift experts that I know first hand in IBM from Kitura team
On Mon, Feb 26, 2018 at 10:55 AM Michele Sciabarra <mi...@sciabarra.com>
wrote:
>
> On Mon, Feb 26, 2018, at 4:39 AM, Carlos Santana wrote:
> > Great progress Michele +1
> >
> > I agree with the order of your steps.
> >
> > Step #1
> > Let's nock step 1 first.
> > I will summarize it as having first class support for Go at the same
> level
> > of JavaScript, Python, Java, Swift, PHP
> > Meaning from end user support from CLI, docs, runtime image.
> > wsk action create goaction goaction.go
> > wsk action create goaction goexec --kind go:1.10
> > wsk action create goaction goaction.zip --kind go:1.10
> > extend the runtime image with anything you want to add to the image
> > From example:
> > FROM openwhisk/action-go-v1.10
> > apt-get install -y imagemagick
> > docker build . -t csantanapr/action-go-v1.10
> > wsk action create goaction goaction.go --docker
> csantanapr/action-go-v1.10
> > I guess will have a FDK (Functions Developer Kit) :-), making it super
> easy
> > to import it and just write a simple go main method/function the FDK
> > handles the json input, output, etc..
> >
> >
> > I can help by requesting a new runtime repo
> > "apache/incubator-openwhisk-runtime-go"
> > I can setup a boilerplate repo content with some structure for gradle,
> > build, travis, scala tests
> > Then you can submit PR to the repo with go specifics.
> > I guess since your go guru you can handle the few changes to the wsk cli
> > Do you have a repo with some code already I can take a look?
> I have all the code done so far in
> https://github.com/sciabarracom/openwhisk-runtime-go.
> However since I followed my own way, please let do this way:
>
> 1. create an apache/incubator-openwhisk-runtime-go repo, so I can fork it
> and add my code in order to follow your build infrastructure and code
> guidelines. Then I will submit a PR to it.
>
> 2. Regarding the Go FDK, so far you can easily implement a piped executor
> using standard Go libraries. To make it super easy you may want a simple
> function somewhere to import. I think instead of creating a new sdk the
> best way is just to add some code to the existing
> https://github.com/apache/incubator-openwhisk-client-go.
> I can submit a PR for it.
>
> 3. Yes no issues in changing the wsk cli (I already gave a look at it).
> Will submit PR.
>
>
> >
> > Step #2
> > For swift, I can help since I'm working on swift4, it looks like we can
> > have a new proxy instead of using the python one.
> > We do some pre-compilation/magic today with buildandrecord.py [1] to
> > compile as much as possible up front and creating a shell script that has
> > the exact lines for compiling and linking.
> > Do you have some code on swift consuming from pipe instead of ReadLine I
> > can take a look?
>
> Yes this is the code I am using:
>
> https://github.com/sciabarracom/openwhisk-runtime-go/blob/pipe-loop/hello/swift/Sources/Hello/main.swift
>
> Written this way, it works under my new runtime. Note the ugly hack to
> call the flush in a system dependent way. Unfortunately I have not found a
> portable way to do that, and without flushing the stdout the runtime hangs.
>
> But all my knowledge of Swifts comes from "it looks like a bit like Scala"
> (that I know) and "Google says...", so please let some real expert on swift
> give a look at what I did and fix it.
>
> I think it is possible to have a backward compatible swift runtime maybe
> just running it under the new Go runtime, but before I think I do better to
> have stable code for the Go-only runtime.
>
> >
> > Step #3
> > After doing go and swift, we can see what are the common parts we can
> > extract to be common in the docker runtime repo.
> > And work on backwards compatible single image, and handle the new pipe
> with
> > some annotation, or maybe a new image, we can discuss more deeply once we
> > get there.
> Yup. Once solved Swift it should be easier...
>
> >
> >
> > [1]
> >
> https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
> >
> >
> >
> >
> >
> >
> > On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <
> openwhisk@sciabarra.com>
> > wrote:
> >
> > > Hello, whiskers
> > >
> > > I am at the risk of expanding too much the scope of what I am trying to
> > > do, so I want to try to focus and work step by step.
> > >
> > > So I am writing down my plan for review.
> > >
> > > # Step one: implement a good support for GoLang, with piped standard
> input
> > > and output, and log in stderr.
> > >
> > > Because there is not a past support for Go, except the generic docker
> > > support, I think I have some freedom here.
> > >
> > > The support will then work this way (please review carefully):
> > >
> > > ## /init
> > >
> > > The runtime expects posted in /init the usual
> > >
> > > {
> > > "binary": ...
> > > "code": ...
> > > "main": ..
> > > }
> > >
> > > where "code" is either a zip file (with binary: true) or a source code
> > > (with binary: false or absent).
> > >
> > > If it source code, it expects to be a go source.
> > > It places in /action/main/exec.go and compiles it with go build -o
> > > /action/exec /action/main/exec.go, getting a /action/exec to execute.
> > >
> > > If is is a zip file, it will unzip it in /action. If there is a
> > > /action/exec file in the zip it will run it.
> > >
> > > If there is not, it will look if there is a "main" parameter, getting
> the
> > > <main-value>.
> > >
> > > It will compile a /action/main/<main-value>.go. If the <main-value> is
> > > missing, it will default to /action/main/exec.go.
> > >
> > > Note the zip can include other source files that will be stored in
> > > /action/<folders>. So I will have to use GOPATH=/action to find them.
> > >
> > > ## /run
> > >
> > > The /run behaviour is:
> > >
> > > At every request, if no action process is running,it will start one and
> > > will pass the first line as an argument and also in standard output.
> > >
> > > If the executable does not terminate, it will keep feeding more input
> in
> > > standard input, expecting output in standard output (line by line) and
> logs
> > > in stderr.
> > >
> > > The server will have to collect stderr logs and write them in docker
> > > stdout, and terminate the output with the markers when an action
> completes:
> > >
> > > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> > > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> > >
> > > It the executable DOES terminate, it will execute again at the next
> /run
> > > request
> > >
> > > ## Step 2: need to implement the same logic for swift actions (we
> have to
> > > keep the existing compilation infrastructure)
> > >
> > > Issues to solve here are the compilation and the "warm" startup
> problem. I
> > > have some ideas but... I do better to discuss them once the Go support
> is
> > > finished.
> > >
> > > ## Step 3: will discuss with the list if it is worth to extend the
> support
> > > to the generic dockerskeleton, that is based on the python proxy.
> > >
> > > There are compatibility issues, because the current docker support logs
> > > everything in stdout and the last line is the answer, while for using
> the
> > > pipe you need to do separate log.
> > >
> > > How does it sound?
> > > Suggested or required changes?
> > > Am I missing something?
> > > Approved?
> > >
> > >
> > > --
> > > Michele Sciabarra
> > > openwhisk@sciabarra.com
> > >
>
Re: Implementation plan for Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
On Mon, Feb 26, 2018, at 4:39 AM, Carlos Santana wrote:
> Great progress Michele +1
>
> I agree with the order of your steps.
>
> Step #1
> Let's nock step 1 first.
> I will summarize it as having first class support for Go at the same level
> of JavaScript, Python, Java, Swift, PHP
> Meaning from end user support from CLI, docs, runtime image.
> wsk action create goaction goaction.go
> wsk action create goaction goexec --kind go:1.10
> wsk action create goaction goaction.zip --kind go:1.10
> extend the runtime image with anything you want to add to the image
> From example:
> FROM openwhisk/action-go-v1.10
> apt-get install -y imagemagick
> docker build . -t csantanapr/action-go-v1.10
> wsk action create goaction goaction.go --docker csantanapr/action-go-v1.10
> I guess will have a FDK (Functions Developer Kit) :-), making it super easy
> to import it and just write a simple go main method/function the FDK
> handles the json input, output, etc..
>
>
> I can help by requesting a new runtime repo
> "apache/incubator-openwhisk-runtime-go"
> I can setup a boilerplate repo content with some structure for gradle,
> build, travis, scala tests
> Then you can submit PR to the repo with go specifics.
> I guess since your go guru you can handle the few changes to the wsk cli
> Do you have a repo with some code already I can take a look?
I have all the code done so far in https://github.com/sciabarracom/openwhisk-runtime-go.
However since I followed my own way, please let do this way:
1. create an apache/incubator-openwhisk-runtime-go repo, so I can fork it and add my code in order to follow your build infrastructure and code guidelines. Then I will submit a PR to it.
2. Regarding the Go FDK, so far you can easily implement a piped executor using standard Go libraries. To make it super easy you may want a simple function somewhere to import. I think instead of creating a new sdk the best way is just to add some code to the existing https://github.com/apache/incubator-openwhisk-client-go.
I can submit a PR for it.
3. Yes no issues in changing the wsk cli (I already gave a look at it). Will submit PR.
>
> Step #2
> For swift, I can help since I'm working on swift4, it looks like we can
> have a new proxy instead of using the python one.
> We do some pre-compilation/magic today with buildandrecord.py [1] to
> compile as much as possible up front and creating a shell script that has
> the exact lines for compiling and linking.
> Do you have some code on swift consuming from pipe instead of ReadLine I
> can take a look?
Yes this is the code I am using:
https://github.com/sciabarracom/openwhisk-runtime-go/blob/pipe-loop/hello/swift/Sources/Hello/main.swift
Written this way, it works under my new runtime. Note the ugly hack to call the flush in a system dependent way. Unfortunately I have not found a portable way to do that, and without flushing the stdout the runtime hangs.
But all my knowledge of Swifts comes from "it looks like a bit like Scala" (that I know) and "Google says...", so please let some real expert on swift give a look at what I did and fix it.
I think it is possible to have a backward compatible swift runtime maybe just running it under the new Go runtime, but before I think I do better to have stable code for the Go-only runtime.
>
> Step #3
> After doing go and swift, we can see what are the common parts we can
> extract to be common in the docker runtime repo.
> And work on backwards compatible single image, and handle the new pipe with
> some annotation, or maybe a new image, we can discuss more deeply once we
> get there.
Yup. Once solved Swift it should be easier...
>
>
> [1]
> https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
>
>
>
>
>
>
> On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <op...@sciabarra.com>
> wrote:
>
> > Hello, whiskers
> >
> > I am at the risk of expanding too much the scope of what I am trying to
> > do, so I want to try to focus and work step by step.
> >
> > So I am writing down my plan for review.
> >
> > # Step one: implement a good support for GoLang, with piped standard input
> > and output, and log in stderr.
> >
> > Because there is not a past support for Go, except the generic docker
> > support, I think I have some freedom here.
> >
> > The support will then work this way (please review carefully):
> >
> > ## /init
> >
> > The runtime expects posted in /init the usual
> >
> > {
> > "binary": ...
> > "code": ...
> > "main": ..
> > }
> >
> > where "code" is either a zip file (with binary: true) or a source code
> > (with binary: false or absent).
> >
> > If it source code, it expects to be a go source.
> > It places in /action/main/exec.go and compiles it with go build -o
> > /action/exec /action/main/exec.go, getting a /action/exec to execute.
> >
> > If is is a zip file, it will unzip it in /action. If there is a
> > /action/exec file in the zip it will run it.
> >
> > If there is not, it will look if there is a "main" parameter, getting the
> > <main-value>.
> >
> > It will compile a /action/main/<main-value>.go. If the <main-value> is
> > missing, it will default to /action/main/exec.go.
> >
> > Note the zip can include other source files that will be stored in
> > /action/<folders>. So I will have to use GOPATH=/action to find them.
> >
> > ## /run
> >
> > The /run behaviour is:
> >
> > At every request, if no action process is running,it will start one and
> > will pass the first line as an argument and also in standard output.
> >
> > If the executable does not terminate, it will keep feeding more input in
> > standard input, expecting output in standard output (line by line) and logs
> > in stderr.
> >
> > The server will have to collect stderr logs and write them in docker
> > stdout, and terminate the output with the markers when an action completes:
> >
> > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >
> > It the executable DOES terminate, it will execute again at the next /run
> > request
> >
> > ## Step 2: need to implement the same logic for swift actions (we have to
> > keep the existing compilation infrastructure)
> >
> > Issues to solve here are the compilation and the "warm" startup problem. I
> > have some ideas but... I do better to discuss them once the Go support is
> > finished.
> >
> > ## Step 3: will discuss with the list if it is worth to extend the support
> > to the generic dockerskeleton, that is based on the python proxy.
> >
> > There are compatibility issues, because the current docker support logs
> > everything in stdout and the last line is the answer, while for using the
> > pipe you need to do separate log.
> >
> > How does it sound?
> > Suggested or required changes?
> > Am I missing something?
> > Approved?
> >
> >
> > --
> > Michele Sciabarra
> > openwhisk@sciabarra.com
> >
Re: Implementation plan for Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Yep I agree we OpenWhisk community need to be louder and do better for
users to understand that for compiled languages Swift, Go, Java, Rust, etc
Using a source at the time to invocations is better and best practice for
production for the actions to have a precompiled avoid those slow cold not
useful to the action owner or OW provider.
Like we are already used to do things OpenWhisk we do things in iterative
way and small steps so we would not wait for everything to be in place, we
can move forward having precompiled only support and later have source
support.
— Carlos
On Mon, Feb 26, 2018 at 7:24 AM Rodric Rabbah <ro...@gmail.com> wrote:
> Supporting go as source is nice for parity but you can also stage the
> changes focusing on the binary case first, get the /run details nailed, and
> working through logs and full experience first. Go cross-compilation works
> well and if you’re writing in go, beyond the convenience of quickly getting
> started with a source snippet, I would speculate the common use case is for
> the binary path.
>
> -r
>
> > On Feb 25, 2018, at 10:39 PM, Carlos Santana <cs...@gmail.com>
> wrote:
> >
> > Great progress Michele +1
> >
> > I agree with the order of your steps.
> >
> > Step #1
> > Let's nock step 1 first.
> > I will summarize it as having first class support for Go at the same
> level
> > of JavaScript, Python, Java, Swift, PHP
> > Meaning from end user support from CLI, docs, runtime image.
> > wsk action create goaction goaction.go
> > wsk action create goaction goexec --kind go:1.10
> > wsk action create goaction goaction.zip --kind go:1.10
> > extend the runtime image with anything you want to add to the image
> > From example:
> > FROM openwhisk/action-go-v1.10
> > apt-get install -y imagemagick
> > docker build . -t csantanapr/action-go-v1.10
> > wsk action create goaction goaction.go --docker
> csantanapr/action-go-v1.10
> > I guess will have a FDK (Functions Developer Kit) :-), making it super
> easy
> > to import it and just write a simple go main method/function the FDK
> > handles the json input, output, etc..
> >
> >
> > I can help by requesting a new runtime repo
> > "apache/incubator-openwhisk-runtime-go"
> > I can setup a boilerplate repo content with some structure for gradle,
> > build, travis, scala tests
> > Then you can submit PR to the repo with go specifics.
> > I guess since your go guru you can handle the few changes to the wsk cli
> > Do you have a repo with some code already I can take a look?
> >
> > Step #2
> > For swift, I can help since I'm working on swift4, it looks like we can
> > have a new proxy instead of using the python one.
> > We do some pre-compilation/magic today with buildandrecord.py [1] to
> > compile as much as possible up front and creating a shell script that has
> > the exact lines for compiling and linking.
> > Do you have some code on swift consuming from pipe instead of ReadLine I
> > can take a look?
> >
> > Step #3
> > After doing go and swift, we can see what are the common parts we can
> > extract to be common in the docker runtime repo.
> > And work on backwards compatible single image, and handle the new pipe
> with
> > some annotation, or maybe a new image, we can discuss more deeply once we
> > get there.
> >
> >
> > [1]
> >
> https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
> >
> >
> >
> >
> >
> >
> > On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <
> openwhisk@sciabarra.com>
> > wrote:
> >
> >> Hello, whiskers
> >>
> >> I am at the risk of expanding too much the scope of what I am trying to
> >> do, so I want to try to focus and work step by step.
> >>
> >> So I am writing down my plan for review.
> >>
> >> # Step one: implement a good support for GoLang, with piped standard
> input
> >> and output, and log in stderr.
> >>
> >> Because there is not a past support for Go, except the generic docker
> >> support, I think I have some freedom here.
> >>
> >> The support will then work this way (please review carefully):
> >>
> >> ## /init
> >>
> >> The runtime expects posted in /init the usual
> >>
> >> {
> >> "binary": ...
> >> "code": ...
> >> "main": ..
> >> }
> >>
> >> where "code" is either a zip file (with binary: true) or a source code
> >> (with binary: false or absent).
> >>
> >> If it source code, it expects to be a go source.
> >> It places in /action/main/exec.go and compiles it with go build -o
> >> /action/exec /action/main/exec.go, getting a /action/exec to execute.
> >>
> >> If is is a zip file, it will unzip it in /action. If there is a
> >> /action/exec file in the zip it will run it.
> >>
> >> If there is not, it will look if there is a "main" parameter, getting
> the
> >> <main-value>.
> >>
> >> It will compile a /action/main/<main-value>.go. If the <main-value> is
> >> missing, it will default to /action/main/exec.go.
> >>
> >> Note the zip can include other source files that will be stored in
> >> /action/<folders>. So I will have to use GOPATH=/action to find them.
> >>
> >> ## /run
> >>
> >> The /run behaviour is:
> >>
> >> At every request, if no action process is running,it will start one and
> >> will pass the first line as an argument and also in standard output.
> >>
> >> If the executable does not terminate, it will keep feeding more input in
> >> standard input, expecting output in standard output (line by line) and
> logs
> >> in stderr.
> >>
> >> The server will have to collect stderr logs and write them in docker
> >> stdout, and terminate the output with the markers when an action
> completes:
> >>
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >>
> >> It the executable DOES terminate, it will execute again at the next /run
> >> request
> >>
> >> ## Step 2: need to implement the same logic for swift actions (we have
> to
> >> keep the existing compilation infrastructure)
> >>
> >> Issues to solve here are the compilation and the "warm" startup
> problem. I
> >> have some ideas but... I do better to discuss them once the Go support
> is
> >> finished.
> >>
> >> ## Step 3: will discuss with the list if it is worth to extend the
> support
> >> to the generic dockerskeleton, that is based on the python proxy.
> >>
> >> There are compatibility issues, because the current docker support logs
> >> everything in stdout and the last line is the answer, while for using
> the
> >> pipe you need to do separate log.
> >>
> >> How does it sound?
> >> Suggested or required changes?
> >> Am I missing something?
> >> Approved?
> >>
> >>
> >> --
> >> Michele Sciabarra
> >> openwhisk@sciabarra.com
> >>
>
Re: Implementation plan for Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
I agree on focusing on binary support for now. There is only one "annoyance" here: I noticed the dockerskeleton always expects a zip file including an "exec". I would like to be able to support also binaries pushed straight, and I was trying to do that using magic file detection (there is a Go library for that).
To be honest, as I told already on Slack, I am a bit skeptic on source support tot court as it is implemented for example in the Swift image. It is noticeably slow. Go is a bit faster but the problem with Go is downloading dependencies (that can be a lot...).
You will have a long startup time when you scale, and having a long startup time when you need quickly more instances is definitely a killer problem for scaling support.
As I mentioned, for binaries like swift and go, and also for Java, I was thinking we should have something similar to what Kubernetes allows to do: an image for building, producing an executable, to be cached somewhere, to be reused when you launch more images.
Just my 2cents of ideas....
--
Michele Sciabarra
michele@sciabarra.com
On Mon, Feb 26, 2018, at 1:24 PM, Rodric Rabbah wrote:
> Supporting go as source is nice for parity but you can also stage the
> changes focusing on the binary case first, get the /run details nailed,
> and working through logs and full experience first. Go cross-compilation
> works well and if you’re writing in go, beyond the convenience of
> quickly getting started with a source snippet, I would speculate the
> common use case is for the binary path.
>
> -r
>
> > On Feb 25, 2018, at 10:39 PM, Carlos Santana <cs...@gmail.com> wrote:
> >
> > Great progress Michele +1
> >
> > I agree with the order of your steps.
> >
> > Step #1
> > Let's nock step 1 first.
> > I will summarize it as having first class support for Go at the same level
> > of JavaScript, Python, Java, Swift, PHP
> > Meaning from end user support from CLI, docs, runtime image.
> > wsk action create goaction goaction.go
> > wsk action create goaction goexec --kind go:1.10
> > wsk action create goaction goaction.zip --kind go:1.10
> > extend the runtime image with anything you want to add to the image
> > From example:
> > FROM openwhisk/action-go-v1.10
> > apt-get install -y imagemagick
> > docker build . -t csantanapr/action-go-v1.10
> > wsk action create goaction goaction.go --docker csantanapr/action-go-v1.10
> > I guess will have a FDK (Functions Developer Kit) :-), making it super easy
> > to import it and just write a simple go main method/function the FDK
> > handles the json input, output, etc..
> >
> >
> > I can help by requesting a new runtime repo
> > "apache/incubator-openwhisk-runtime-go"
> > I can setup a boilerplate repo content with some structure for gradle,
> > build, travis, scala tests
> > Then you can submit PR to the repo with go specifics.
> > I guess since your go guru you can handle the few changes to the wsk cli
> > Do you have a repo with some code already I can take a look?
> >
> > Step #2
> > For swift, I can help since I'm working on swift4, it looks like we can
> > have a new proxy instead of using the python one.
> > We do some pre-compilation/magic today with buildandrecord.py [1] to
> > compile as much as possible up front and creating a shell script that has
> > the exact lines for compiling and linking.
> > Do you have some code on swift consuming from pipe instead of ReadLine I
> > can take a look?
> >
> > Step #3
> > After doing go and swift, we can see what are the common parts we can
> > extract to be common in the docker runtime repo.
> > And work on backwards compatible single image, and handle the new pipe with
> > some annotation, or maybe a new image, we can discuss more deeply once we
> > get there.
> >
> >
> > [1]
> > https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
> >
> >
> >
> >
> >
> >
> > On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <op...@sciabarra.com>
> > wrote:
> >
> >> Hello, whiskers
> >>
> >> I am at the risk of expanding too much the scope of what I am trying to
> >> do, so I want to try to focus and work step by step.
> >>
> >> So I am writing down my plan for review.
> >>
> >> # Step one: implement a good support for GoLang, with piped standard input
> >> and output, and log in stderr.
> >>
> >> Because there is not a past support for Go, except the generic docker
> >> support, I think I have some freedom here.
> >>
> >> The support will then work this way (please review carefully):
> >>
> >> ## /init
> >>
> >> The runtime expects posted in /init the usual
> >>
> >> {
> >> "binary": ...
> >> "code": ...
> >> "main": ..
> >> }
> >>
> >> where "code" is either a zip file (with binary: true) or a source code
> >> (with binary: false or absent).
> >>
> >> If it source code, it expects to be a go source.
> >> It places in /action/main/exec.go and compiles it with go build -o
> >> /action/exec /action/main/exec.go, getting a /action/exec to execute.
> >>
> >> If is is a zip file, it will unzip it in /action. If there is a
> >> /action/exec file in the zip it will run it.
> >>
> >> If there is not, it will look if there is a "main" parameter, getting the
> >> <main-value>.
> >>
> >> It will compile a /action/main/<main-value>.go. If the <main-value> is
> >> missing, it will default to /action/main/exec.go.
> >>
> >> Note the zip can include other source files that will be stored in
> >> /action/<folders>. So I will have to use GOPATH=/action to find them.
> >>
> >> ## /run
> >>
> >> The /run behaviour is:
> >>
> >> At every request, if no action process is running,it will start one and
> >> will pass the first line as an argument and also in standard output.
> >>
> >> If the executable does not terminate, it will keep feeding more input in
> >> standard input, expecting output in standard output (line by line) and logs
> >> in stderr.
> >>
> >> The server will have to collect stderr logs and write them in docker
> >> stdout, and terminate the output with the markers when an action completes:
> >>
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >>
> >> It the executable DOES terminate, it will execute again at the next /run
> >> request
> >>
> >> ## Step 2: need to implement the same logic for swift actions (we have to
> >> keep the existing compilation infrastructure)
> >>
> >> Issues to solve here are the compilation and the "warm" startup problem. I
> >> have some ideas but... I do better to discuss them once the Go support is
> >> finished.
> >>
> >> ## Step 3: will discuss with the list if it is worth to extend the support
> >> to the generic dockerskeleton, that is based on the python proxy.
> >>
> >> There are compatibility issues, because the current docker support logs
> >> everything in stdout and the last line is the answer, while for using the
> >> pipe you need to do separate log.
> >>
> >> How does it sound?
> >> Suggested or required changes?
> >> Am I missing something?
> >> Approved?
> >>
> >>
> >> --
> >> Michele Sciabarra
> >> openwhisk@sciabarra.com
> >>
Re: Implementation plan for Go actions
Posted by Rodric Rabbah <ro...@gmail.com>.
Supporting go as source is nice for parity but you can also stage the changes focusing on the binary case first, get the /run details nailed, and working through logs and full experience first. Go cross-compilation works well and if you’re writing in go, beyond the convenience of quickly getting started with a source snippet, I would speculate the common use case is for the binary path.
-r
> On Feb 25, 2018, at 10:39 PM, Carlos Santana <cs...@gmail.com> wrote:
>
> Great progress Michele +1
>
> I agree with the order of your steps.
>
> Step #1
> Let's nock step 1 first.
> I will summarize it as having first class support for Go at the same level
> of JavaScript, Python, Java, Swift, PHP
> Meaning from end user support from CLI, docs, runtime image.
> wsk action create goaction goaction.go
> wsk action create goaction goexec --kind go:1.10
> wsk action create goaction goaction.zip --kind go:1.10
> extend the runtime image with anything you want to add to the image
> From example:
> FROM openwhisk/action-go-v1.10
> apt-get install -y imagemagick
> docker build . -t csantanapr/action-go-v1.10
> wsk action create goaction goaction.go --docker csantanapr/action-go-v1.10
> I guess will have a FDK (Functions Developer Kit) :-), making it super easy
> to import it and just write a simple go main method/function the FDK
> handles the json input, output, etc..
>
>
> I can help by requesting a new runtime repo
> "apache/incubator-openwhisk-runtime-go"
> I can setup a boilerplate repo content with some structure for gradle,
> build, travis, scala tests
> Then you can submit PR to the repo with go specifics.
> I guess since your go guru you can handle the few changes to the wsk cli
> Do you have a repo with some code already I can take a look?
>
> Step #2
> For swift, I can help since I'm working on swift4, it looks like we can
> have a new proxy instead of using the python one.
> We do some pre-compilation/magic today with buildandrecord.py [1] to
> compile as much as possible up front and creating a shell script that has
> the exact lines for compiling and linking.
> Do you have some code on swift consuming from pipe instead of ReadLine I
> can take a look?
>
> Step #3
> After doing go and swift, we can see what are the common parts we can
> extract to be common in the docker runtime repo.
> And work on backwards compatible single image, and handle the new pipe with
> some annotation, or maybe a new image, we can discuss more deeply once we
> get there.
>
>
> [1]
> https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
>
>
>
>
>
>
> On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <op...@sciabarra.com>
> wrote:
>
>> Hello, whiskers
>>
>> I am at the risk of expanding too much the scope of what I am trying to
>> do, so I want to try to focus and work step by step.
>>
>> So I am writing down my plan for review.
>>
>> # Step one: implement a good support for GoLang, with piped standard input
>> and output, and log in stderr.
>>
>> Because there is not a past support for Go, except the generic docker
>> support, I think I have some freedom here.
>>
>> The support will then work this way (please review carefully):
>>
>> ## /init
>>
>> The runtime expects posted in /init the usual
>>
>> {
>> "binary": ...
>> "code": ...
>> "main": ..
>> }
>>
>> where "code" is either a zip file (with binary: true) or a source code
>> (with binary: false or absent).
>>
>> If it source code, it expects to be a go source.
>> It places in /action/main/exec.go and compiles it with go build -o
>> /action/exec /action/main/exec.go, getting a /action/exec to execute.
>>
>> If is is a zip file, it will unzip it in /action. If there is a
>> /action/exec file in the zip it will run it.
>>
>> If there is not, it will look if there is a "main" parameter, getting the
>> <main-value>.
>>
>> It will compile a /action/main/<main-value>.go. If the <main-value> is
>> missing, it will default to /action/main/exec.go.
>>
>> Note the zip can include other source files that will be stored in
>> /action/<folders>. So I will have to use GOPATH=/action to find them.
>>
>> ## /run
>>
>> The /run behaviour is:
>>
>> At every request, if no action process is running,it will start one and
>> will pass the first line as an argument and also in standard output.
>>
>> If the executable does not terminate, it will keep feeding more input in
>> standard input, expecting output in standard output (line by line) and logs
>> in stderr.
>>
>> The server will have to collect stderr logs and write them in docker
>> stdout, and terminate the output with the markers when an action completes:
>>
>> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
>> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
>>
>> It the executable DOES terminate, it will execute again at the next /run
>> request
>>
>> ## Step 2: need to implement the same logic for swift actions (we have to
>> keep the existing compilation infrastructure)
>>
>> Issues to solve here are the compilation and the "warm" startup problem. I
>> have some ideas but... I do better to discuss them once the Go support is
>> finished.
>>
>> ## Step 3: will discuss with the list if it is worth to extend the support
>> to the generic dockerskeleton, that is based on the python proxy.
>>
>> There are compatibility issues, because the current docker support logs
>> everything in stdout and the last line is the answer, while for using the
>> pipe you need to do separate log.
>>
>> How does it sound?
>> Suggested or required changes?
>> Am I missing something?
>> Approved?
>>
>>
>> --
>> Michele Sciabarra
>> openwhisk@sciabarra.com
>>
Re: Implementation plan for Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Great progress Michele +1
I agree with the order of your steps.
Step #1
Let's nock step 1 first.
I will summarize it as having first class support for Go at the same level
of JavaScript, Python, Java, Swift, PHP
Meaning from end user support from CLI, docs, runtime image.
wsk action create goaction goaction.go
wsk action create goaction goexec --kind go:1.10
wsk action create goaction goaction.zip --kind go:1.10
extend the runtime image with anything you want to add to the image
From example:
FROM openwhisk/action-go-v1.10
apt-get install -y imagemagick
docker build . -t csantanapr/action-go-v1.10
wsk action create goaction goaction.go --docker csantanapr/action-go-v1.10
I guess will have a FDK (Functions Developer Kit) :-), making it super easy
to import it and just write a simple go main method/function the FDK
handles the json input, output, etc..
I can help by requesting a new runtime repo
"apache/incubator-openwhisk-runtime-go"
I can setup a boilerplate repo content with some structure for gradle,
build, travis, scala tests
Then you can submit PR to the repo with go specifics.
I guess since your go guru you can handle the few changes to the wsk cli
Do you have a repo with some code already I can take a look?
Step #2
For swift, I can help since I'm working on swift4, it looks like we can
have a new proxy instead of using the python one.
We do some pre-compilation/magic today with buildandrecord.py [1] to
compile as much as possible up front and creating a shell script that has
the exact lines for compiling and linking.
Do you have some code on swift consuming from pipe instead of ReadLine I
can take a look?
Step #3
After doing go and swift, we can see what are the common parts we can
extract to be common in the docker runtime repo.
And work on backwards compatible single image, and handle the new pipe with
some annotation, or maybe a new image, we can discuss more deeply once we
get there.
[1]
https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <op...@sciabarra.com>
wrote:
> Hello, whiskers
>
> I am at the risk of expanding too much the scope of what I am trying to
> do, so I want to try to focus and work step by step.
>
> So I am writing down my plan for review.
>
> # Step one: implement a good support for GoLang, with piped standard input
> and output, and log in stderr.
>
> Because there is not a past support for Go, except the generic docker
> support, I think I have some freedom here.
>
> The support will then work this way (please review carefully):
>
> ## /init
>
> The runtime expects posted in /init the usual
>
> {
> "binary": ...
> "code": ...
> "main": ..
> }
>
> where "code" is either a zip file (with binary: true) or a source code
> (with binary: false or absent).
>
> If it source code, it expects to be a go source.
> It places in /action/main/exec.go and compiles it with go build -o
> /action/exec /action/main/exec.go, getting a /action/exec to execute.
>
> If is is a zip file, it will unzip it in /action. If there is a
> /action/exec file in the zip it will run it.
>
> If there is not, it will look if there is a "main" parameter, getting the
> <main-value>.
>
> It will compile a /action/main/<main-value>.go. If the <main-value> is
> missing, it will default to /action/main/exec.go.
>
> Note the zip can include other source files that will be stored in
> /action/<folders>. So I will have to use GOPATH=/action to find them.
>
> ## /run
>
> The /run behaviour is:
>
> At every request, if no action process is running,it will start one and
> will pass the first line as an argument and also in standard output.
>
> If the executable does not terminate, it will keep feeding more input in
> standard input, expecting output in standard output (line by line) and logs
> in stderr.
>
> The server will have to collect stderr logs and write them in docker
> stdout, and terminate the output with the markers when an action completes:
>
> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
>
> It the executable DOES terminate, it will execute again at the next /run
> request
>
> ## Step 2: need to implement the same logic for swift actions (we have to
> keep the existing compilation infrastructure)
>
> Issues to solve here are the compilation and the "warm" startup problem. I
> have some ideas but... I do better to discuss them once the Go support is
> finished.
>
> ## Step 3: will discuss with the list if it is worth to extend the support
> to the generic dockerskeleton, that is based on the python proxy.
>
> There are compatibility issues, because the current docker support logs
> everything in stdout and the last line is the answer, while for using the
> pipe you need to do separate log.
>
> How does it sound?
> Suggested or required changes?
> Am I missing something?
> Approved?
>
>
> --
> Michele Sciabarra
> openwhisk@sciabarra.com
>