You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <op...@sciabarra.com> on 2018/02/13 22:07:33 UTC
First Implementation of the self-replacing Go support for OpenWhisk
As promised I released a first implementation of Go support using the technique I described before.
In short, a library implementing the proxy and serving both /run and /init, with the ability of replace itself with a new version.
Using the library, implementing a function in Go looks like this:
---
package main
import (
"encoding/json"
"fmt"
"github.com/sciabarracom/openwhisk-runtime-go/openwhisk"
)
func hello(event json.RawMessage) (json.RawMessage, error) {
// input and output
var input struct{ Name string }
var output struct {
Greetings string `json:"greetings"`
}
// read the input event
json.Unmarshal(event, &input)
if input.Name != "" {
// handle the event
output.Greetings = "Hello, " + input.Name
fmt.Println(output.Greetings)
return json.Marshal(output)
}
return nil, fmt.Errorf("no name specified")
}
func main() {
openwhisk.Start(ciao)
}
---
Actually in practice it is better to place the function in a separate package for implementing some tests, because apparently adding tests in the main package does not work.
Source code of the library is here:
https://github.com/sciabarracom/openwhisk-runtime-go
Here is a simple transcription of how it works and how I tested it.
First you build a couple of executable, and for simplicity you also prepare the json payload for the init.
$ cd test
$ go build -o hello ../main/hello.go
$ go build -o ciao ../main/ciao.go
$ echo '{"value":{"binary":true,"code":"'$(base64 hello)'"}}' >hello.json
$ echo '{"value":{"binary":true,"code":"'$(base64 ciao)'"}}' >ciao.json
Now you can start the actual server
$ go run ../main/exec.go
Now the magic happens.
Default behaviour (no executable)
```
$ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
{"error":"the action failed to locate a binary"}
```
Now post the `hello` handler and run it:
```
$ curl -XPOST http://localhost:8080/init -d @hello.json
OK
$ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
{"greetings":"Hello, Mike"}
```
As you can see, the function changed and now it implements the "hello" handler.
But the replaced server is still able to run init so let's do it again, replacing with the "ciao" handler.
```
$ curl -XPOST http://localhost:8080/init -d @ciao.json
OK
$ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
{"saluti":"Ciao, Mike"}
```
---
I want to say "thank you" to James Thomas, I basically copied his code for implementing the /run handler since it was ready, then I added my code for the /init. I also had to dig into the internals of the http Go package more than I expected.
Work is not yet complete, I have to manage the environment variables (stealing more code from James Thomas implementation), package in Docker, integrate in OpenWhisk. But definitely I believe looks like a promising start.
--
Michele Sciabarra
openwhisk@sciabarra.com
Re: First Implementation of the self-replacing Go support for OpenWhisk
Posted by Rodric Rabbah <ro...@gmail.com>.
- on notifying the caller of a failed exec: logs should be available even if container fails between init and run. As long as the logs appear on stdout and stderr they are fetched and amended to the activation record. This would appear as a generic error message that’s system generated and inspection of the logs should show the rest; your idea of stashing the error to construct a better message could also work but I’m not sure it’s worth the added complexity.
- as I noted on a previous related thread about this implementation path, the race on /run may require some attention from the invoker. In particular the connection in the past was retried, but, was removed because retries violate the invariant assumed by the invoker that after init, the container is ready. See this pr for related patch: https://github.com/apache/incubator-openwhisk/pull/1888
-r
Re: First Implementation of the self-replacing Go support for OpenWhisk
Posted by Michele Sciabarra <mi...@sciabarra.com>.
On Wed, Feb 14, 2018, at 10:45 AM, James Thomas wrote:
> Michele,
>
> Great work on getting an implemention working so quickly... I've been
> playing with it locally and it works perfectly.
Yes. It is actually in the state where I can run some performance tests.
That is my next step. I want to be sure it is actually more efficient than the "launch an executable" approach.
It should be, but better have some numbers...
Also I still have to integrate the rest of your code supporting the various environment variables...
> I have a couple of
> questions about the error handling and exec process.
>
> 1. Notifying user when exec fails?
Hmm if the exec fails, the server restarts. Probably I should send an error message if it fails, your right...
I want also include a "validation" step. Trying to execute once with a special parameter asking for the version, and execute the replacement if the version is compatible (so basically the user used the right library).
>
> Looking at the code, the exec to replace the binary happens after the
> `/init` HTTP response is returned. (I assume this is because the
> socket would be closed otherwise?)
> https://github.com/sciabarracom/openwhisk-runtime-go/blob/master/openwhisk/initHandler.go#L71
Actually I had some problems here... I had to Flush the request to avoid an error when you where uploading.
>
> If the exec fails, the library logs out a mesage to the console but
> this won't be available to the end user IIRC. I think it's important
> to surface this error to the user. One of the main issues people face
> with serverless is the pain in debugging.
Yup, will do that. And also implement a validation.
>
> If someone compiles the binary for the wrong platform or makes another
> mistake building the deployment archive, it will fail silently and
> then return the default response on run. Could we store this error
> message internally and return in the default `run` response if that
> happens?
>
totally agree.
> 2. Can you explain how the HTTP socket is managed when replacing the
> binary? Does it get automatically closed and then re-opened by the new
> process? I can't find enough information in the exec man page to help
> me understand what happens in this instance.
That is basically a kernel stuff. When you exec a new process, all the files are closed.
>
> 2.1 If the socket does get closed and re-opened, they will be a tiny
> amount of time when the HTTP server is not available. It'd be good to
> hear from someone with more experience in the platform that there's no
> way this race condition could be triggered, i.e. a `/run` request is
> fired before HTTP service is available again.
There is a race condition yes, but here we are replacing the implementation.
I have no idea how OpenWhisk handle this but some mentioned in the discussion the invoker can support this situation with a retry... Can someone tell us the behaviour of the Invoker ?
>
> I'm looking forward to trying out this binary runtime with other languages!
Actually, to support more languages I thought there could be another way to do that: pipes.
We could start another process as a child and read and write in input/output to communicate with the process.
This solution could solve the race issue and generalize the approach.
>
> On 13 February 2018 at 22:07, Michele Sciabarra <op...@sciabarra.com> wrote:
> > As promised I released a first implementation of Go support using the technique I described before.
> >
> > In short, a library implementing the proxy and serving both /run and /init, with the ability of replace itself with a new version.
> >
> > Using the library, implementing a function in Go looks like this:
> >
> > ---
> > package main
> >
> > import (
> > "encoding/json"
> > "fmt"
> >
> > "github.com/sciabarracom/openwhisk-runtime-go/openwhisk"
> > )
> >
> > func hello(event json.RawMessage) (json.RawMessage, error) {
> > // input and output
> > var input struct{ Name string }
> > var output struct {
> > Greetings string `json:"greetings"`
> > }
> > // read the input event
> > json.Unmarshal(event, &input)
> > if input.Name != "" {
> > // handle the event
> > output.Greetings = "Hello, " + input.Name
> > fmt.Println(output.Greetings)
> > return json.Marshal(output)
> > }
> > return nil, fmt.Errorf("no name specified")
> > }
> >
> > func main() {
> > openwhisk.Start(ciao)
> > }
> > ---
> >
> > Actually in practice it is better to place the function in a separate package for implementing some tests, because apparently adding tests in the main package does not work.
> >
> > Source code of the library is here:
> >
> > https://github.com/sciabarracom/openwhisk-runtime-go
> >
> > Here is a simple transcription of how it works and how I tested it.
> >
> > First you build a couple of executable, and for simplicity you also prepare the json payload for the init.
> >
> > $ cd test
> > $ go build -o hello ../main/hello.go
> > $ go build -o ciao ../main/ciao.go
> > $ echo '{"value":{"binary":true,"code":"'$(base64 hello)'"}}' >hello.json
> > $ echo '{"value":{"binary":true,"code":"'$(base64 ciao)'"}}' >ciao.json
> >
> > Now you can start the actual server
> >
> > $ go run ../main/exec.go
> >
> > Now the magic happens.
> >
> > Default behaviour (no executable)
> >
> > ```
> > $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> > {"error":"the action failed to locate a binary"}
> > ```
> >
> > Now post the `hello` handler and run it:
> >
> > ```
> > $ curl -XPOST http://localhost:8080/init -d @hello.json
> > OK
> > $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> > {"greetings":"Hello, Mike"}
> > ```
> >
> > As you can see, the function changed and now it implements the "hello" handler.
> >
> > But the replaced server is still able to run init so let's do it again, replacing with the "ciao" handler.
> >
> >
> > ```
> > $ curl -XPOST http://localhost:8080/init -d @ciao.json
> > OK
> > $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> > {"saluti":"Ciao, Mike"}
> > ```
> >
> > ---
> >
> > I want to say "thank you" to James Thomas, I basically copied his code for implementing the /run handler since it was ready, then I added my code for the /init. I also had to dig into the internals of the http Go package more than I expected.
> >
> > Work is not yet complete, I have to manage the environment variables (stealing more code from James Thomas implementation), package in Docker, integrate in OpenWhisk. But definitely I believe looks like a promising start.
> >
> > --
> > Michele Sciabarra
> > openwhisk@sciabarra.com
>
>
>
> --
> Regards,
> James Thomas
Re: First Implementation of the self-replacing Go support for OpenWhisk
Posted by James Thomas <jt...@gmail.com>.
Michele,
Great work on getting an implemention working so quickly... I've been
playing with it locally and it works perfectly. I have a couple of
questions about the error handling and exec process.
1. Notifying user when exec fails?
Looking at the code, the exec to replace the binary happens after the
`/init` HTTP response is returned. (I assume this is because the
socket would be closed otherwise?)
https://github.com/sciabarracom/openwhisk-runtime-go/blob/master/openwhisk/initHandler.go#L71
If the exec fails, the library logs out a mesage to the console but
this won't be available to the end user IIRC. I think it's important
to surface this error to the user. One of the main issues people face
with serverless is the pain in debugging.
If someone compiles the binary for the wrong platform or makes another
mistake building the deployment archive, it will fail silently and
then return the default response on run. Could we store this error
message internally and return in the default `run` response if that
happens?
2. Can you explain how the HTTP socket is managed when replacing the
binary? Does it get automatically closed and then re-opened by the new
process? I can't find enough information in the exec man page to help
me understand what happens in this instance.
2.1 If the socket does get closed and re-opened, they will be a tiny
amount of time when the HTTP server is not available. It'd be good to
hear from someone with more experience in the platform that there's no
way this race condition could be triggered, i.e. a `/run` request is
fired before HTTP service is available again.
I'm looking forward to trying out this binary runtime with other languages!
On 13 February 2018 at 22:07, Michele Sciabarra <op...@sciabarra.com> wrote:
> As promised I released a first implementation of Go support using the technique I described before.
>
> In short, a library implementing the proxy and serving both /run and /init, with the ability of replace itself with a new version.
>
> Using the library, implementing a function in Go looks like this:
>
> ---
> package main
>
> import (
> "encoding/json"
> "fmt"
>
> "github.com/sciabarracom/openwhisk-runtime-go/openwhisk"
> )
>
> func hello(event json.RawMessage) (json.RawMessage, error) {
> // input and output
> var input struct{ Name string }
> var output struct {
> Greetings string `json:"greetings"`
> }
> // read the input event
> json.Unmarshal(event, &input)
> if input.Name != "" {
> // handle the event
> output.Greetings = "Hello, " + input.Name
> fmt.Println(output.Greetings)
> return json.Marshal(output)
> }
> return nil, fmt.Errorf("no name specified")
> }
>
> func main() {
> openwhisk.Start(ciao)
> }
> ---
>
> Actually in practice it is better to place the function in a separate package for implementing some tests, because apparently adding tests in the main package does not work.
>
> Source code of the library is here:
>
> https://github.com/sciabarracom/openwhisk-runtime-go
>
> Here is a simple transcription of how it works and how I tested it.
>
> First you build a couple of executable, and for simplicity you also prepare the json payload for the init.
>
> $ cd test
> $ go build -o hello ../main/hello.go
> $ go build -o ciao ../main/ciao.go
> $ echo '{"value":{"binary":true,"code":"'$(base64 hello)'"}}' >hello.json
> $ echo '{"value":{"binary":true,"code":"'$(base64 ciao)'"}}' >ciao.json
>
> Now you can start the actual server
>
> $ go run ../main/exec.go
>
> Now the magic happens.
>
> Default behaviour (no executable)
>
> ```
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"error":"the action failed to locate a binary"}
> ```
>
> Now post the `hello` handler and run it:
>
> ```
> $ curl -XPOST http://localhost:8080/init -d @hello.json
> OK
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"greetings":"Hello, Mike"}
> ```
>
> As you can see, the function changed and now it implements the "hello" handler.
>
> But the replaced server is still able to run init so let's do it again, replacing with the "ciao" handler.
>
>
> ```
> $ curl -XPOST http://localhost:8080/init -d @ciao.json
> OK
> $ curl -XPOST http://localhost:8080/run -d '{"value":{"name":"Mike"}}'
> {"saluti":"Ciao, Mike"}
> ```
>
> ---
>
> I want to say "thank you" to James Thomas, I basically copied his code for implementing the /run handler since it was ready, then I added my code for the /init. I also had to dig into the internals of the http Go package more than I expected.
>
> Work is not yet complete, I have to manage the environment variables (stealing more code from James Thomas implementation), package in Docker, integrate in OpenWhisk. But definitely I believe looks like a promising start.
>
> --
> Michele Sciabarra
> openwhisk@sciabarra.com
--
Regards,
James Thomas