You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <op...@sciabarra.com> on 2018/03/09 11:13:25 UTC
[LONG] Discussing my implementation of Go actions
I just did a PR of my version of the Golang action implementation. It does some "breaking" changes and there is some discussion on the slack channel.
So I report the current situation n here, looking for advices and change recommendations. Since I am a bit confused, if I remember well, one Apache rule is the mailing list is the ultimate source for the truth...
It currently works this way (I call it the "pipe-loop" protocol)
A golang action (or a generic binary) is expected to follow this "protocol":
* starts with {"openwhisk": 1}
* reads on line in stardard input, expecting a json ON A SINGLE LINE
* process the line, emits logs in stderr (can be multiple lines)
* outputs a line in stdout in json format ON A SINGLE LINE
* repeat forever
It is important to note this design is easy to implement and works even for bash scripts, but it is easy to use also perl, ruby, haskell in an EFFICIENT way. Indeed this bash script (with jq) is part of my tests:
---
#!/bin/bash
echo '{"openwhisk":1}'
while read line
do
name="$(echo $line | jq -r .name)"
logger -s "name=$name"
hello="Hello, $name"
logger -s "sent response"
echo '{"hello":"'$hello'"}'
done
---
Things discussed:
1) remove the header {"openwhisk":1}
Actually initially it was not there. But I decided to add this requirements because the action need to speak a protocol ANYWAY.
Most important, I explain why I require it starts with "{"openwhisk: 1}".
The main reason is: I start the child process at init time, and I wanted to detect when it does not behave properly.
The simplest problem happens when the action crashes immediately. For example, a common reason for this problem is uploading a binary using some dynamic libraries not available in the runtime. For example a swift action. By defaults it load a lot of different libraries, it crashes immediately but I cannot detect it until I try to read its stdin.
I can remove this requirement if someone can show me the go code to check that cmd.Start("true") or cmd.Start("pwd") exited 😃
If it is not doable, and I skip the handshake, even if the command crashed, I will not detect the problem until a /run is executed and the action times out...
Carlos say it is fine. It is ok for me but I still think an early problem detection would be better. Also James recommended me to provide as much as error detection to the user as early as possible. Kinda of conflicting directives here...
Suggestions?
2) more checks at init time
I added some sanity checks. Probably too many. I tried to detect the error at deployment time, not at invocation time.
This is different from what currently for example dockerskeleton does.
If I upload for example something wrong, like a non-zip, a non-elf executable, my init returns {"error": "description"}, while currently the dockerskeleton returns always OK.
Recommendations here?
3) output to another channel the result
Currently I require logs goes to stderr, and stdout is for interacting with the parent process.
Rodric suggested to output to a separate channel (channel 3?) and use stdout and stderr for logs.
While doable, I need to provision another pipe, and the implementation should probably do some syscalls to retrieve file descriptor 3. It would complicate implementation, while currently it is straightforward for any language that does not have a library available. For swift, even to flush stdout I needed to write "linux specific" code... I do not dare to think what I need to do to write in fd3...
My opinion is that using stdout for I/O and stderr for logs is a better choice than opening another file descriptor.
Thoughts here?
--
Michele Sciabarra
openwhisk@sciabarra.com
Re: [LONG] Discussing my implementation of Go actions
Posted by Rodric Rabbah <ro...@gmail.com>.
> On Mar 10, 2018, at 3:54 AM, Michele Sciabarra <mi...@sciabarra.com> wrote:
>
> As someone noted, OpenFaas claims to be able to do that.
openfaas builds a container per function.
-r
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
# the problem of the protocol
I am aligned on your view. Indeed what I actually did was to reverse engineering some of the runtimes (most notably) the dockerskeleton to implement the Go based docker skeleton. For efficiency, I had to depart from some of the current practices.
As you remarked, the basic problem is while java, python, node can "load code" dynamically and the protocol can be hidden in the proxy, for binaries you cannot. SO I have to mandate a protocol for the binaries.
We can hide this protocol in a library but since there are infinite ways of generate a binary (not just go and swift, but also rust, haskell, even C#. And actually executing a binary in practice means also running any interpreted language, so you could support efficiently Ruby or bash or whatever is not already available on the supported programming language.
As someone noted, OpenFaas claims to be able to do that. I should investigate better on what OpenFaas does, maybe to copy some ideas :). I know, we should focus on Go but it is not the nature of the problem. The nature of the problem is supporting generic executables
So what really need to do, is to create AND document a protocol for generic unix executable to be able to interface with OpenWhisk. And the protocol should be simple enough that can be implemented without libraries!
----
# current protocol situation
At this stage, we have implicitly a protocol for native actions. Let's call it "OpenWhisk Native Protocol", currently at "version 0.9"
This protocol is actually in use at least for swift actions, where I guess there is a significant user base already, and it is somewhat documented in the dockerskeleton. So what we are really discussing here is, in my interpretation:
How can we evolve the protocol to make easier the transition for the existing code base?
So let's try to put it in a formal way, what we have discussing here.
For "OpenWhisk Native Protocol, v0" the current one, we have:
actions will receive the input in stdin AND on the command line, will produce as much output as they like in stdin and stderr as long as the last one in stdout is a valid json object. It is no-brainer it is not a very efficient implementation.
Also binaries are not required to identify themselves (something that reminds me of HTTP/0.9) and the protocol they speak.
---
# the dicussion for protocol v1
For the "OpenWhisk Native Protocol v1", the one I am trying to implement, I am proposing this solution:
- the native binary must identify himself (for error detection) with {openwhisk: 1} (with a view to became 2, 3, 4 for the streaming support)
- it will loop on stdin, produce output on stdout, one json per line, and log on stderr
HOWEVER, as it has been noted, this will create problems with existing Swift binaries. Swift users log with Swift print that produces output on stdout.
So I was recommended to use a different channel (channel 3?) and skip the handshake.
My concerns are the worse error detection (for the good or the bad, I still think that detecting a misbehaving binary not supporting the protocol at init time is a good thing), and the fact that coding will became a bit awkward for both Go and Swift users. And I believe the handshake should be used anyway.
At this stage, my idea is just to add to my go proxy a couple of environment variables, like:
OPENWHISK_OUTPUT_CHANNEL=3
OPENWHISK_REQUIRE_HANDSHAKE=no
so the proxy can be used with no changes for swift actions, while leaving a natural behaviour for Go actions. A new docker image is required anyway to support. However I think this idea should be discussed.
# proposal to document the native protocol
However, I am proposing here just create a page, openwhisk-native-protocol.md and discuss the protocol before, and write down this behaviour as the "openwhisk native protocol" v1, and be prepared for more evolutions of the protocol to support streaming and other planned features.
On Fri, Mar 9, 2018, at 9:53 PM, Rodric Rabbah wrote:
> This is a good discussion - thanks for bringing it to the dev list.
>
> In essence, native actions push the boundary of how much of the function
> abstraction we can maintain. For some of the managed runtimes which include
> Node.js, Python and Java, we are able to hide the protocol you allude to in
> what we loosely have called the runtime proxy. The proxy is where the
> initialization and run protocols are relevant. We deliberately resisted
> publishing and documenting the proxy protocol for some time.
>
> As you observed, the initialization must handshake to the invoker - we do
> this today with a generic HTTP response code of 2xx. Anything else is
> treated as error. If your current proposal, this is equivalent to
> {"openwhisk": 1}.
>
> The function can only reach the run stage if initialization returns
> successfully. This does not however provide any strong guarantee that the
> action is well formed or valid. Only that for Node.js for example, that it
> parses OK. All errors are eventually detected at the run phase.
>
> In the work you've done, because of the nature of the native functions, one
> way to look at the changes is that you're exposing the implicit proxy
> protocol to the "function" itself. It must confirm that it's ready (for
> whatever notion of readiness it deems valid), and only then is the run
> executed.
>
> In this way, one can say that a function execution in this model follows a
> different model, where there are at least two (although can admit three)
> stages: the function must implement an initializer and a run method, and if
> you admit the third, a shutdown method (we don't have this today in general
> but it would be nice to allow a "function" to shutdown cleanly before its
> resources are killed).
>
> These changes are a departure from the simple interface that some of the
> languages have, but already for native functions, they _have_ to do
> something different. Namely, they receive the input from stdin, and they
> produce the result on stdout (as the last line); this is not a nice and
> clean function interface compared to say Node.js, Python, or Java.
>
> So I like to think of this and suggest it as a way of grounding the
> discussion in terms of a foreign function interfaces for OpenWhisk. A
> native function already breaks some of the clean abstractions. Before I
> elaborate further, I will pause here for feedback.
>
> -r
>
> On Fri, Mar 9, 2018 at 6:13 AM, Michele Sciabarra <op...@sciabarra.com>
> wrote:
>
> > I just did a PR of my version of the Golang action implementation. It
> > does some "breaking" changes and there is some discussion on the slack
> > channel.
> >
> > So I report the current situation n here, looking for advices and change
> > recommendations. Since I am a bit confused, if I remember well, one Apache
> > rule is the mailing list is the ultimate source for the truth...
> >
> > It currently works this way (I call it the "pipe-loop" protocol)
> >
> > A golang action (or a generic binary) is expected to follow this
> > "protocol":
> >
> > * starts with {"openwhisk": 1}
> > * reads on line in stardard input, expecting a json ON A SINGLE LINE
> > * process the line, emits logs in stderr (can be multiple lines)
> > * outputs a line in stdout in json format ON A SINGLE LINE
> > * repeat forever
> >
> > It is important to note this design is easy to implement and works even
> > for bash scripts, but it is easy to use also perl, ruby, haskell in an
> > EFFICIENT way. Indeed this bash script (with jq) is part of my tests:
> >
> > ---
> > #!/bin/bash
> > echo '{"openwhisk":1}'
> > while read line
> > do
> > name="$(echo $line | jq -r .name)"
> > logger -s "name=$name"
> > hello="Hello, $name"
> > logger -s "sent response"
> > echo '{"hello":"'$hello'"}'
> > done
> > ---
> >
> > Things discussed:
> >
> > 1) remove the header {"openwhisk":1}
> >
> > Actually initially it was not there. But I decided to add this
> > requirements because the action need to speak a protocol ANYWAY.
> >
> > Most important, I explain why I require it starts with "{"openwhisk: 1}".
> >
> > The main reason is: I start the child process at init time, and I wanted
> > to detect when it does not behave properly.
> >
> > The simplest problem happens when the action crashes immediately. For
> > example, a common reason for this problem is uploading a binary using some
> > dynamic libraries not available in the runtime. For example a swift
> > action. By defaults it load a lot of different libraries, it crashes
> > immediately but I cannot detect it until I try to read its stdin.
> >
> > I can remove this requirement if someone can show me the go code to check
> > that cmd.Start("true") or cmd.Start("pwd") exited 😃
> >
> > If it is not doable, and I skip the handshake, even if the command
> > crashed, I will not detect the problem until a /run is executed and the
> > action times out...
> >
> > Carlos say it is fine. It is ok for me but I still think an early problem
> > detection would be better. Also James recommended me to provide as much as
> > error detection to the user as early as possible. Kinda of conflicting
> > directives here...
> >
> > Suggestions?
> >
> > 2) more checks at init time
> >
> > I added some sanity checks. Probably too many. I tried to detect the
> > error at deployment time, not at invocation time.
> >
> > This is different from what currently for example dockerskeleton does.
> >
> > If I upload for example something wrong, like a non-zip, a non-elf
> > executable, my init returns {"error": "description"}, while currently the
> > dockerskeleton returns always OK.
> >
> > Recommendations here?
> >
> > 3) output to another channel the result
> >
> > Currently I require logs goes to stderr, and stdout is for interacting
> > with the parent process.
> >
> > Rodric suggested to output to a separate channel (channel 3?) and use
> > stdout and stderr for logs.
> >
> > While doable, I need to provision another pipe, and the implementation
> > should probably do some syscalls to retrieve file descriptor 3. It would
> > complicate implementation, while currently it is straightforward for any
> > language that does not have a library available. For swift, even to flush
> > stdout I needed to write "linux specific" code... I do not dare to think
> > what I need to do to write in fd3...
> >
> > My opinion is that using stdout for I/O and stderr for logs is a better
> > choice than opening another file descriptor.
> >
> > Thoughts here?
> >
> >
> >
> >
> >
> >
> >
> > --
> > Michele Sciabarra
> > openwhisk@sciabarra.com
> >
Re: [LONG] Discussing my implementation of Go actions
Posted by Rodric Rabbah <ro...@gmail.com>.
This is a good discussion - thanks for bringing it to the dev list.
In essence, native actions push the boundary of how much of the function
abstraction we can maintain. For some of the managed runtimes which include
Node.js, Python and Java, we are able to hide the protocol you allude to in
what we loosely have called the runtime proxy. The proxy is where the
initialization and run protocols are relevant. We deliberately resisted
publishing and documenting the proxy protocol for some time.
As you observed, the initialization must handshake to the invoker - we do
this today with a generic HTTP response code of 2xx. Anything else is
treated as error. If your current proposal, this is equivalent to
{"openwhisk": 1}.
The function can only reach the run stage if initialization returns
successfully. This does not however provide any strong guarantee that the
action is well formed or valid. Only that for Node.js for example, that it
parses OK. All errors are eventually detected at the run phase.
In the work you've done, because of the nature of the native functions, one
way to look at the changes is that you're exposing the implicit proxy
protocol to the "function" itself. It must confirm that it's ready (for
whatever notion of readiness it deems valid), and only then is the run
executed.
In this way, one can say that a function execution in this model follows a
different model, where there are at least two (although can admit three)
stages: the function must implement an initializer and a run method, and if
you admit the third, a shutdown method (we don't have this today in general
but it would be nice to allow a "function" to shutdown cleanly before its
resources are killed).
These changes are a departure from the simple interface that some of the
languages have, but already for native functions, they _have_ to do
something different. Namely, they receive the input from stdin, and they
produce the result on stdout (as the last line); this is not a nice and
clean function interface compared to say Node.js, Python, or Java.
So I like to think of this and suggest it as a way of grounding the
discussion in terms of a foreign function interfaces for OpenWhisk. A
native function already breaks some of the clean abstractions. Before I
elaborate further, I will pause here for feedback.
-r
On Fri, Mar 9, 2018 at 6:13 AM, Michele Sciabarra <op...@sciabarra.com>
wrote:
> I just did a PR of my version of the Golang action implementation. It
> does some "breaking" changes and there is some discussion on the slack
> channel.
>
> So I report the current situation n here, looking for advices and change
> recommendations. Since I am a bit confused, if I remember well, one Apache
> rule is the mailing list is the ultimate source for the truth...
>
> It currently works this way (I call it the "pipe-loop" protocol)
>
> A golang action (or a generic binary) is expected to follow this
> "protocol":
>
> * starts with {"openwhisk": 1}
> * reads on line in stardard input, expecting a json ON A SINGLE LINE
> * process the line, emits logs in stderr (can be multiple lines)
> * outputs a line in stdout in json format ON A SINGLE LINE
> * repeat forever
>
> It is important to note this design is easy to implement and works even
> for bash scripts, but it is easy to use also perl, ruby, haskell in an
> EFFICIENT way. Indeed this bash script (with jq) is part of my tests:
>
> ---
> #!/bin/bash
> echo '{"openwhisk":1}'
> while read line
> do
> name="$(echo $line | jq -r .name)"
> logger -s "name=$name"
> hello="Hello, $name"
> logger -s "sent response"
> echo '{"hello":"'$hello'"}'
> done
> ---
>
> Things discussed:
>
> 1) remove the header {"openwhisk":1}
>
> Actually initially it was not there. But I decided to add this
> requirements because the action need to speak a protocol ANYWAY.
>
> Most important, I explain why I require it starts with "{"openwhisk: 1}".
>
> The main reason is: I start the child process at init time, and I wanted
> to detect when it does not behave properly.
>
> The simplest problem happens when the action crashes immediately. For
> example, a common reason for this problem is uploading a binary using some
> dynamic libraries not available in the runtime. For example a swift
> action. By defaults it load a lot of different libraries, it crashes
> immediately but I cannot detect it until I try to read its stdin.
>
> I can remove this requirement if someone can show me the go code to check
> that cmd.Start("true") or cmd.Start("pwd") exited 😃
>
> If it is not doable, and I skip the handshake, even if the command
> crashed, I will not detect the problem until a /run is executed and the
> action times out...
>
> Carlos say it is fine. It is ok for me but I still think an early problem
> detection would be better. Also James recommended me to provide as much as
> error detection to the user as early as possible. Kinda of conflicting
> directives here...
>
> Suggestions?
>
> 2) more checks at init time
>
> I added some sanity checks. Probably too many. I tried to detect the
> error at deployment time, not at invocation time.
>
> This is different from what currently for example dockerskeleton does.
>
> If I upload for example something wrong, like a non-zip, a non-elf
> executable, my init returns {"error": "description"}, while currently the
> dockerskeleton returns always OK.
>
> Recommendations here?
>
> 3) output to another channel the result
>
> Currently I require logs goes to stderr, and stdout is for interacting
> with the parent process.
>
> Rodric suggested to output to a separate channel (channel 3?) and use
> stdout and stderr for logs.
>
> While doable, I need to provision another pipe, and the implementation
> should probably do some syscalls to retrieve file descriptor 3. It would
> complicate implementation, while currently it is straightforward for any
> language that does not have a library available. For swift, even to flush
> stdout I needed to write "linux specific" code... I do not dare to think
> what I need to do to write in fd3...
>
> My opinion is that using stdout for I/O and stderr for logs is a better
> choice than opening another file descriptor.
>
> Thoughts here?
>
>
>
>
>
>
>
> --
> Michele Sciabarra
> openwhisk@sciabarra.com
>
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
On Fri, Mar 9, 2018, at 12:37 PM, Carlos Santana wrote:
> Thank you Michele for the contribution.
> I already provided some initial feedback in the PR
>
> https://github.com/apache/incubator-openwhisk-runtime-go/pull/1
Yes I see you asked to remove the handshake()
As I told you, an eventually crashing exec will be detectable only at the first /run.
I am still convinced I should instead be detected at /init time.
I still hope to find a way to remove the handshake AND not wait until /run to detect a wrong exec.
Of course if I do not find a solution will settle for removing the handshake()
>
> On Fri, Mar 9, 2018 at 6:13 AM Michele Sciabarra <op...@sciabarra.com>
> wrote:
>
> > I just did a PR of my version of the Golang action implementation. It
> > does some "breaking" changes and there is some discussion on the slack
> > channel.
> >
> > So I report the current situation n here, looking for advices and change
> > recommendations. Since I am a bit confused, if I remember well, one Apache
> > rule is the mailing list is the ultimate source for the truth...
> >
> > It currently works this way (I call it the "pipe-loop" protocol)
> >
> > A golang action (or a generic binary) is expected to follow this
> > "protocol":
> >
> > * starts with {"openwhisk": 1}
> > * reads on line in stardard input, expecting a json ON A SINGLE LINE
> > * process the line, emits logs in stderr (can be multiple lines)
> > * outputs a line in stdout in json format ON A SINGLE LINE
> > * repeat forever
> >
> > It is important to note this design is easy to implement and works even
> > for bash scripts, but it is easy to use also perl, ruby, haskell in an
> > EFFICIENT way. Indeed this bash script (with jq) is part of my tests:
> >
> > ---
> > #!/bin/bash
> > echo '{"openwhisk":1}'
> > while read line
> > do
> > name="$(echo $line | jq -r .name)"
> > logger -s "name=$name"
> > hello="Hello, $name"
> > logger -s "sent response"
> > echo '{"hello":"'$hello'"}'
> > done
> > ---
> >
> > Things discussed:
> >
> > 1) remove the header {"openwhisk":1}
> >
> > Actually initially it was not there. But I decided to add this
> > requirements because the action need to speak a protocol ANYWAY.
> >
> > Most important, I explain why I require it starts with "{"openwhisk: 1}".
> >
> > The main reason is: I start the child process at init time, and I wanted
> > to detect when it does not behave properly.
> >
> > The simplest problem happens when the action crashes immediately. For
> > example, a common reason for this problem is uploading a binary using some
> > dynamic libraries not available in the runtime. For example a swift
> > action. By defaults it load a lot of different libraries, it crashes
> > immediately but I cannot detect it until I try to read its stdin.
> >
> > I can remove this requirement if someone can show me the go code to check
> > that cmd.Start("true") or cmd.Start("pwd") exited 😃
> >
> > If it is not doable, and I skip the handshake, even if the command
> > crashed, I will not detect the problem until a /run is executed and the
> > action times out...
> >
> > Carlos say it is fine. It is ok for me but I still think an early problem
> > detection would be better. Also James recommended me to provide as much as
> > error detection to the user as early as possible. Kinda of conflicting
> > directives here...
> >
> > Suggestions?
> >
> > 2) more checks at init time
> >
> > I added some sanity checks. Probably too many. I tried to detect the
> > error at deployment time, not at invocation time.
> >
> > This is different from what currently for example dockerskeleton does.
> >
> > If I upload for example something wrong, like a non-zip, a non-elf
> > executable, my init returns {"error": "description"}, while currently the
> > dockerskeleton returns always OK.
> >
> > Recommendations here?
> >
> > 3) output to another channel the result
> >
> > Currently I require logs goes to stderr, and stdout is for interacting
> > with the parent process.
> >
> > Rodric suggested to output to a separate channel (channel 3?) and use
> > stdout and stderr for logs.
> >
> > While doable, I need to provision another pipe, and the implementation
> > should probably do some syscalls to retrieve file descriptor 3. It would
> > complicate implementation, while currently it is straightforward for any
> > language that does not have a library available. For swift, even to flush
> > stdout I needed to write "linux specific" code... I do not dare to think
> > what I need to do to write in fd3...
> >
> > My opinion is that using stdout for I/O and stderr for logs is a better
> > choice than opening another file descriptor.
> >
> > Thoughts here?
> >
> >
> >
> >
> >
> >
> >
> > --
> > Michele Sciabarra
> > openwhisk@sciabarra.com
> >
Re: [LONG] Discussing my implementation of Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Thank you Michele for the contribution.
I already provided some initial feedback in the PR
https://github.com/apache/incubator-openwhisk-runtime-go/pull/1
On Fri, Mar 9, 2018 at 6:13 AM Michele Sciabarra <op...@sciabarra.com>
wrote:
> I just did a PR of my version of the Golang action implementation. It
> does some "breaking" changes and there is some discussion on the slack
> channel.
>
> So I report the current situation n here, looking for advices and change
> recommendations. Since I am a bit confused, if I remember well, one Apache
> rule is the mailing list is the ultimate source for the truth...
>
> It currently works this way (I call it the "pipe-loop" protocol)
>
> A golang action (or a generic binary) is expected to follow this
> "protocol":
>
> * starts with {"openwhisk": 1}
> * reads on line in stardard input, expecting a json ON A SINGLE LINE
> * process the line, emits logs in stderr (can be multiple lines)
> * outputs a line in stdout in json format ON A SINGLE LINE
> * repeat forever
>
> It is important to note this design is easy to implement and works even
> for bash scripts, but it is easy to use also perl, ruby, haskell in an
> EFFICIENT way. Indeed this bash script (with jq) is part of my tests:
>
> ---
> #!/bin/bash
> echo '{"openwhisk":1}'
> while read line
> do
> name="$(echo $line | jq -r .name)"
> logger -s "name=$name"
> hello="Hello, $name"
> logger -s "sent response"
> echo '{"hello":"'$hello'"}'
> done
> ---
>
> Things discussed:
>
> 1) remove the header {"openwhisk":1}
>
> Actually initially it was not there. But I decided to add this
> requirements because the action need to speak a protocol ANYWAY.
>
> Most important, I explain why I require it starts with "{"openwhisk: 1}".
>
> The main reason is: I start the child process at init time, and I wanted
> to detect when it does not behave properly.
>
> The simplest problem happens when the action crashes immediately. For
> example, a common reason for this problem is uploading a binary using some
> dynamic libraries not available in the runtime. For example a swift
> action. By defaults it load a lot of different libraries, it crashes
> immediately but I cannot detect it until I try to read its stdin.
>
> I can remove this requirement if someone can show me the go code to check
> that cmd.Start("true") or cmd.Start("pwd") exited 😃
>
> If it is not doable, and I skip the handshake, even if the command
> crashed, I will not detect the problem until a /run is executed and the
> action times out...
>
> Carlos say it is fine. It is ok for me but I still think an early problem
> detection would be better. Also James recommended me to provide as much as
> error detection to the user as early as possible. Kinda of conflicting
> directives here...
>
> Suggestions?
>
> 2) more checks at init time
>
> I added some sanity checks. Probably too many. I tried to detect the
> error at deployment time, not at invocation time.
>
> This is different from what currently for example dockerskeleton does.
>
> If I upload for example something wrong, like a non-zip, a non-elf
> executable, my init returns {"error": "description"}, while currently the
> dockerskeleton returns always OK.
>
> Recommendations here?
>
> 3) output to another channel the result
>
> Currently I require logs goes to stderr, and stdout is for interacting
> with the parent process.
>
> Rodric suggested to output to a separate channel (channel 3?) and use
> stdout and stderr for logs.
>
> While doable, I need to provision another pipe, and the implementation
> should probably do some syscalls to retrieve file descriptor 3. It would
> complicate implementation, while currently it is straightforward for any
> language that does not have a library available. For swift, even to flush
> stdout I needed to write "linux specific" code... I do not dare to think
> what I need to do to write in fd3...
>
> My opinion is that using stdout for I/O and stderr for logs is a better
> choice than opening another file descriptor.
>
> Thoughts here?
>
>
>
>
>
>
>
> --
> Michele Sciabarra
> openwhisk@sciabarra.com
>
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
Hi Vadim, your suggestion worked.
I was able to simplify the code a bit, but thanks to this I can remove the handshake while detecting immediately crashing executable. Fo the records, here is the code and the test:
// startAndCheck
func startAndCheck(cmd *exec.Cmd) error {
//fmt.Println(cmd.Path)
err := cmd.Start()
if err != nil {
return err
}
ch := make(chan error)
go func() { ch <- cmd.Wait() }()
select {
case <-ch:
return fmt.Errorf("command exited")
case <-time.After(100 * time.Millisecond):
return nil
}
}
func Example_startAndCheck() {
// err
cmd := exec.Command("/does/not/exists")
cmd.StdinPipe()
cmd.StdoutPipe()
fmt.Println(startAndCheck(cmd))
// immediate exit
cmd = exec.Command("/bin/true")
cmd.StdinPipe()
cmd.StdoutPipe()
fmt.Println(startAndCheck(cmd))
// immediate exit with output
cmd = exec.Command("/bin/pwd")
cmd.StdinPipe()
cmd.StdoutPipe()
fmt.Println(startAndCheck(cmd))
// unwanted banner
cmd = exec.Command("/usr/bin/bc")
cmd.StdinPipe()
cmd.StderrPipe()
fmt.Println(startAndCheck(cmd))
// pipe loop
cmd = exec.Command("/bin/cat")
cmd.StdinPipe()
cmd.StderrPipe()
fmt.Println(startAndCheck(cmd))
// Output:
// fork/exec /does/not/exists: no such file or directory
// command exited
// command exited
// <nil>
// <nil>
}
--
Michele Sciabarra
michele@sciabarra.com
On Sat, Mar 10, 2018, at 9:47 PM, Vadim Raskin wrote:
> >> So, how can I check the process is actually terminated ?
>
> Hi Michele,
>
> what about using cmd.Wait() to check whether the process exited after you
> started a go action? It will return a non-nil in case of error, otherwise
> blocks forever waiting the process to finish. Waiting for a reasonable
> amount of time to make sure that process doesn't exit and close the /init
> call afterwards, would it cover the case you mentioned?
>
> Just to make sure we talk about the same thing:
>
> errorChan := chan string
>
> cmd := exec.Command("userBinary")
> cmd.Start()
> // wait for failure to happen
> go func(){
> err := cmd.Wait()
> if(err != nil){
> errorChan <- "exited with failure"
> }
> else {
> close(errorChan)
> }
> }()
> // wait at most 10ms for an error to happen
> go func(){
> time.Sleep(10 * time.Millisecond)
> errorChan <- "happy"
> }
> runResult <- errorChan
> // further processing.
>
> I haven't tested this code, it also needs to do some clean up of the Wait
> go routine, but hopefully the logic is clear.
>
> regards,
> Vadim.
>
> On Fri, Mar 9, 2018 at 3:18 PM Michele Sciabarra <mi...@sciabarra.com>
> wrote:
>
> > > I would prefer it not be there, but can see the convenience of detecting
> > > that an app has immediately crashed. If we can find another way to do
> > > that via process inspection, that would be better in my view.
> > >
> > The problem can be summarised into this code:
> >
> > // this command exits
> > cmd := exec.Command("true")
> > out, err := cmd.StdoutPipe()
> >
> > err = cmd.Start()
> > fmt.Println(err)
> > // this is nil! no error!
> >
> > // even worse! attempted to detect
> > err = cmd.Process.Signal(syscall.Signal(0))
> > // this is nil too! no error!
> > fmt.Println(err)
> >
> > So, how can I check the process is actually terminated ?
> >
Re: [LONG] Discussing my implementation of Go actions
Posted by Carlos Santana <cs...@gmail.com>.
Thanks Vadim for helping out Michele
Michele excellent work and thanks for posting the dev list.
I agree with Rodric
> Functions with their own builtin proxy (go, swift, rust, and also has
less copying of data at runtime)
For max performance actions would need built in proxies.
For Swift I have being looking into kitura and vapor for this.
For Java I think we kind already have since it’s single JVM for
proxy&function. So any optimization to achieve that fastest init on jvm is
welcome and Param is looking into that.
Init needs to be fast to the point of noop/async from the openwhisk invoker
perspective, then on top is user land on what they want to do on init that
is not to be done on every subsequent run.
—Carlos
On Sun, Mar 11, 2018 at 9:02 AM Rodric Rabbah <ro...@gmail.com> wrote:
> Continuing on my earlier post about the native function interface, and
> following up on the latest changes as suggest by Vadim, i wanted to also
> point out that the /init protocol today is synchronous, whereas to do
> validation on the binary or to exec a new process, it’s worth considering
> an asynchronous handoff.
>
> So one thought is to augment the protocol to allow /init to initiate the
> new binary (load it, exec it) and a /ready to poll which waits for a
> response. This bring you closer to the original prototype where you exec’ed
> the binary.
>
> It also makes for a consistent protocol for a native Functions with their
> own builtin proxy (go, swift, rust, and also has less copying of data at
> runtime), and later can be adapted for a native function like a bash script
> which needs a proxy to relay payloads and results.
> The /init equivalent for the latter is starting the script, and /ready is
> waiting for {openwhisk: 1}.
>
> -r
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
At this stage, I think for the "openwhisk protocol 0" (that I am going to write down and document), I plan to just follow your suggestions:
- get rid of the handshake
- output to channel 3 ALSO for Go (it solves a possibly non deterministic section in the code)
For future expansions we can think of the protocol version again, especially for streaming, where the whole initialization and termination is more important (they become long running processes)
--
Michele Sciabarra
michele@sciabarra.com
On Sun, Mar 11, 2018, at 2:02 PM, Rodric Rabbah wrote:
> Continuing on my earlier post about the native function interface, and
> following up on the latest changes as suggest by Vadim, i wanted to also
> point out that the /init protocol today is synchronous, whereas to do
> validation on the binary or to exec a new process, it’s worth
> considering an asynchronous handoff.
>
> So one thought is to augment the protocol to allow /init to initiate the
> new binary (load it, exec it) and a /ready to poll which waits for a
> response. This bring you closer to the original prototype where you
> exec’ed the binary.
>
> It also makes for a consistent protocol for a native Functions with
> their own builtin proxy (go, swift, rust, and also has less copying of
> data at runtime), and later can be adapted for a native function like a
> bash script which needs a proxy to relay payloads and results.
> The /init equivalent for the latter is starting the script, and /ready
> is waiting for {openwhisk: 1}.
>
> -r
Re: [LONG] Discussing my implementation of Go actions
Posted by Rodric Rabbah <ro...@gmail.com>.
Continuing on my earlier post about the native function interface, and following up on the latest changes as suggest by Vadim, i wanted to also point out that the /init protocol today is synchronous, whereas to do validation on the binary or to exec a new process, it’s worth considering an asynchronous handoff.
So one thought is to augment the protocol to allow /init to initiate the new binary (load it, exec it) and a /ready to poll which waits for a response. This bring you closer to the original prototype where you exec’ed the binary.
It also makes for a consistent protocol for a native Functions with their own builtin proxy (go, swift, rust, and also has less copying of data at runtime), and later can be adapted for a native function like a bash script which needs a proxy to relay payloads and results.
The /init equivalent for the latter is starting the script, and /ready is waiting for {openwhisk: 1}.
-r
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <op...@sciabarra.com>.
Wow! A channel and 2 go routines for that checking a process exited. I
wrote a similar code except I did not think to using a timeout to always
unlock the channel. So yes, that could work thanks. I posted here
exactly to get a help like this. Will try.
--
Michele Sciabarra
openwhisk@sciabarra.com
On Sat, Mar 10, 2018, at 9:47 PM, Vadim Raskin wrote:
> >> So, how can I check the process is actually terminated ?
>
> Hi Michele,
>
> what about using cmd.Wait() to check whether the process exited
> after you> started a go action? It will return a non-nil in case of error,
> otherwise> blocks forever waiting the process to finish. Waiting for a reasonable> amount of time to make sure that process doesn't exit and close
> the /init> call afterwards, would it cover the case you mentioned?
>
> Just to make sure we talk about the same thing:
>
> errorChan := chan string
>
> cmd := exec.Command("userBinary")
> cmd.Start()
> // wait for failure to happen
> go func(){
> err := cmd.Wait()
> if(err != nil){
> errorChan <- "exited with failure"
> }
> else {
> close(errorChan)
> }
> }()
> // wait at most 10ms for an error to happen
> go func(){
> time.Sleep(10 * time.Millisecond)
> errorChan <- "happy"
> }
> runResult <- errorChan
> // further processing.
>
> I haven't tested this code, it also needs to do some clean up of
> the Wait> go routine, but hopefully the logic is clear.
>
> regards,
> Vadim.
>
> On Fri, Mar 9, 2018 at 3:18 PM Michele Sciabarra
> <mi...@sciabarra.com>> wrote:
>
> > > I would prefer it not be there, but can see the convenience of
> > > detecting> > > that an app has immediately crashed. If we can find another way
> > > to do> > > that via process inspection, that would be better in my view.
> > >
> > The problem can be summarised into this code:
> >
> > // this command exits
> > cmd := exec.Command("true")
> > out, err := cmd.StdoutPipe()
> >
> > err = cmd.Start()
> > fmt.Println(err)
> > // this is nil! no error!
> >
> > // even worse! attempted to detect
> > err = cmd.Process.Signal(syscall.Signal(0))
> > // this is nil too! no error!
> > fmt.Println(err)
> >
> > So, how can I check the process is actually terminated ?
> >
Re: [LONG] Discussing my implementation of Go actions
Posted by Vadim Raskin <ra...@gmail.com>.
>> So, how can I check the process is actually terminated ?
Hi Michele,
what about using cmd.Wait() to check whether the process exited after you
started a go action? It will return a non-nil in case of error, otherwise
blocks forever waiting the process to finish. Waiting for a reasonable
amount of time to make sure that process doesn't exit and close the /init
call afterwards, would it cover the case you mentioned?
Just to make sure we talk about the same thing:
errorChan := chan string
cmd := exec.Command("userBinary")
cmd.Start()
// wait for failure to happen
go func(){
err := cmd.Wait()
if(err != nil){
errorChan <- "exited with failure"
}
else {
close(errorChan)
}
}()
// wait at most 10ms for an error to happen
go func(){
time.Sleep(10 * time.Millisecond)
errorChan <- "happy"
}
runResult <- errorChan
// further processing.
I haven't tested this code, it also needs to do some clean up of the Wait
go routine, but hopefully the logic is clear.
regards,
Vadim.
On Fri, Mar 9, 2018 at 3:18 PM Michele Sciabarra <mi...@sciabarra.com>
wrote:
> > I would prefer it not be there, but can see the convenience of detecting
> > that an app has immediately crashed. If we can find another way to do
> > that via process inspection, that would be better in my view.
> >
> The problem can be summarised into this code:
>
> // this command exits
> cmd := exec.Command("true")
> out, err := cmd.StdoutPipe()
>
> err = cmd.Start()
> fmt.Println(err)
> // this is nil! no error!
>
> // even worse! attempted to detect
> err = cmd.Process.Signal(syscall.Signal(0))
> // this is nil too! no error!
> fmt.Println(err)
>
> So, how can I check the process is actually terminated ?
>
Re: [LONG] Discussing my implementation of Go actions
Posted by Michele Sciabarra <mi...@sciabarra.com>.
> I would prefer it not be there, but can see the convenience of detecting
> that an app has immediately crashed. If we can find another way to do
> that via process inspection, that would be better in my view.
>
The problem can be summarised into this code:
// this command exits
cmd := exec.Command("true")
out, err := cmd.StdoutPipe()
err = cmd.Start()
fmt.Println(err)
// this is nil! no error!
// even worse! attempted to detect
err = cmd.Process.Signal(syscall.Signal(0))
// this is nil too! no error!
fmt.Println(err)
So, how can I check the process is actually terminated ?
Re: [LONG] Discussing my implementation of Go actions
Posted by Rob Allen <ro...@akrabat.com>.
Hey Michele,
Great to see your work become a PR.
> On 9 Mar 2018, at 11:13, Michele Sciabarra <op...@sciabarra.com> wrote:
>
> 1) remove the header {"openwhisk":1}
>
> Actually initially it was not there. But I decided to add this requirements because the action need to speak a protocol ANYWAY.
>
> Most important, I explain why I require it starts with "{"openwhisk: 1}".
>
> The main reason is: I start the child process at init time, and I wanted to detect when it does not behave properly.
>
> The simplest problem happens when the action crashes immediately. For example, a common reason for this problem is uploading a binary using some dynamic libraries not available in the runtime. For example a swift action. By defaults it load a lot of different libraries, it crashes immediately but I cannot detect it until I try to read its stdin.
>
> I can remove this requirement if someone can show me the go code to check that cmd.Start("true") or cmd.Start("pwd") exited 😃
>
> If it is not doable, and I skip the handshake, even if the command crashed, I will not detect the problem until a /run is executed and the action times out...
>
> Carlos say it is fine. It is ok for me but I still think an early problem detection would be better. Also James recommended me to provide as much as error detection to the user as early as possible. Kinda of conflicting directives here...
>
> Suggestions?
I would prefer it not be there, but can see the convenience of detecting that an app has immediately crashed. If we can find another way to do that via process inspection, that would be better in my view.
> 2) more checks at init time
>
> I added some sanity checks. Probably too many. I tried to detect the error at deployment time, not at invocation time.
>
> This is different from what currently for example dockerskeleton does.
>
> If I upload for example something wrong, like a non-zip, a non-elf executable, my init returns {"error": "description"}, while currently the dockerskeleton returns always OK.
>
> Recommendations here?
Personally, I'm in favour of detecting early when possible. I think most of the language specific runtimes do sanity checking in /init.
> 3) output to another channel the result
>
> Currently I require logs goes to stderr, and stdout is for interacting with the parent process.
>
> Rodric suggested to output to a separate channel (channel 3?) and use stdout and stderr for logs.
>
> While doable, I need to provision another pipe, and the implementation should probably do some syscalls to retrieve file descriptor 3. It would complicate implementation, while currently it is straightforward for any language that does not have a library available. For swift, even to flush stdout I needed to write "linux specific" code... I do not dare to think what I need to do to write in fd3...
>
> My opinion is that using stdout for I/O and stderr for logs is a better choice than opening another file descriptor.
In my view, stdout for the response and stderr for any logs is easy enough to understand. Writing to a third pipe would increase the complexity for people writing an `exec` app.
Ideally, you'd take the last line of stdout only for the response so that any app that print outs a welcome banner to stdout would work. I appreciate this is hard to determine when in a loop though!
Regards,
Rob...