Binary actions with a piped subprocess: benchmarking the implementation

[Michele Sciabarra <op...@sciabarra.com>]

Hello, Whiskers.

I already coded a first implementation of the generic binary action runner using the pipes. And I am happy to say it works and it is efficient. I was afraid it was not.

If there is a good reason for using Go is the awesome simplicity of the implementation using goroutines. Also Go provides out of the box the PipeStdIn and PipeStdout for commands so I do not have to fight with pipe creation  using unix syscalls.

I do not know how to do in python but I suspect you need to use async io and not just Flask, and it can be a bit more complex than this...

Code is not yet really ready because I need to do a better error checks, implementing unzip of the actions and fix an issue when I replace the action.  And I have not yet updated the documentatio. However I am eager to share the result of the benchmark.

+----------+-------+---------+--------+----------+----------+----------+-----+------+
|  Label   | Hits  | Average | Median | 90% Line | 95% Line | 99% Line | Min | Max  |
+----------+-------+---------+--------+----------+----------+----------+-----+------+
| Docker   | 10000 |    2043 |   2038 |     2396 |     2605 |     3031 |  94 | 4016 |
| GoServer | 10000 |       4 |      4 |        7 |        8 |       16 |   1 |   61 |
| PipeExec | 10000 |       5 |      5 |        9 |       11 |       20 |   2 |   13 |
| TOTAL    | 30000 |     684 |      6 |     2121 |     2258 |     2689 |   1 | 4016 |
+----------+-------+---------+--------+----------+----------+----------+-----+------+

Docker is using the current docker skeleton, GoServer is a native, all-in-one go server for init and run, that replaces itself. PipeExec is the new code that starts a subprocess and keep it running, feeding new requests in standard input and getting results in standard output, line by line.

Using a native, all-in one action executor the average is 4 milliseconds, while using pipes the average is 5 ms. I think losing one millisecond (mostly due, I think, to the fact I decode then re-encode the json)  is worth because of the added generality. This server should work with binaries written in any language. Minimum is 2 millisecond but strangely max is 13 ms while the max with a native server is 61.

Below there  is the code of the function, I used a slightly modified version of the example in the IBM Cloud documentation. You do not need to use any library, just follow the convention of reading ore line and writing one line, and log in stderr.

Note the example it can be used both in the current dockerskeleton and in my new implementation. If the first argument is provided, the action will run only once. If no arguments are provided it will start a read-write loop, reading a line, feeding to the action and outputting the result.


```
package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"os"
)

func hello(arg string) string {
	var obj map[string]interface{}
	json.Unmarshal([]byte(arg), &obj)
	name, ok := obj["name"].(string)
	if !ok {
		name = "Stranger"
	}
	msg := map[string]string{"message": ("Hello, " + name + "!")}
	res, _ := json.Marshal(msg)
	return string(res)
}

func main() {
	// native actions receive one argument, the JSON object as a string
	if len(os.Args) > 1 {
		fmt.Println(hello(os.Args[1]))
		return
	}
	// read loop
	reader := bufio.NewReader(os.Stdin)
	for {
		event, err := reader.ReadString('\n')
		if err != nil {
			break
		}
		fmt.Println(hello(event))
	}
}
```

Ok, there is still a consistent amount of work to do to manage misbehaving actions, I know. Working on it.


-- 
  Michele Sciabarra
  openwhisk@sciabarra.com

Re: Binary actions with a piped subprocess: benchmarking the implementation

[Rodric Rabbah <ro...@gmail.com>]

Excellent work!

You could use these tests down the line for emulating various behaviors and
failure modes:

https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/actionContainers/ActionProxyContainerTests.scala

We tried to have a canonical set of tests for new action containers in a
single place.

-r