You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@openwhisk.apache.org by dg...@apache.org on 2018/07/07 16:35:26 UTC
[incubator-openwhisk] branch master updated: Document adding a new
runtime kind. (#3837)
This is an automated email from the ASF dual-hosted git repository.
dgrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-openwhisk.git
The following commit(s) were added to refs/heads/master by this push:
new 9c05de4 Document adding a new runtime kind. (#3837)
9c05de4 is described below
commit 9c05de4494d6edbd198fd494d458b908bbc0cd0a
Author: rodric rabbah <ro...@gmail.com>
AuthorDate: Sat Jul 7 12:35:23 2018 -0400
Document adding a new runtime kind. (#3837)
Document the steps required to add a new runtime kind to the main repo so that it is available to the OpenWhisk deployment.
---
docs/actions-new.md | 251 ++++++++++++++++++++++++++++++++++++++++++++++++++
docs/actions-swift.md | 60 ++++++------
docs/actions.md | 15 ++-
3 files changed, 291 insertions(+), 35 deletions(-)
diff --git a/docs/actions-new.md b/docs/actions-new.md
new file mode 100644
index 0000000..7ae6b16
--- /dev/null
+++ b/docs/actions-new.md
@@ -0,0 +1,251 @@
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+OpenWhisk supports [several languages and runtimes](actions.md#languages-and-runtimes) but
+there may be other languages or runtimes that are important for your organization, and for
+which you want tighter integration with the platform. The OpenWhisk platform is extensible
+and you can add new languages or runtimes (with custom packages and third-party dependencies)
+following the guide described here.
+
+### Unit of execution
+The unit of execution for all functions is a [Docker container](https://docs.docker.com).
+The container implements a specific interface which:
+1. accepts an initialization payload (the code) and prepared for execution,
+2. accepts runtime payload (the input parameters) and
+ - prepares the activation context,
+ - runs the function,
+ - returns the function result,
+3. flushes all `stdout` and `stderr` logs and adds a frame marker at the end of the activation.
+
+Any container which implements [the interface](#action-interface) may be used as an action.
+It is in this way that you can add support for other languages or customized runtimes.
+
+The interface is enforced via a [canonical test suite](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+which validates the initialization protocol, the runtime protocol, ensures the activation context is correctly prepared,
+and that the logs are properly framed. Your runtime should extend this test suite, and of course include additional tests
+as needed.
+
+The runtime support is best implemented in its own repository to permit a management
+lifecycle independent of the rest of the OpenWhisk platform which requires the following
+additions:
+1. introduce the runtime specification into the [runtimes manifest](../ansible/files/runtimes.json),
+2. add a new `actions-<your runtime>.md` file to the [docs](.) directory,
+3. add a link to your new language or runtime to the [top level index](actions.md#languages-and-runtimes),
+4. add the runtime to the [Swagger file](../core/controller/src/main/resources/apiv1swagger.json),
+5. add a standard test action to the [tests artifacts directory](../tests/dat/actions/unicode.tests).
+
+### The runtime manifest
+
+Actions when created specify the desired runtime for the function via a property called "kind".
+When using the `wsk` CLI, this is specified as `--kind <runtime-kind>`. The value is a typically
+a string describing the language (e.g., `nodejs`) followed by a colon and the version for the runtime
+as in `nodejs:8` or `php:7.2`.
+
+The manifest is a map of runtime family names to an array of specific kinds. The details of the
+schema are found in the [Exec Manifest](../common/scala/src/main/scala/whisk/core/entity/ExecManifest.scala).
+As an example, the following entry add a new runtime family called `nodejs` with a single kind
+`nodejs:6`.
+
+```json
+{
+ "nodejs": [{
+ "kind": "nodejs:6",
+ "default": true,
+ "image": {
+ "prefix": "openwhisk",
+ "name": "nodejs6action",
+ "tag": "latest"
+ }
+ }]
+}
+```
+
+The `default` property indicates if the corresponding kind should be treated as the
+default for the runtime family. The `image` structure defines the Docker image name
+that is used for actions of this kind (e.g., `openwhisk/nodejs6action:latest` here).
+
+### The test action
+
+The standard test action is shown below in JavaScript. It should be adapted for the
+new language and added to the [test artifacts directory](../tests/dat/actions/unicode.tests)
+with the name `<runtime-kind>.txt` for plain text file or `<runtime-kind>.bin` for a
+a binary file. The `<runtime-kind>` must match the value used for `kind` in the corresponding
+runtime manifest entry.
+
+```js
+function main(args) {
+ var str = args.delimiter + " ☃ " + args.delimiter;
+ console.log(str);
+ return { "winter": str };
+}
+```
+
+### Canonical runtime repository
+
+The runtime repository should follow the canonical structure used by other runtimes.
+
+```
+/path/to/runtime
+├── build.gradle # Gradle build file to integrate with rest of build and test framework
+├── core
+│ └── <runtime name and version>
+│ ├── Dockerfile # builds the runtime's Docker image
+│ └── ... # your runtimes files which implement the action proxy
+└── tests
+ └── src # tests suits...
+ └── ... # ... which extend canonical interface plus additional runtime specific tests
+```
+
+The [Docker skeleton repository](https://github.com/apache/incubator-openwhisk-runtime-docker)
+is an example starting point to fork and modify for your new runtime.
+
+### Action Interface
+
+An action consists of the user function (and its dependencies) along with a _proxy_ that implements a
+canonical protocol to integrate with the OpenWhisk platform.
+
+The proxy is a web server with two endpoints.
+* It listens on port `8080`.
+* It implements `/init` to initialize the container.
+* It also implements `/run` to activate the function.
+
+The proxy also prepares the
+[execution context](actions.md#accessing-action-metadata-within-the-action-body),
+and flushes the logs produced by the function to stdout and stderr.
+
+#### Initialization
+
+The initialization route is `/init`. It must accept a `POST` request with a JSON object as follows:
+```
+{
+ "value": {
+ "name" : String,
+ "main" : String,
+ "code" : String,
+ "binary": Boolean
+ }
+}
+```
+
+* `name` is the name of the action.
+* `main` is the name of the function to execute.
+* `code` is either plain text or a base64 encoded string for binary functions (i.e., a compiled executable).
+* `binary` is false if `code` is in plain text, and true if `code` is base64 encoded.
+
+The initialization route is called exactly once by the OpenWhisk platform, before executing a function.
+The route should report an error if called more than once. It is possible however that a single initialization
+will be followed by many activations (via `/run`).
+
+**Successful initialization:** The route should respond with `200 OK` if the initialization is successful and
+the function is ready to execute. Any content provided in the response is ignored.
+
+**Failures to initialize:** Any response other than `200 OK` is treated as an error to initialize. The response
+from the handler if provided must be a JSON object with a single field called `error` describing the failure.
+The value of the error field may be any valid JSON value. The proxy should make sure to generate meaningful log
+message on failure to aid the end user in understanding the failure.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 seconds). The initialization
+must complete within the allowed duration. Failure to complete initialization within the allowed time frame
+will destroy the container.
+
+**Limitation:** The proxy does not currently receive any of the activation context at initialization time.
+There are scenarios where the context is convenient if present during initialization. This will require a
+change in the OpenWhisk platform itself. Note that even if the context is available during initialization,
+it must be reset with every new activation since the information will change with every execution.
+
+#### Activation
+
+The proxy is ready to execute a function once it has successfully completed initialization. The OpenWhisk
+platform will invoke the function by posting an HTTP request to `/run` with a JSON object providing a new
+activation context and the input parameters for the function. There may be many activations of the same
+function against the same proxy (viz. container). Currently, the activations are guaranteed not to overlap
+— that is, at any given time, there is at most one request to `/run` from the OpenWhisk platform.
+
+The route must accept a JSON object and respond with a JSON object, otherwise the OpenWhisk platform will
+treat the activation as a failure and proceed to destroy the container. The JSON object provided by the
+platform follows the following schema:
+```
+{
+ "value": JSON,
+ "namespace": String,
+ "action_name": String,
+ "api_host": String,
+ "api_key": String,
+ "activation_id": String,
+ "deadline": Number
+}
+```
+
+* `value` is a JSON object and contains all the parameters for the function activation.
+* `namespace` is the OpenWhisk namespace for the action (e.g., `whisk.system`).
+* `action_name` is the [fully qualified name](reference.md#fully-qualified-names) of the action.
+* `activation_id` is a unique ID for this activation.
+* `deadline` is the deadline for the function.
+* `api_key` is the API key used to invoke the action.
+
+The `value` is the function parameters. The rest of the properties become part of the activation context
+which is a set of environment variables constructed by capitalizing each of the property names, and prefixing
+the result with `__OW_`. Additionally, the context must define `__OW_API_HOST` whose value
+is the OpenWhisk API host. This value is currently provided as an environment variable defined at container
+startup time and hence already available in the context.
+
+**Successful activation:** The route must respond with `200 OK` if the activation is successful and
+the function has produced a JSON object as its result. The response body is recorded as the [result
+of the activation](actions.md#understanding-the-activation-record).
+
+**Failed activation:** Any response other than `200 OK` is treated as an activation error. The response
+from the handler must be a JSON object with a single field called `error` describing the failure.
+The value of the error field may be any valid JSON value. Should the proxy fail to respond with a JSON
+object, the OpenWhisk platform will treat the failure as an uncaught exception. These two failures modes are
+distinguished by the value of the `response.status` in the [activation record](actions.md#understanding-the-activation-record)
+which is "application error" if the proxy returned an "error" object, and "action developer error" otherwise.
+
+**Time limit:** Every action in OpenWhisk has a defined time limit (e.g., 60 seconds). The activation
+must complete within the allowed duration. Failure to complete activation within the allowed time frame
+will destroy the container.
+
+#### Logs
+
+The proxy must flush all the logs produced during initialization and execution and add a frame marker
+to denote the end of the log stream for an activation. This is done by emitting the token
+[`XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX`](https://github.com/apache/incubator-openwhisk/blob/59abfccf91b58ee39f184030374203f1bf372f2d/core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainer.scala#L51)
+as the last log line for the `stdout` _and_ `stderr` streams. Failure to emit this marker will cause delayed
+or truncated activation logs.
+
+### Testing the new runtime
+
+There is a [canonical test harness](../tests/src/test/scala/actionContainers/BasicActionRunnerTests.scala)
+for validating a new runtime. The harness will performing the following:
+* Test the proxy can handle the identity functions (initialize and run).
+* Test the proxy can properly handle functions with Unicode characters.
+* Test the proxy properly constructs the activation context.
+* Test the proxy can handle large payloads (more than 1MB).
+* Test the proxy does not permit re-initialization.
+* Test the error handling for an action returning an invalid response.
+
+The canonical test suite should be extended by the new runtime tests. Additional
+tests will be required depending on the feature set provided by the runtime.
+
+Since the OpenWhisk platform is language and runtime agnostic, it is generally not
+necessary to add integration tests. That is the unit tests verifying the protocol are
+sufficient. However, it may be necessary in some cases to modify the `wsk` CLI or
+other OpenWhisk clients. In which case, appropriate tests should be added as necessary.
+The OpenWhisk platform will perform a generic integration test as part of its basic
+system tests. This integration test will require a [test function](#the-test-action) to
+be available so that the test harness can create, invoke, and delete the action.
diff --git a/docs/actions-swift.md b/docs/actions-swift.md
index 859583d..5db3687 100644
--- a/docs/actions-swift.md
+++ b/docs/actions-swift.md
@@ -100,9 +100,9 @@ wsk action invoke --result helloSwift --param name World
```
```json
- {
- "greeting": "Hello World!"
- }
+{
+ "greeting": "Hello World!"
+}
```
Find out more about parameters in the [Working with parameters](./parameters.md) section.
@@ -120,12 +120,12 @@ You can use a script to automate the packaging of the action. Create script `co
set -ex
if [ -z "$1" ] ; then
- echo 'Error: Missing action name'
- exit 1
+ echo 'Error: Missing action name'
+ exit 1
fi
if [ -z "$2" ] ; then
- echo 'Error: Missing kind, for example swift:4.1'
- exit 2
+ echo 'Error: Missing kind, for example swift:4.1'
+ exit 2
fi
OUTPUT_DIR="build"
if [ ${2} == "swift:3.1.1" ]; then
@@ -144,14 +144,14 @@ DEST_PACKAGE_SWIFT="$BASE_PATH/spm-build/Package.swift"
BUILD_FLAGS=""
if [ -n "$3" ] ; then
- BUILD_FLAGS=${3}
+ BUILD_FLAGS=${3}
fi
echo "Using runtime $RUNTIME to compile swift"
docker run --rm --name=compile-ow-swift -it -v "$(pwd):/owexec" $RUNTIME bash -ex -c "
if [ -f \"/owexec/$OUTPUT_DIR/$1.zip\" ] ; then
- rm \"/owexec/$OUTPUT_DIR/$1.zip\"
+ rm \"/owexec/$OUTPUT_DIR/$1.zip\"
fi
echo 'Setting up build...'
@@ -159,8 +159,8 @@ cp /owexec/actions/$1/Sources/*.swift $DEST_SOURCE/
# action file can be either {action name}.swift or main.swift
if [ -f \"$DEST_SOURCE/$1.swift\" ] ; then
- echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
- mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
+ echo 'renaming $DEST_SOURCE/$1.swift $DEST_SOURCE/main.swift'
+ mv \"$DEST_SOURCE/$1.swift\" $DEST_SOURCE/main.swift
fi
# Add in the OW specific bits
cat $BASE_PATH/epilogue.swift >> $DEST_SOURCE/main.swift
@@ -202,14 +202,14 @@ For Swift 3 here is an example:
import PackageDescription
let package = Package(
- name: "Action",
- dependencies: [
- .Package(url: "https://github.com/apple/example-package-deckofplayingcards.git", majorVersion: 3),
- .Package(url: "https://github.com/IBM-Swift/CCurl.git", "0.2.3"),
- .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.7.10"),
- .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "15.0.1"),
- .Package(url: "https://github.com/watson-developer-cloud/swift-sdk.git", "0.16.0")
- ]
+ name: "Action",
+ dependencies: [
+ .Package(url: "https://github.com/apple/example-package-deckofplayingcards.git", majorVersion: 3),
+ .Package(url: "https://github.com/IBM-Swift/CCurl.git", "0.2.3"),
+ .Package(url: "https://github.com/IBM-Swift/Kitura-net.git", "1.7.10"),
+ .Package(url: "https://github.com/IBM-Swift/SwiftyJSON.git", "15.0.1"),
+ .Package(url: "https://github.com/watson-developer-cloud/swift-sdk.git", "0.16.0")
+ ]
)
```
For Swift 4 here is an example:
@@ -220,20 +220,20 @@ For Swift 3 here is an example:
let package = Package(
name: "Action",
products: [
- .executable(
- name: "Action",
- targets: ["Action"]
- )
+ .executable(
+ name: "Action",
+ targets: ["Action"]
+ )
],
dependencies: [
- .package(url: "https://github.com/apple/example-package-deckofplayingcards.git", .upToNextMajor(from: "3.0.0"))
+ .package(url: "https://github.com/apple/example-package-deckofplayingcards.git", .upToNextMajor(from: "3.0.0"))
],
targets: [
- .target(
- name: "Action",
- dependencies: ["DeckOfPlayingCards"],
- path: "."
- )
+ .target(
+ name: "Action",
+ dependencies: ["DeckOfPlayingCards"],
+ path: "."
+ )
]
)
```
@@ -281,7 +281,7 @@ enum VendingMachineError: Error {
}
func main(param: Input, completion: (Output?, Error?) -> Void) -> Void {
// Return real error
- do{
+ do {
throw VendingMachineError.insufficientFunds(coinsNeeded: 5)
} catch {
completion(nil, error)
diff --git a/docs/actions.md b/docs/actions.md
index 0097b12..f365ccd 100644
--- a/docs/actions.md
+++ b/docs/actions.md
@@ -51,16 +51,21 @@ Longer tutorials that are specific to a language of your choice are listed below
We recommend reading the basics in this document first, which are language agnostic, before getting deeper
into a language-specific tutorial. If your preferred language isn't supported directly, you may find
the [Docker](docker-actions.md) action or [native binary](docker-actions.md#creating-native-actions)
-paths more suitable. Multiple actions may be composed together to create a longer processing pipeline called a
-[sequence](#creating-action-sequences). A more advanced form of composition is described [here](conductors.md).
+paths more suitable. Or, you can [create a new runtime](#actions-new.md).
-* [JavaScript](actions-node.md)
-* [Python](actions-python.md)
+* [Go](actions-go.md)
* [Java](actions-java.md)
+* [JavaScript](actions-node.md)
* [PHP](actions-php.md)
+* [Python](actions-python.md)
* [Swift](actions-swift.md)
* [Docker and native binaries](actions-docker.md)
-* [Go](actions-go.md)
+
+Multiple actions from different languages may be composed together to create a longer processing
+pipeline called a [sequence](#creating-action-sequences). The polyglot nature of the composition is
+powerful in that it affords you the ability to use the right language for the problem you're solving,
+and separates the orchestration of the dataflow between functions from the choice of language.
+A more advanced form of composition is described [here](conductors.md).
## The basics