You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bifromq.apache.org by po...@apache.org on 2026/09/29 04:08:23 UTC
(bifromq-tenon) branch main updated: Add an example script for better quick start. (#10)
This is an automated email from the ASF dual-hosted git repository.
popduke pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/bifromq-tenon.git
The following commit(s) were added to refs/heads/main by this push:
new fcea034 Add an example script for better quick start. (#10)
fcea034 is described below
commit fcea0346f37055aabeaaf45ede73c4babf754025
Author: Gu Jiawei <gu...@foxmail.com>
AuthorDate: Mon Sep 28 21:08:18 2026 -0700
Add an example script for better quick start. (#10)
---
guide/quickstart.md | 81 ++++---------
plugin/README.md | 2 +-
tools/example/local.sh | 316 +++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 343 insertions(+), 56 deletions(-)
diff --git a/guide/quickstart.md b/guide/quickstart.md
index d3ee125..99aac46 100644
--- a/guide/quickstart.md
+++ b/guide/quickstart.md
@@ -21,91 +21,62 @@ under the License.
Tenon is part of Apache BifroMQ (Incubating). See the [incubation disclaimer](../DISCLAIMER).
-This example builds from a source checkout. A generated dual-interface plugin sends one text record through Lua and writes it to a file, so no broker or published Tenon package is needed. Run on a supported native platform with the [build prerequisites](../README.md#build-and-try), Python 3 and curl. Use a free local port; the example uses 18080.
+This example builds from a source checkout and runs two independent Pipelines in one Runner. One Pipeline uses a generated source-and-sink plugin and writes a record to a file. The other uses the built-in Dummy Source and Stdout Sink, with a Lua timer producing output. This shows both the plugin scaffold path and prebuilt repository plugins; no broker or published Tenon package is needed. Run on a supported native platform with the [build prerequisites](../README.md#build-and-try), Pytho [...]
-## Build the Runner and tools
+## Run the complete example
From the Tenon source root:
```sh
-tenon_root="$PWD"
-cargo build --locked --bin tenon
-cargo install cargo-generate --version 0.24.0 --locked
-cargo install --path sdk/rust/cargo-tenon --locked
-demo_root="$(mktemp -d "${TMPDIR:-/tmp}/tenon-quickstart.XXXXXX")"
-cd "$demo_root"
-cargo generate --path "$tenon_root/sdk/rust/rust-plugin-scaffold" \
- --name hello-tenon --define interface=source-and-sink \
- --silent --vcs none --no-workspace
-cd hello-tenon
-cargo tenon bundle \
- --config "patch.crates-io.tenon-plugin-sdk.path=\"$tenon_root/sdk/rust/plugin-sdk\"" \
- --config "patch.crates-io.tenon-ipc.path=\"$tenon_root/sdk/rust/ipc\"" \
- > "$demo_root/bundle.json"
-bundle="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["bundle"])' "$demo_root/bundle.json")"
+tools/example/local.sh
```
-The local Cargo patches select the SDK and IPC code in this checkout. The bundle command reports the actual bundle path in JSON. Keep this shell open so the variables remain available.
+The script builds the Runner, generates and bundles the scaffold plugin, bundles the two built-in plugins, starts Tenon, installs all three Programs, saves two Documents, and checks both Pipelines. It installs `cargo-generate` 0.24.0 if the Cargo subcommand is missing. This provides the `cargo generate` command; installing the Rust toolchain alone does not install it. The script adds Cargo's binary directory to `PATH` so Cargo can find the generator.
-## Configure and start
+It writes the generated project, the three bundles, `runner.jsonc`, and two input Documents in a temporary directory. `runner.jsonc` tells Tenon where to keep local state and to listen on `127.0.0.1`. `hello.jsonc` defines one Instance of the generated source-and-sink Program and one Flow that binds it as both Source and Sink; Lua copies the Source message into its Sink payload. `builtins.jsonc` defines a second Pipeline with one Flow from Dummy Source to Stdout Sink; Lua's timer emits a [...]
+
+When the example succeeds, the script prints both outputs and follows the Runner log. Press Ctrl-C to stop the Runner. It leaves its temporary directory in place and prints its path so you can inspect the generated project, Documents, status snapshots, bundles, logs and output. The Runner listens only on localhost. No Document cleanup is needed because stopping the temporary Runner ends this example.
+
+If port 18080 is already in use, choose another port:
```sh
-python3 - "$demo_root" <<'PYTHON'
-import json, pathlib, sys
-root = pathlib.Path(sys.argv[1])
-(root / "runner.jsonc").write_text(json.dumps({
- "stateDirectory": str(root / "state"),
- "http": {"listenAddress": "127.0.0.1:18080"},
- "pipeline": {"retryBackoff": {"initialDelayMs": 100, "maximumDelayMs": 30000}},
- "lua": {"cpuTimeLimitMs": 50, "memoryLimitBytes": 16777216}
-}))
-(root / "hello.jsonc").write_text(json.dumps({
- "specVersion": "1", "id": "hello",
- "pluginInstances": {"example": {
- "programName": "com.example.hello-tenon", "exactVersion": "0.1.0",
- "config": {"message": "Hello Tenon", "outputFile": str(root / "output.txt")}
- }},
- "flows": {"main": {
- "source": "example", "sinks": ["example"],
- "process": {"script": 'local b = registry:getBuilder("com.example.hello-tenon@0.1.0")\nfunction main(event)\n b:setMessage(event.payload.message)\n emit(b:build())\nend'}
- }}
-}))
-PYTHON
-"$tenon_root/target/debug/tenon" --config "$demo_root/runner.jsonc" \
- > "$demo_root/runner.log" 2>&1 &
-runner_pid=$!
+TENON_PORT=18081 tools/example/local.sh
```
-Wait until this request succeeds; if the process exits, inspect `runner.log`:
+## Manual steps
+
+To run each stage yourself, first build the Runner and install the two Cargo tools:
```sh
-curl --fail http://127.0.0.1:18080/openapi.json > "$demo_root/openapi.json"
+cargo build --locked --bin tenon
+cargo install cargo-generate --version 0.24.0 --locked
+cargo install --path sdk/rust/cargo-tenon --locked
```
-## Install and run
+`cargo generate` is provided by `cargo-generate`. The install command adds that executable to Cargo's bin directory. Then follow the script's steps: generate the scaffold, bundle it against the local SDK, bundle the two built-in plugins, write `runner.jsonc`, `hello.jsonc` and `builtins.jsonc`, start the Runner, install the three Programs, and PUT both Documents. The script at [tools/example/local.sh](../tools/example/local.sh) contains the exact commands and example JSON.
+
+Once the Runner is running, the essential API calls are:
```sh
curl --fail -i -X POST http://127.0.0.1:18080/plugins \
-H 'Content-Type: application/vnd.apache.tenon.plugin+tar+gzip' \
- --data-binary "@$bundle"
+ --data-binary @/absolute/path/to/hello-tenon.tar.gz
curl --fail -i -X PUT http://127.0.0.1:18080/documents/hello \
-H 'Content-Type: application/jsonc' -H 'If-None-Match: *' \
- --data-binary "@$demo_root/hello.jsonc"
+ --data-binary @/absolute/path/to/hello.jsonc
curl --fail http://127.0.0.1:18080/pipelines/hello
-cat "$demo_root/output.txt"
+cat /absolute/path/to/output.txt
```
Installation and creation return 201. Application is asynchronous: repeat the status read until `state` is `running`, `appliedDocumentEtag` equals `documentEtag`, and the plugin is running. The output file must contain `Hello Tenon`. Wait until the file appears and contains that line. Each fresh plugin process emits the example message, so a restart can append another line.
-## Stop and clean up
+The Document PUT response confirms it was saved, not that the Pipeline has applied it. Poll `GET /pipelines/hello` until the state is `running`, `appliedDocumentEtag` matches `documentEtag`, and the plugin instance is running. Then check the output file. Replace the example absolute paths above with the paths where you generated those files.
+
+To stop a manually started Runner, send it SIGTERM and wait for it to exit:
```sh
-etag="$(curl --fail -sS -D - -o /dev/null http://127.0.0.1:18080/documents/hello \
- | tr -d '\r' | sed -n 's/^[Ee][Tt][Aa][Gg]: //p')"
-curl --fail -i -X DELETE http://127.0.0.1:18080/documents/hello \
- -H "If-Match: $etag"
kill -TERM "$runner_pid"
wait "$runner_pid"
```
-The Document is removed and the Runner stops its processes. The temporary directory retains the package, logs, configuration and result for inspection; remove it when no longer needed. A deployed system should use its service manager and an appropriate persistent state directory instead.
+A deployed system should use its service manager and an appropriate persistent state directory instead.
diff --git a/plugin/README.md b/plugin/README.md
index 5a9bcab..395803f 100644
--- a/plugin/README.md
+++ b/plugin/README.md
@@ -47,7 +47,7 @@ cargo tenon bundle --locked \
These are native debug builds. Use `--release` for distribution and build a bundle for each intended platform. See [distribution guidance](../guide/plugins.md#build-artifacts-for-distribution).
-Start a local Runner using the [quickstart configuration](../guide/quickstart.md#configure-and-start), or use your existing development Runner. The following examples assume `http://127.0.0.1:18080`; supply your deployment's authentication headers when required.
+Start a local Runner with the [quickstart script](../guide/quickstart.md#run-the-complete-example), or use your existing development Runner. The following examples assume `http://127.0.0.1:18080`; supply your deployment's authentication headers when required.
```sh
for report in /tmp/tenon-dummy-bundle.json /tmp/tenon-stdout-bundle.json; do
diff --git a/tools/example/local.sh b/tools/example/local.sh
new file mode 100755
index 0000000..7c54663
--- /dev/null
+++ b/tools/example/local.sh
@@ -0,0 +1,316 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+tenon_root="$(cd -- "$script_dir/../.." && pwd)"
+
+if ! command -v cargo >/dev/null 2>&1; then
+ printf 'cargo is not on PATH. Install Rust using the project toolchain, then retry.\n' >&2
+ exit 1
+fi
+if ! command -v python3 >/dev/null 2>&1; then
+ printf 'python3 is required to prepare the example Document.\n' >&2
+ exit 1
+fi
+if ! command -v curl >/dev/null 2>&1; then
+ printf 'curl is required to call the local Runner API.\n' >&2
+ exit 1
+fi
+
+# Cargo locates installed subcommands through PATH. IDEs can invoke Cargo by
+# absolute path while omitting its sibling binaries from PATH.
+cargo_bin="$(command -v cargo)"
+export PATH="$(dirname -- "$cargo_bin"):$PATH"
+
+if ! cargo generate --version >/dev/null 2>&1; then
+ printf 'Installing the scaffold-required cargo-generate 0.24.0...\n'
+ cargo install cargo-generate --version 0.24.0 --locked
+fi
+
+port="${TENON_PORT:-18080}"
+case "$port" in
+ ''|*[!0-9]*)
+ printf 'TENON_PORT must be a port number.\n' >&2
+ exit 1
+ ;;
+esac
+
+demo_root="$(mktemp -d "${TMPDIR:-/tmp}/tenon-quickstart.XXXXXX")"
+bundle_json="$demo_root/bundle.json"
+runner_pid=''
+diagnostics_pid=''
+
+cleanup() {
+ if [[ -n "$diagnostics_pid" ]] && kill -0 "$diagnostics_pid" 2>/dev/null; then
+ kill "$diagnostics_pid" 2>/dev/null || true
+ wait "$diagnostics_pid" 2>/dev/null || true
+ fi
+ if [[ -n "$runner_pid" ]] && kill -0 "$runner_pid" 2>/dev/null; then
+ kill -TERM "$runner_pid" 2>/dev/null || true
+ wait "$runner_pid" 2>/dev/null || true
+ fi
+ printf '\nQuickstart files are kept at: %s\n' "$demo_root"
+}
+trap cleanup EXIT
+trap 'exit 130' INT TERM
+
+printf 'Building the Tenon Runner...\n'
+cargo build --locked --bin tenon --manifest-path "$tenon_root/Cargo.toml"
+
+printf 'Generating the example source-and-sink plugin...\n'
+cargo generate --path "$tenon_root/sdk/rust/rust-plugin-scaffold" \
+ --name hello-tenon --define interface=source-and-sink \
+ --silent --vcs none --no-workspace --destination "$demo_root"
+
+printf 'Bundling the plugin against the SDK in this checkout...\n'
+cargo run --manifest-path "$tenon_root/sdk/rust/Cargo.toml" \
+ --package cargo-tenon -- bundle \
+ --manifest-path "$demo_root/hello-tenon/Cargo.toml" \
+ --config "patch.crates-io.tenon-plugin-sdk.path=\"$tenon_root/sdk/rust/plugin-sdk\"" \
+ --config "patch.crates-io.tenon-ipc.path=\"$tenon_root/ipc/rust/tenon-ipc\"" \
+ > "$bundle_json"
+bundle="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["bundle"])' "$bundle_json")"
+
+printf 'Bundling the built-in Dummy Source and Stdout Sink...\n'
+builtin_bundles=()
+for plugin in dummy-source stdout-sink; do
+ case "$plugin" in
+ dummy-source) manifest="$tenon_root/plugin/bifromq-tenon-dummy-source-plugin/Cargo.toml" ;;
+ stdout-sink) manifest="$tenon_root/plugin/bifromq-tenon-stdout-sink-plugin/Cargo.toml" ;;
+ esac
+ report="$demo_root/$plugin-bundle.json"
+ cargo run --manifest-path "$tenon_root/sdk/rust/Cargo.toml" \
+ --package cargo-tenon -- bundle \
+ --manifest-path "$manifest" \
+ --config "patch.crates-io.tenon-plugin-sdk.path=\"$tenon_root/sdk/rust/plugin-sdk\"" \
+ --config "patch.crates-io.tenon-ipc.path=\"$tenon_root/ipc/rust/tenon-ipc\"" \
+ > "$report"
+ builtin_bundles+=("$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["bundle"])' "$report")")
+done
+
+printf 'Preparing the local Runner configuration and example Document...\n'
+python3 - "$demo_root" "$port" <<'PYTHON'
+import json
+import pathlib
+import sys
+
+root = pathlib.Path(sys.argv[1])
+port = int(sys.argv[2])
+(root / "runner.jsonc").write_text(json.dumps({
+ "stateDirectory": str(root / "state"),
+ "http": {"listenAddress": f"127.0.0.1:{port}"},
+ "pipeline": {"retryBackoff": {"initialDelayMs": 100, "maximumDelayMs": 30000}},
+ "lua": {"cpuTimeLimitMs": 50, "memoryLimitBytes": 16777216},
+}))
+(root / "hello.jsonc").write_text(json.dumps({
+ "specVersion": "1",
+ "id": "hello",
+ "pluginInstances": {"example": {
+ "programName": "com.example.hello-tenon",
+ "exactVersion": "0.1.0",
+ "config": {
+ "message": "Hello Tenon",
+ "outputFile": str(root / "output.txt"),
+ },
+ }},
+ "flows": {"main": {
+ "source": "example",
+ "sinks": ["example"],
+ "process": {"script": (
+ 'local b = registry:getBuilder("com.example.hello-tenon@0.1.0")\n'
+ 'function main(event)\n'
+ ' b:setMessage(event.payload.message)\n'
+ ' emit(b:build())\n'
+ 'end'
+ )},
+ }},
+}))
+(root / "builtins.jsonc").write_text(json.dumps({
+ "specVersion": "1",
+ "id": "builtins",
+ "pluginInstances": {
+ "idle": {
+ "programName": "org.apache.bifromq.tenon.dummy-source",
+ "exactVersion": "0.1.0",
+ "config": {},
+ },
+ "console": {
+ "programName": "org.apache.bifromq.tenon.stdout-sink",
+ "exactVersion": "0.1.0",
+ "config": {},
+ },
+ },
+ "flows": {"probe": {
+ "source": "idle",
+ "sinks": ["console"],
+ "process": {"script": (
+ 'local b = registry:getBuilder("org.apache.bifromq.tenon.stdout-sink@0.1.0")\n'
+ 'local count = 0\n'
+ 'setTimeout(1000)\n'
+ 'function main(event)\n'
+ ' if event.type == "timer" then\n'
+ ' count = count + 1\n'
+ ' b:setMessage("quickstart " .. count)\n'
+ ' emit(b:build())\n'
+ ' setTimeout(1000)\n'
+ ' end\n'
+ 'end'
+ )},
+ }},
+}))
+PYTHON
+
+api="http://127.0.0.1:$port"
+if curl --silent --fail --max-time 1 "$api/openapi.json" -o /dev/null 2>/dev/null; then
+ printf 'A Tenon Runner already answers on port %s. Set TENON_PORT to a free port.\n' \
+ "$port" >&2
+ exit 1
+fi
+
+printf 'Starting Tenon at %s...\n' "$api"
+"$tenon_root/target/debug/tenon" --config "$demo_root/runner.jsonc" \
+ > "$demo_root/runner.log" 2>&1 &
+runner_pid=$!
+
+runner_ready=0
+for _ in $(seq 1 60); do
+ if ! kill -0 "$runner_pid" 2>/dev/null; then
+ break
+ fi
+ if curl --silent --fail --max-time 1 "$api/openapi.json" \
+ -o "$demo_root/openapi.json" 2>/dev/null; then
+ runner_ready=1
+ break
+ fi
+ sleep 1
+done
+if [[ "$runner_ready" != 1 ]]; then
+ printf 'Runner did not become ready. See %s/runner.log\n' "$demo_root" >&2
+ exit 1
+fi
+
+printf 'Installing the generated and built-in Programs...\n'
+for package in "$bundle" "${builtin_bundles[@]}"; do
+ curl --silent --show-error --fail -X POST "$api/plugins" \
+ -H 'Content-Type: application/vnd.apache.tenon.plugin+tar+gzip' \
+ --data-binary "@$package" -o /dev/null
+done
+printf 'Saving the two Pipeline Documents...\n'
+for document in hello builtins; do
+ curl --silent --show-error --fail -X PUT "$api/documents/$document" \
+ -H 'Content-Type: application/jsonc' -H 'If-None-Match: *' \
+ --data-binary "@$demo_root/$document.jsonc" -o /dev/null
+done
+
+printf 'Waiting for the Pipeline and plugin to run...\n'
+status_file="$demo_root/pipeline.json"
+ready=0
+for _ in $(seq 1 60); do
+ if curl --silent --show-error --fail "$api/pipelines/hello" -o "$status_file" \
+ && python3 - "$status_file" <<'PYTHON'
+import json
+import sys
+
+status = json.load(open(sys.argv[1]))
+instances = status.get("pluginInstances", [])
+raise SystemExit(not (
+ status.get("state") == "running"
+ and status.get("appliedDocumentEtag") == status.get("documentEtag")
+ and instances
+ and all(instance.get("state") == "running" for instance in instances)
+))
+PYTHON
+ then
+ ready=1
+ break
+ fi
+ sleep 1
+done
+if [[ "$ready" != 1 ]]; then
+ printf 'Pipeline did not become ready. See %s/runner.log and %s/pipeline.json\n' \
+ "$demo_root" "$demo_root" >&2
+ exit 1
+fi
+
+printf 'Waiting for the built-in Pipeline and its plugin instances to run...\n'
+builtin_status_file="$demo_root/builtins-pipeline.json"
+builtin_ready=0
+for _ in $(seq 1 60); do
+ if curl --silent --show-error --fail "$api/pipelines/builtins" -o "$builtin_status_file" \
+ && python3 - "$builtin_status_file" <<'PYTHON'
+import json
+import sys
+
+status = json.load(open(sys.argv[1]))
+instances = status.get("pluginInstances", [])
+raise SystemExit(not (
+ status.get("state") == "running"
+ and status.get("appliedDocumentEtag") == status.get("documentEtag")
+ and len(instances) == 2
+ and all(instance.get("state") == "running" for instance in instances)
+))
+PYTHON
+ then
+ builtin_ready=1
+ break
+ fi
+ sleep 1
+done
+if [[ "$builtin_ready" != 1 ]]; then
+ printf 'Built-in Pipeline did not become ready. See %s/runner.log and %s\n' \
+ "$demo_root" "$builtin_status_file" >&2
+ exit 1
+fi
+
+for _ in $(seq 1 30); do
+ if [[ -f "$demo_root/output.txt" ]] && grep -Fq 'Hello Tenon' "$demo_root/output.txt"; then
+ break
+ fi
+ sleep 1
+done
+if [[ ! -f "$demo_root/output.txt" ]] || ! grep -Fq 'Hello Tenon' "$demo_root/output.txt"; then
+ printf 'The example output did not appear. See %s/runner.log\n' "$demo_root" >&2
+ exit 1
+fi
+
+printf 'Checking output from the built-in Dummy Source → Lua → Stdout Sink Flow...\n'
+curl --silent --show-error --fail -N \
+ -H 'Accept: text/event-stream' \
+ "$api/pipelines/builtins/diagnostics?target=plugin%3Aconsole" \
+ > "$demo_root/stdout-events.txt" 2>/dev/null &
+diagnostics_pid=$!
+for _ in $(seq 1 15); do
+ if grep -Fq 'quickstart' "$demo_root/stdout-events.txt"; then
+ break
+ fi
+ sleep 1
+done
+if ! grep -Fq 'quickstart' "$demo_root/stdout-events.txt"; then
+ printf 'Built-in Stdout Sink output did not appear. See %s/stdout-events.txt\n' \
+ "$demo_root" >&2
+ exit 1
+fi
+
+printf '\nSuccess: %s/output.txt contains:\n' "$demo_root"
+cat "$demo_root/output.txt"
+printf '\nBuilt-in plugin diagnostic output:\n'
+cat "$demo_root/stdout-events.txt"
+printf '\nRunner log: %s/runner.log\nPress Ctrl-C to stop Tenon.\n\n' "$demo_root"
+tail -f "$demo_root/runner.log"