You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by ri...@apache.org on 2022/11/29 19:46:14 UTC
[streampipes-website] branch dev updated: docs: Add documentation for functions (apache/streampipes#751)
This is an automated email from the ASF dual-hosted git repository.
riemer pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/streampipes-website.git
The following commit(s) were added to refs/heads/dev by this push:
new d6809819 docs: Add documentation for functions (apache/streampipes#751)
d6809819 is described below
commit d6809819c4d88a44cd27b91432b6188abbc9c56c
Author: Dominik Riemer <do...@gmail.com>
AuthorDate: Mon Nov 28 20:44:44 2022 +0100
docs: Add documentation for functions (apache/streampipes#751)
---
documentation/docs/06_extend-sdk-functions.md | 125 ++++++++++++++++++++++++++
documentation/website/i18n/en.json | 6 +-
documentation/website/pages/en/versions.js | 2 +-
documentation/website/sidebars.json | 30 +------
4 files changed, 133 insertions(+), 30 deletions(-)
diff --git a/documentation/docs/06_extend-sdk-functions.md b/documentation/docs/06_extend-sdk-functions.md
new file mode 100644
index 00000000..f2f27302
--- /dev/null
+++ b/documentation/docs/06_extend-sdk-functions.md
@@ -0,0 +1,125 @@
+---
+id: extend-sdk-functions
+title: SDK Guide: Functions
+sidebar_label: SDK: Functions
+---
+
+## Introduction
+
+Pipeline elements such as data processors and data sinks are a great way
+to create _reusable_ components that can be part of pipelines.
+However, creating a pipeline element is not always the best choice:
+
+* The behaviour of a data processor is bound to a specific input stream _and_
+* A data processor doesn't contain any user-defined configuration _and_
+* The intended action is fixed or known at build time and the data processor shouldn't be available in the pipeline editor.
+
+To cover such use cases, we provide _StreamPipes Functions_. Functions
+are a great way to define custom processing logic based on previously
+connected data streams.
+
+Functions can be registered in a similar way to pipeline elements, but define expected input
+streams at startup time. Functions are started once the corresponding _extensions service_ starts
+and run until the service is stopped.
+
+## Writing a function
+
+<div class="admonition warning">
+ <div class="admonition-title">Work in Progress</div>
+ <p>Functions are currently in preview mode and are not yet recommended for production usage.
+APIs are subject to change in a future version.</p>
+</div>
+
+To define a function, create a new extensions service using the [Maven Archetypes](06_extend-archetypes.md) or use an already existing service.
+
+### Skeleton
+
+Functions can be defined by creating a new class which extends the ``StreamPipesFunction`` class.
+
+The basic skeleton looks like this:
+
+```java
+public class StreamPipesFunctionExample extends StreamPipesFunction {
+
+ @Override
+ public FunctionId getFunctionId() {
+ return FunctionId.from("my-function-id", 1);
+ }
+
+ @Override
+ public List<String> requiredStreamIds() {
+ return List.of("<id of the required stream>");
+ }
+
+ @Override
+ public void onServiceStarted(FunctionContext context) {
+ // called when the service is started
+ }
+
+ @Override
+ public void onEvent(Event event, String streamId) {
+ // called when an event arrives
+ }
+
+ @Override
+ public void onServiceStopped() {
+ // called when the service is stopped
+ }
+}
+
+```
+
+The structure of a function class is easy to understand:
+* _getFunctionId_ requires an identifier in form of a ``FunctionId``, which defines the id itself along with a version number that can be freely chosen.
+* _requiredStreamIds_ expects a list of references to data streams that are already available in StreamPipes. See below to learn how to find the id of a stream in StreamPipes.
+* _onServiceStarted_ is called once the extensions service is started and can be used to initialize the function.
+* _onEvent_ is called every time a new event arrives and provides a ``streamId`` as a reference to the corresponding stream, which is useful in case multiple data streams are received by the function.
+* _onServiceStopped_ is called when the extensions service is stopped and can be used to perform any required cleanup.
+
+### Getting a stream ID
+
+Functions require a reference to all data streams that should be retrieved by the function.
+Currently, the only way to get the ID of a function is by navigating to the ``Asset Management`` view in the StreamPipes UI.
+Create a new asset, click on ``Edit Asset`` and open ``Add Link`` in the _Linked Resources_ panel.
+Choose ``Data Source`` as link type, select one of the available sources, copy the ``Resource ID`` and provide this ID in the ``requiredStreamIds`` method.
+
+### Function Context
+
+The ``onServiceStarted`` method provides a function context which provides several convenience methods to work with functions:
+
+* _getFunctionId_ returns the current function identifier
+* _getConfig_ returns a reference to configuration options of the extensions service
+* _getClient_ returns a reference to the StreamPipes client to interact with features from the REST API.
+* _getStreams_ returns the data model of all data streams defined in the ``requiredStreamIds`` method.
+* _getSchema_ returns the schema of a specific data stream by providing the ``streamId``
+
+
+## Registering a function
+
+Registering a function is easy and can be done in the _Init_ class of the service.
+E.g., considering a service definition as illustrated below, simply call ``registerFunction`` and
+provide an instance of your function.
+
+```java
+
+ @Override
+ public SpServiceDefinition provideServiceDefinition() {
+ return SpServiceDefinitionBuilder.create("my-service-id",
+ "StreamPipes Function Example",
+ "",
+ 8090)
+ .registerFunction(new MyExampleFunction())
+ .registerMessagingFormats(
+ new JsonDataFormatFactory())
+ .registerMessagingProtocols(
+ new SpNatsProtocolFactory())
+ .build();
+ }
+
+```
+
+## Metrics & Monitoring
+
+Similar to pipeline elements, function register at the StreamPipes core.
+Running functions can be seen in the pipeline view of the user interface under _Functions_, right below the list of available pipelines.
+Similar to pipelines, simple metrics, monitoring info and exceptions can be viewed in the _Details_ section of each function.
diff --git a/documentation/website/i18n/en.json b/documentation/website/i18n/en.json
index 45ba5ae6..65034d3e 100644
--- a/documentation/website/i18n/en.json
+++ b/documentation/website/i18n/en.json
@@ -101,6 +101,10 @@
"title": "SDK Guide: Event Model",
"sidebar_label": "SDK: Event Model"
},
+ "extend-sdk-functions": {
+ "title": "SDK Guide: Functions",
+ "sidebar_label": "SDK: Functions"
+ },
"extend-sdk-migration-service-discovery": {
"title": "Migration Guide: New Service Discovery in 0.69.0",
"sidebar_label": "Migration Guide: 0.69.0"
@@ -2544,11 +2548,11 @@
"💻 Extend StreamPipes": "💻 Extend StreamPipes",
"🔧 Technicals": "🔧 Technicals",
"👪 Community": "👪 Community",
+ "FAQ": "FAQ",
"Basics": "Basics",
"Tutorials": "Tutorials",
"SDK Reference": "SDK Reference",
"Configuration": "Configuration",
- "FAQ": "FAQ",
"Getting Started": "Getting Started",
"Pipeline Elements": "Pipeline Elements",
"Extending StreamPipes": "Extending StreamPipes",
diff --git a/documentation/website/pages/en/versions.js b/documentation/website/pages/en/versions.js
index 1e211d8e..0473f7d1 100644
--- a/documentation/website/pages/en/versions.js
+++ b/documentation/website/pages/en/versions.js
@@ -65,7 +65,7 @@ function Versions(props) {
<a href="https://github.com/apache/streampipes/tree/dev">🖥 Development Branch</a>
</td>
<td>
- <a href="https://streampipes.apache.org/docs/docs/next/user-guide-introduction">🚧 Documentation</a>
+ <a href={`${siteConfig.baseUrl}docs/next/user-guide-introduction`}>🚧 Documentation</a>
</td>
<td>
<a href="https://streampipes.apache.org/mailinglists.html">📯 Mailing list</a>
diff --git a/documentation/website/sidebars.json b/documentation/website/sidebars.json
index 15dc995c..cb00e81c 100644
--- a/documentation/website/sidebars.json
+++ b/documentation/website/sidebars.json
@@ -187,6 +187,7 @@
"extend-tutorial-data-sources",
"extend-tutorial-data-processors",
"extend-tutorial-data-sinks",
+ "extend-sdk-functions",
"extend-sdk-event-model",
"extend-sdk-stream-requirements",
"extend-sdk-static-properties",
@@ -198,40 +199,13 @@
"technicals-user-guidance",
"technicals-runtime-wrappers",
"technicals-messaging",
- "technical-configuration"
+ "technicals-configuration"
],
"\uD83D\uDC6A Community": [
"community-get-help",
"community-contribute"
]
},
- "developer-guide": {
- "Basics": [
- "dev-guide-introduction",
- "dev-guide-architecture",
- "dev-guide-development-environment",
- "dev-guide-archetype",
- "dev-guide-migration"
- ],
- "Tutorials": [
- "dev-guide-tutorial-sources",
- "dev-guide-tutorial-processors",
- "dev-guide-tutorial-sinks"
- ],
- "SDK Reference": [
- "dev-guide-sdk-guide-sources",
- "dev-guide-sdk-guide-processors",
- "dev-guide-sdk-guide-sinks",
- "dev-guide-stream-requirements",
- "dev-guide-static-properties",
- "dev-guide-output-strategies",
- "dev-guide-event-model"
- ],
- "Configuration": [
- "dev-guide-ssl",
- "dev-guide-configuration"
- ]
- },
"faq": {
"FAQ": [
"faq-common-problems"