You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by bo...@apache.org on 2023/10/23 09:07:13 UTC

[streampipes-website] branch update-adapter-docu updated (7a4b9a29 -> 139da374)

This is an automated email from the ASF dual-hosted git repository.

bossenti pushed a change to branch update-adapter-docu
in repository https://gitbox.apache.org/repos/asf/streampipes-website.git


    from 7a4b9a29 small update
     add 2a7b302e chore(python-docs): update python doc assets to recent development state (#122)
     add 3d6e3314 Add architecture section (#123)
     new 99421cea Merge remote-tracking branch 'origin/dev' into update-adapter-docu
     new 139da374 add section about UI configuration of adapters

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs-python/dev/reference/client/client/index.html |  25 +++
 docs-python/dev/scripts/gen_ref_pages/index.html   |  12 +-
 docs-python/dev/search/search_index.json           |   2 +-
 docs-python/dev/sitemap.xml                        |  96 +++++------
 docs-python/dev/sitemap.xml.gz                     | Bin 704 -> 704 bytes
 docs/06_extend-tutorial-adapters.md                | 182 +++++++++++++++++++--
 docs/07_technicals-architecture.md                 | 122 +++++++++-----
 docs/07_technicals-configuration.md                |  74 ---------
 docs/07_technicals-messaging.md                    |  37 ++++-
 website-v2/sidebars.json                           |   5 +-
 .../img/06_tutorial_adapter/adapter-demo.gif       | Bin 0 -> 1650273 bytes
 .../06_tutorial_adapter/adapter-with-ui-dialog.png | Bin 0 -> 52810 bytes
 .../static/img/07_technicals/architecture.png      | Bin 0 -> 127397 bytes
 .../static/img/tutorial-sources/pe-overview.PNG    | Bin 29556 -> 0 bytes
 website-v2/static/img/tutorial-sources/pe-rdf.PNG  | Bin 81915 -> 0 bytes
 .../img/tutorial-sources/project-structure.PNG     | Bin 27916 -> 0 bytes
 .../version-0.92.0/07_technicals-architecture.md   | 122 +++++++++-----
 .../version-0.92.0/07_technicals-configuration.md  |  58 -------
 .../version-0.92.0/07_technicals-messaging.md      |  37 ++++-
 .../version-0.92.0-sidebars.json                   |   3 +-
 20 files changed, 493 insertions(+), 282 deletions(-)
 delete mode 100644 docs/07_technicals-configuration.md
 create mode 100644 website-v2/static/img/06_tutorial_adapter/adapter-demo.gif
 create mode 100644 website-v2/static/img/06_tutorial_adapter/adapter-with-ui-dialog.png
 create mode 100644 website-v2/static/img/07_technicals/architecture.png
 delete mode 100644 website-v2/static/img/tutorial-sources/pe-overview.PNG
 delete mode 100644 website-v2/static/img/tutorial-sources/pe-rdf.PNG
 delete mode 100644 website-v2/static/img/tutorial-sources/project-structure.PNG
 delete mode 100644 website-v2/versioned_docs/version-0.92.0/07_technicals-configuration.md


[streampipes-website] 02/02: add section about UI configuration of adapters

Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

bossenti pushed a commit to branch update-adapter-docu
in repository https://gitbox.apache.org/repos/asf/streampipes-website.git

commit 139da37491843ef0f27829899b35c0a9e6eccfb5
Author: bossenti <bo...@posteo.de>
AuthorDate: Mon Oct 23 11:07:01 2023 +0200

    add section about UI configuration of adapters
---
 docs/06_extend-tutorial-adapters.md                | 182 +++++++++++++++++++--
 .../img/06_tutorial_adapter/adapter-demo.gif       | Bin 0 -> 1650273 bytes
 .../06_tutorial_adapter/adapter-with-ui-dialog.png | Bin 0 -> 52810 bytes
 .../static/img/tutorial-sources/pe-overview.PNG    | Bin 29556 -> 0 bytes
 website-v2/static/img/tutorial-sources/pe-rdf.PNG  | Bin 81915 -> 0 bytes
 .../img/tutorial-sources/project-structure.PNG     | Bin 27916 -> 0 bytes
 6 files changed, 168 insertions(+), 14 deletions(-)

diff --git a/docs/06_extend-tutorial-adapters.md b/docs/06_extend-tutorial-adapters.md
index 1d06f470..a5017a08 100644
--- a/docs/06_extend-tutorial-adapters.md
+++ b/docs/06_extend-tutorial-adapters.md
@@ -42,7 +42,7 @@ The events are published in JSON format as follows
 In the following section, we will show you how to develop an adapter that is capable of generating this stream so that
 it is available for further processing in StreamPipes.
 
-## Project setup
+## Project Set Up
 
 Instead of creating a new project from scratch, we recommend to use our Maven archetype to create a new project
 skeleton (`streampipes-archetype-extensions-jvm`).
@@ -117,7 +117,7 @@ In addition to the basic project skeleton, the sample project also includes a sa
 package your application into a Docker container.
 :::
 
-## Building a basic adapter
+## Building a Basic Adapter
 
 In the following, we will demonstrate how to use the boilerplate code generated by the Maven plugin (
 see [Project setup](#project-setup)).
@@ -220,7 +220,7 @@ This interface requires us to implement four methods:
 * `onSchemaRequested()`: This method is expected to return the schema of the event stream. This is ideally done
   dynamically based on some incoming data (*guess*) or provided statically if not otherwise possible.
 
-### Describing the adapter via the configuration
+### Describing the Adapter via the Configuration
 
 The standard code generated here is already sufficient for us.
 So let's have a quick look at the important aspects:
@@ -258,7 +258,7 @@ org.apache.streampipes.pe.parcelcontrol.specificadapter.description=This adapter
 
 Now we have successfully configured our adapter and prepared all descriptive elements, we can focus on the actual logic.
 
-### Creating the data stream
+### Creating the Data Stream
 
 The logic that creates events that are then populated via StreamPipes is defined in `onAdapterStarted()`.
 Within this method, connectors usually connect to the data source and extract data.
@@ -335,7 +335,7 @@ follows:
 This is already enough to get a data stream into StreamPipes.
 As the next step we need to describe to event schema.
 
-### Defining the event schema
+### Defining the Event Schema
 
 In StreamPipes, each data stream comes with an event schema that describes what information the event contains,
 in what data formats, and some semantic type information.
@@ -416,7 +416,7 @@ full potential and feature richness.
 
 This makes our adapter almost complete, there is only one little step left.
 
-### Defining the adapter termination
+### Defining the Adapter Termination
 
 As a final step, we need to define what should happen if the adapter is stopped.
 In general, the adapter should not fire any events after that.
@@ -435,27 +435,181 @@ public void onAdapterStopped(IAdapterParameterExtractor extractor,
 
 Now it's time to start our adapter and observe it in action!
 
-### Register and Run the adapter
+### Register and Run the Adapter
 
-### Building a more advanced adapter by processing UI Input
+Before we actually use our adapter, let's take a quick look at the `Init` class. This class is responsible for
+registering our adapter service with the core to make the adapter available in StreamPipes.
+This is done within `provideServiceDefinition()`. Since we don't have the generic adapter ready yet,
+we'll comment out its registration (`line 7`). Now we can run the `Init` class to register the adapter with your running
+StreamPipes instance. If you don't have a running instance at your hand,
+you can take a look at our [Installation Guide](./01_try-installation.md).
 
-### Read more
+```java jsx {7-8} showLineNumbers
+@Override
+public SpServiceDefinition provideServiceDefinition() {
+  return SpServiceDefinitionBuilder.create("org.apache.streampipes",
+          "human-readable service name",
+          "human-readable service description", 8090)
+      .registerRuntimeProvider(new StandaloneStreamPipesRuntimeProvider())
+      //.registerAdapter(new ParcelControlStationGenericAdapter())
+      .registerAdapter(new ParcelControlStationSpecificAdapter())
+      .registerMessagingFormats(
+          new JsonDataFormatFactory(),
+          new CborDataFormatFactory(),
+          new SmileDataFormatFactory(),
+          new FstDataFormatFactory())
+      .registerMessagingProtocols(
+          new SpKafkaProtocolFactory(),
+          new SpJmsProtocolFactory(),
+          new SpMqttProtocolFactory(),
+          new SpNatsProtocolFactory(),
+          new SpPulsarProtocolFactory())
+      .build();
+}
+ ```
+
+:::tip
+When executing the `main()` method of the `Init` class, make sure that all environment variables are set from
+the `development/env` file are set.
+If they are not set, the adapter may not be able to register with StreamPipes.
+:::
+
+Once you see the following log message in the console, the adapter is ready, and you can switch to the UI of your
+StreamPipes instance.
+
+```bash
+s.s.e.c.ConnectWorkerRegistrationService : Successfully connected to master. Worker is now running.
+```
+
+Please go to the connect module and click on `New Adapter`,
+you should now be able to see your adapter `Parcel Control Station (simple)`:
+<img src="/img/06_tutorial_adapter/adapter-demo.gif" alt="Demo of parcel adapter"/>
+
+The adapter runs successfully in StreamPipes, you can now play around with the data stream that the
+adapter, or continue with the next section to learn how to make an adapter configurable through the UI.
+
+### Building a more Advanced Adapter by Processing UI Input
+
+In this section, we will extend our previous build apter by adding the ability to configure the minimum and maximum
+package
+in the UI from which the weight value is retrieved.
+The beauty of building adapters for StreamPipes is that you don't have to worry about the UI.
+StreamPipes provides a set of pre-built input elements for adapters that you can simply add to your adapter
+configuration.
+So the first thing we need to customize is `declareConfig()`:
+
+```java jsx {10-11} showLineNumbers
+@Override
+public IAdapterConfiguration declareConfig() {
+  return AdapterConfigurationBuilder.create(
+                  "org.apache.streampipes.pe.parcelcontrol.specificadapter",
+                  ParcelControlStationSpecificAdapter::new
+          )
+          .withAssets(Assets.DOCUMENTATION, Assets.ICON)
+          .withCategory(AdapterType.Manufacturing)
+          .withLocales(Locales.EN)
+          .requiredFloatParameter(Labels.withId("min-weight"), 0.0f)
+          .requiredFloatParameter(Labels.withId("max-weight"), 10.f)
+          .buildConfiguration();
+}
+
+```
+
+In line `9-10` we have introduced two input parameters that expect float values as input. They have a default value
+of `0` or `10` resp. The defined identifier (`min-weight` and `max-weight`) can be used two provide a caption and
+a description via the `strings.en` file:
+
+```text
+min-weight.title=Minimum Parcel Weight
+min-weight.description=The lower bound from which the weight values are sampled randomly.
+
+max-weight.title=Maximum Parcel Weight
+max-weight.description=The upper bound from which the weight values are sampled randomly.
+```
+
+As a last step, we now need to modify the calculation of the parcel weight, so that the provided parameters are actually
+applied.
+This is done in `onAdapterStarted()`.
+
+```java jsx {6-9,24} showLineNumbers
+@Override
+public void onAdapterStarted(IAdapterParameterExtractor extractor,
+                             IEventCollector collector,
+                             IAdapterRuntimeContext adapterRuntimeContext) throws AdapterException {
+
+  var ex = extractor.getStaticPropertyExtractor();
+  
+  float minWeight = ex.singleValueParameter("min-weight", Float.class);
+  float maxWeight = ex.singleValueParameter("max-weight", Float.class);
+
+  Runnable parcelControl = () -> {
+    while (running) {
+
+      // get the current time in seconds
+      long timestamp = System.currentTimeMillis();
+      long timeInSeconds = (int) timestamp / 1000;
+
+      // make event
+      Map<String, Object> event = new HashMap<>();
+      event.put("timestamp", timestamp);
+
+      if (timeInSeconds % 5 == 0) {
+        event.put("parcelPresent", true);
+        event.put("weight", ThreadLocalRandom.current().nextDouble(minWeight, maxWeight));
+
+      } else {
+        event.put("parcelPresent", false);
+        event.put("weight", 0);
+      }
+
+      // forward the event to the adapter pipeline
+      collector.collect(event);
+
+      try {
+        Thread.sleep(1000);
+      } catch (InterruptedException e) {
+        throw new RuntimeException(e);
+      }
+    }
+  };
+  running = true;
+  new Thread(parcelControl).start();
+}
+```
+
+* line `6-9`: We use an `StaticPropertyExtractor` to retrieve both user inputs
+* line `24`: We calculate the parcel weight by passing the configured values vor the minimum and maximum value.
+
+You can now run the `main()` method of the `Init` class to register the adapter at StreamPipes.
+The UI dialog to create a new instance of our parcel control station adapter looks now the following:
+<img src="/img/06_tutorial_adapter/adapter-with-ui-dialog.png" alt="Adapter with UI dialog"/>
+
+:::caution
+Please make sure that you uninstall the parcel adapter in `Install Pipeline Elements` before 
+you restart the execution of the `Init` class, if you have already done so.
+Otherwise, the changes made in this section will have no effect.
+:::
+
+### Read More
 
 Congratulations! You've just created your first StreamPipes adapter 🎉 <br />
 
 There are many more things to explore and data sources can be defined in much more detail.
 If this is of interest to you, the [advanced section](#advanced) will satisfy your needs.
 
+If anything within this tutorial did not work for you or you had problems following it,
+please feel free to provide some feedback by opening an [issue on GitHub](https://github.com/apache/streampipes/issues/new?assignees=&labels=bug%2Cdocumentation%2Cwebsite&projects=&template=doc_website_issue_report.yml).
+
 ## Advanced
 
 There is more to come about advanced features of our adapters. Stay tuned!
 
-### Forward log messages to the UI to allow debugging for the end user
+### Forward Log Messages to the UI to Allow Debugging for the End User
 
-### Determine the event schema based on the concrete data structure
+### Determine the Event Schema Based on the Concrete Data Structure
 
-### Implement a generic adapter that is capable of multiple input formats
+### Implement a Generic Adapter that is Capable of Multiple Input Formats
 
-### Extend the existing pool of parser by your own parser
+### Extend the Existing Pool of Parsers by Your Own Parser
 
-### Introduce a new version of an adapter and ensure migration of existing adapters
+### Introduce a Vew Version of an Adapter and Ensure Migration of Existing Adapters
diff --git a/website-v2/static/img/06_tutorial_adapter/adapter-demo.gif b/website-v2/static/img/06_tutorial_adapter/adapter-demo.gif
new file mode 100644
index 00000000..1a6ec37e
Binary files /dev/null and b/website-v2/static/img/06_tutorial_adapter/adapter-demo.gif differ
diff --git a/website-v2/static/img/06_tutorial_adapter/adapter-with-ui-dialog.png b/website-v2/static/img/06_tutorial_adapter/adapter-with-ui-dialog.png
new file mode 100644
index 00000000..fbd0130d
Binary files /dev/null and b/website-v2/static/img/06_tutorial_adapter/adapter-with-ui-dialog.png differ
diff --git a/website-v2/static/img/tutorial-sources/pe-overview.PNG b/website-v2/static/img/tutorial-sources/pe-overview.PNG
deleted file mode 100644
index 31af43b5..00000000
Binary files a/website-v2/static/img/tutorial-sources/pe-overview.PNG and /dev/null differ
diff --git a/website-v2/static/img/tutorial-sources/pe-rdf.PNG b/website-v2/static/img/tutorial-sources/pe-rdf.PNG
deleted file mode 100644
index 5208c5ed..00000000
Binary files a/website-v2/static/img/tutorial-sources/pe-rdf.PNG and /dev/null differ
diff --git a/website-v2/static/img/tutorial-sources/project-structure.PNG b/website-v2/static/img/tutorial-sources/project-structure.PNG
deleted file mode 100644
index dccf183c..00000000
Binary files a/website-v2/static/img/tutorial-sources/project-structure.PNG and /dev/null differ


[streampipes-website] 01/02: Merge remote-tracking branch 'origin/dev' into update-adapter-docu

Posted by bo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

bossenti pushed a commit to branch update-adapter-docu
in repository https://gitbox.apache.org/repos/asf/streampipes-website.git

commit 99421cea451539d8e8f61998c73020bf5d940db3
Merge: 7a4b9a29 3d6e3314
Author: bossenti <bo...@posteo.de>
AuthorDate: Mon Oct 23 10:07:37 2023 +0200

    Merge remote-tracking branch 'origin/dev' into update-adapter-docu

 docs-python/dev/reference/client/client/index.html |  25 +++++
 docs-python/dev/scripts/gen_ref_pages/index.html   |  12 +-
 docs-python/dev/search/search_index.json           |   2 +-
 docs-python/dev/sitemap.xml                        |  96 ++++++++--------
 docs-python/dev/sitemap.xml.gz                     | Bin 704 -> 704 bytes
 docs/07_technicals-architecture.md                 | 122 ++++++++++++++-------
 docs/07_technicals-configuration.md                |  74 -------------
 docs/07_technicals-messaging.md                    |  37 ++++++-
 website-v2/sidebars.json                           |   5 +-
 .../static/img/07_technicals/architecture.png      | Bin 0 -> 127397 bytes
 .../version-0.92.0/07_technicals-architecture.md   | 122 ++++++++++++++-------
 .../version-0.92.0/07_technicals-configuration.md  |  58 ----------
 .../version-0.92.0/07_technicals-messaging.md      |  37 ++++++-
 .../version-0.92.0-sidebars.json                   |   3 +-
 14 files changed, 325 insertions(+), 268 deletions(-)