You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Florian Micklich <mi...@apache.org> on 2022/09/14 14:52:27 UTC
Re: [DISCUSS] Documentation - Description for Adapters and Processing Elements
Hi all,
I made a rouge structure about a possible documentation.
• Introduction: ◦ Overview ◦ Basic Elements (former
StreamPipes Concept) ◦ Time in StreamPipes (Why timestamp is
important in the data?) ◦ Semantic in StreamPipes ◦ Other
Topics? ◦ Installation ▪ Basic
Installation ▪ Docker Deployment ▪ K8s /
K3s ▪ Use SSL ▪ Security ◦ Getting
Help • Overview (Basic GUI Explanation and Introduction for each GUI
point and why it is implemented (short description text) for example
Dashboard why Dashboard, what you can do. Basically little more deep
diver than Basic Elements section. Section with Most of the
Screenshots, So easy to maintain due changes) ◦ Login Page and
Basics ◦ Welcome Page ◦ Pipelines ◦
Connect ◦ Dashboard ◦ Data Explorer ◦ Apps
(deprecated?) ◦ Install Pipeline Elements ◦ Asset
Management ◦ File Management ◦ Configuration & User
Prefences ◦ Notification • Data Source Elements
(Adapter) ◦ Why Adapter. Connect to the Real World ◦ What
Kind of Sources (Data Set vs Data Stream / Categories) ◦ Basic
Adapter Setup in 4 Steps (at the moment) ▪ Step 1
Settings ▪ Step 2 Select Format • JSON
with Description for Array Field / GeoJSON / Single Object / Array with
Short Data Examples • XML • CSV
• Image ▪ Step 3 configure
fields • Different between Measurement, Dimension
Header • Add Nested / Add static Value / Add
Timestamp • Edit fields (Big Thinks.
Also) • Preview ▪ Step4 start
adapters • Settings / Description (why Important
and where can I find this info later on?) •
Remove Duplicates / Reduce event rates /Persist events ◦ Connect
Adapters Overview ▪ Apache Kafka with Example
Sources ▪ Apache Pulsar ▪ File
Set ▪ File Stream ▪ ... ◦ Connect
Adapters IIOT Overview ▪ ... ◦ How To
Adapter ▪ Setup Adapter Example 1 ▪ Setup
HTTPS Adapter ◦ Technical Concept of Adapters (Dev Part???) •
Data Processors • Data Sinks • Create Pipeline •
Dashboard • Assets
The working example will be in the section HowTo and some example
resources in each adapter description.What do you think?
First step would be to check if it possible to move the files so that
the structure is not destroyed and the simply start with the adapters
:)I made already a rebase between dev and STREAMPIPES-581 branch, due
some changes in the dev branch.
Florian
Am Montag, dem 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder:
> Hi Florian,
> I am not sure if we will find many available public data sources,
> especially for PLCs. But maybe we find some docker containers that
> can be used for the tutorial. If we do not find some, maybe we can
> provide them in our registry.We could also use them for the automated
> tests. That would be great, because currently those tests are not
> integrated in the nightly run.
> I think for most users it is already helpful to have a working
> example for the different adapters, so they do not necessarily have
> to try out the examples.
> As you said we should collect all the information somewhere. Should
> we use thwiki for that?
> Cheers,Philipp________________________________Von: Florian Micklich <
> micklich@apache.org>Gesendet: Samstag, August 27, 2022 1:20 AMAn:
> dev@streampipes.apache.org <de...@streampipes.apache.org>Betreff: Re:
> [DISCUSS] Documentation - Description for Adapters and Processing
> Elements
> Hi,perfekt, I will create this branch. I will think about a
> possiblestructure over the weekend. I let you know.How are you
> testing and using the adapters and wich one are you usingmost? Do you
> have some URL sources you always for developing processor testing
> and what is the typical workflow to add them?Maybe we can start to
> collect them in some way so I can add them later in
> thedocumentation.GreetingsFlorian
> Am Donnerstag, dem 25.08.2022 um 19:38 +0000 schrieb Dominik Riemer:
> > Sounds good!We usually use only the issue name as branch
> > name(STREAMPIPES-581) without the "feature" prefix, but both is
> > totallyfine!Dominik
> > -----Original Message-----From: Florian Micklich <
> > micklich@apache.org
> > > Sent: Wednesday, August 24, 2022 9:44 PMTo:
> > dev@streampipes.apache.org
> > Subject: Re: [DISCUSS] Documentation - Description for Adapters
> > andProcessing ElementsHi Dominik,thanks for the answer and before I
> > ask more questions,I'll simply start :-)Therefore I created a
> > tickethttps://issues.apache.org/jira/browse/STREAMPIPES-581
> > I will also create a feature branch feature/STREAMPIPES-581 or
> > whatis here best practise naming convention?GreetingFlorian
> >
> > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb Dominik Riemer:
> > > Hi Florian,happy to help! 1. There is a header in each
> > > markdownfilewith an ID (e.g. in the 01_try-installation.md). This
> > > idcorrespondsto the id in the sidebar.json2. Yes, all images are
> > > inthe staticfolder 3. Well, they are not really auto-auto
> > > generated😉 There is amaven plugin which extracts the markdown
> > > files ande.g., createsentries for the sidebar.json and modifies
> > > the imagelinks so thatthey work on the website, but it's not
> > > somethingautomated in Githubactions or so, although it would be
> > > awesome tosupport this4. Youdon't need to manually add the
> > > version, this ismanaged byDocusaurus. The markdown files in the
> > > documentationfolder alwaysrefer to the next unreleased version.
> > > When you startthe website, youcan click on the version (0.69.0
> > > should be thedefault) and thenselect "master" which points you to
> > > the unreleasedversion. Aftereach release, we run the "ds-version"
> > > command andthen Docusauruschecks all files in the documentation
> > > folder againstthe files fromthe previous release and creates a
> > > new docs version.E.g., see olderdocumentation files at [1]. In
> > > case you want tobackport changes tothe documentation pages to
> > > previous versions,these files can also bemodified, but that's
> > > rather rare.5. Goodpoint, I _think_ this shouldbe possible
> > > without any problems, atleast it's worth a try 😉Hope this
> > > helps!CheersDominik[1]
> > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs
> > >
> > >
> > > -----Original Message-----From: Florian Micklich <
> > > micklich@apache.org
> > > > Sent: Wednesday, August 24, 2022 7:59 PMTo:
> > > dev@streampipes.apache.org
> > > Subject: Re: [DISCUSS] Documentation - Description for
> > > AdaptersandProcessing ElementsHi,ok, I started docusaurus from
> > > sourceDominik has previouslymentioned and did a build & run
> > > website likementioned in theReadme.Next time I am even faster,
> > > because I willstart the "BuildDocumentation" from the
> > > documentation/websitefolder, as it itclearly mentioned in the
> > > Readme :-D Before I start,I am trying tounderstand the structure
> > > of this documentation and Ihave somequestions.1.So everything is
> > > organised by thesidebar.jsonSo the try-installation [1] is linked
> > > to the 01_try-installation.md file[2]. How is this working if
> > > the names doesnot match?2. I see that all images are stored in
> > > the static websitepath here[3]. Is this correct?3. Do I
> > > understand it correctly thatall filesin the PE folder [4] are
> > > auto generated and will beautogenerated next time again? So if
> > > I change something here,will thisbe overwritten the next
> > > time?4.Is it OK if I add the 0.70version inthe version file here
> > > [5]?5. Is it OK to structure thedocuments insome more sup-
> > > folders? We could discuss the sub-foldernames here ofcourse but I
> > > think that would make the documentationlittle biteasier to
> > > navigate for the authors. This includes alsopictures instep
> > > 2.Greetings Florian[1]
> > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2]
> > >
> > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]
> > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]
> > > https://github.com
> > > /apache/incubator-streampipes-
> > > website/tree/dev/documentation/docs/pe[5]
> > > https://github.com/apache/inc ubator-streampipes-
> > > website/blob/dev/documentation/website/versions.json
> > > Am Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb Philipp
> > > Zehnder:
> > > > Hi together,it would be great to have more content
> > > > inthedocumentation.I am alsoin favor of using docusaurus.
> > > > Itusesmarkdown, which leaves us thepossibility to move to
> > > > anothertoolingat some point in the future ifwe want.@Florian
> > > > pleasewrite if yousee something that should be updated orchanged in thecurrentversion.Cheers,Philipp________________________________Von:DominikRiemer <do...@bytefabrik.ai>Gesendet: Mittwoch,August
> > > > 24,202211:47 AMAn: dev@streampipes.apache.org
> > > > <de...@streampipes.apache.org>Betreff: RE: [DISCUSS]
> > > > Documentation-Description for Adapters andProcessing ElementsHi
> > > > Florian,Cool!Thedocumentation on the website is based on
> > > > Docusauruswhichsupportsthings like docs versioning. Docusaurus
> > > > usesmarkdown files,so werely on markdown for the
> > > > documentation.Thereis a 'website' folderat [1] where you can
> > > > find the npm scriptstostart docusaurus and a'docs' folder which
> > > > includesthedocumentation markdown files forthe next release
> > > > version(selectg"next" in the docs website). Aftera new release,
> > > > we run a"npm ds-version" which then detects changesin the
> > > > markdown filesand createsa new release docs
> > > > version.Thedocuments in the "docs"folder can be freely modified
> > > > andadded.That's a good place wherewe can add new tutorials. The
> > > > "pe"foldercontains thedocumentation of pipeline elements. These
> > > > arecurrentlygeneratedfrom the StreamPipes source code using
> > > > thestreampipes-maven-plugin (as some rewriting is needed, e.g.,
> > > > to fixthe pathsofimages). This process is far from perfect and
> > > > needssome manualwork.It would be nice to have some Github
> > > > actionsscriptwhichautomatically updates the Documentation, but
> > > > that's abitmoreadvanced task I'd say 😉If you want to make
> > > > changes tothestructure, this can be done in the"sidebars.json"
> > > > file [3],whichis also auto-versioned by Docusaurus.[1]
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website
> > > > [2]
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs
> > > > [3]
> > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json
> > > >
> > > > -----Original Message-----From: Florian Micklich <
> > > > micklich@apache.org
> > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo:
> > > > dev@streampipes.apache.org
> > > > Subject: Re: [DISCUSS] Documentation - Description
> > > > forAdaptersandProcessing ElementsHello,I would also prefer a
> > > > shortIn-Appdocumentation and largerdocumentation "outside"
> > > > .Personally, Iprefer to read pdf documentsso I can take
> > > > additionalnotes. ButHTML is also good for a quickonline
> > > > search.We must keepin mindwhat kind of documentation we wantto
> > > > create and should notmixthem. Also, cross-references between
> > > > theindividual"chapters"should be avoided. This makes it easier
> > > > ifsomethingshouldchange. 1. Quick starter
> > > > documentation. 2.SetupDocumentation 3.Technical
> > > > Documentation 4. Usermanualdocumentation with bestpractiseI
> > > > think we should hit theroad andtake a look along the way :)
> > > > Iknow the in appdocumentation is inmarkdown so far?I
> > > > wouldrecommend to write thedocumentation inasciidoc [1] [2]
> > > > This couldbe set up withasciidoctorj [3]. Thismakes it easy to
> > > > includeeverything in Gitand also keep thedocumentation
> > > > alongside thesource code. So ifsomething isrewritten, the
> > > > developer can alsochange thedocumentation. Makesnew screenshots
> > > > and deletes oldones.3] Thecreation of a PDF orHTML file can be
> > > > done withGithubAction. With every commitor with every
> > > > release. Theresults can bepushed to the StreamPipeswebsite, I
> > > > guess?What doyouthink?Florian[1]
> > > > https://docs.asciidoctor.org/asciidoc/latest/[2]
> > > > German Source
> > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]
> > > > htt
> > > > ps://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.202
> > > > 2 um 07:07 +0000 schrieb Dominik Riemer:
> > > > > Hi,yes, we can do both: In-app documentation
> > > > > whichonlycoverstheconfiguration and a short description of
> > > > > theadapterandpipelineelement and additional step-by-
> > > > > stepguides/tutorialswithnicescreenshots that are available in
> > > > > theonlinedocumentation. Wecanhave such step-by-step guides
> > > > > for themostfrequently used adapters.@Florian, do you want to
> > > > > work onsuch aguide on thewebsite? I knowyou had some really
> > > > > gooddocumentationideas somewhile ago 😉CheersDominik-----
> > > > > OriginalMessage-----From: Philipp Zehnder <
> > > > > philipp.zehnder@bytefabrik.ai> Sent:Sunday, August
> > > > > 21,20221:07PMTo: dev@streampipes.apache.org
> > > > > Subject: [DISCUSS] Documentation -
> > > > > DescriptionforAdaptersandProcessing Elements Hi all, here is
> > > > > a new threadtodiscusstheadapter documentation.Here is a link
> > > > > to thepreviousdiscussionin therelease thread [1].@Dominik I
> > > > > know thattheregistration isdifferent, but since alladapters
> > > > > already havethethree files(documentation.md, icon.png,
> > > > > andstring.en),shouldn’tit bepossible to also display the
> > > > > descriptionin theUI.How are wedoingcurrently for the
> > > > > icons?Because they areloaded dynamicallyaswell. I think the
> > > > > maindifference is thatthe icons areloadeddirectly from the
> > > > > workercontainer and notfrom thebackend.Couldn’t we do it
> > > > > similarly forthedocumentation file?(Pleasecorrect me if I am
> > > > > wrong)Regardingthe description on theWebsite.I think it would
> > > > > begreat if we couldgenerate this basedon themarkdown files
> > > > > foreach adapter.The documentation.md filesmostlydescribe
> > > > > theconfigurationparameters of adapters andprocessors. Ithink
> > > > > itwould be great tohave additional exampleshow to use
> > > > > them.(e.g.describe how a CSVfile is uploaded or how aPLC
> > > > > isconnected).Should we include suchexamples
> > > > > intothedocumentation.md file orshould there be
> > > > > anotherlocation forthis?Any thoughts onthat?@Florian you
> > > > > wrote that you would liketo workon thedocumentation.If you
> > > > > need anything to start with,please
> > > > > letusknow.Cheers,Philipp[1]
> > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg
AW: [DISCUSS] Documentation - Description for Adapters and Processing Elements
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Florian,
thanks for sorting the adapters.
Maybe we can extend the category for each type (e.g. for all message brokers I would add ‘Generic’).
The list looks good so far, I guess the mentioned adapters were removed.
Let’s use the adapters in the git repository as our source of truth, I guess the website can be outdated.
Philipp
Von: Florian Micklich <mi...@apache.org>
Datum: Mittwoch, 14. September 2022 um 23:12
An: dev@streampipes.apache.org <de...@streampipes.apache.org>
Betreff: Re: [DISCUSS] Documentation - Description for Adapters and Processing Elements
Hi all,
I sorted the Adapters a little bit, to start the documentation
The following adapters are mentioned on the website, but are not
included in my installation.
Are they outdated or somehow lost?
MySQL Set Adapter
MySql Stream Adapter
NSW Traffic Cameras
The other adapters are divided into 2 groups.
Is something missing or does it need to be corrected in some way?
Some adapters do not belong to any category.
connect-adapters-iiot
NAME | TYPE | CATEGORIES
Apache Kafka | Data Stream
Apache Pulsar | Data Stream
File Set | Data Set
File Stream | Data Stream
HTTP Server | Data Stream
HTTP Set | Data Set
HTTP Stream | Data Stream
InfluxDB Set Adapter | Data Set
InfluxDB Stream Adapter | Data Stream
Machine Data Simulator | Data Stream | Testing &
Debugging
MQTT | Data Stream
OPC UA | Data Stream | Generic
Adapters, Production & Manufactoring
PLC4X MODBUS | Data Stream | Production &
Manufactoring
PLC4X S7 | Data Stream | Production &
Manufactoring
ROS Bridge | Data Stream | Production &
Manufactoring
connect-adapters
NAME | TYPE | CATEGORIES
Coindesk Bitcoin Price | Data Stream | Finance
Flic MQTT | Data Stream | Energy
GDELT | Data Stream
IEX Cloud News | Data Stream | Finance, News
IEX Cloud Stock Quotes | Data Stream | Finance
Image Upload (Set) | Data Set
Image Upload (Stream) | Data Stream
ISS Location | Data Stream | Open Data
NETIO http JSON | Data Stream | Energy
NETIO MQTT M2M | Data Stream | Energy
Random Data Simulator (Set) | Data Set | Testing &
Debugging
Random Data Simulator (Stream) | Data Stream | Testing &
Debugging
Slack | Data Stream | Social Media
TI Sensor Tag | Data Stream | Open Data
Wikipedia Edits | Data Stream | Social Media,
Open Data
Wikipedia New Articles | Data Stream | Social Media,
Open Data
Greetings
Florian
Am Mittwoch, dem 14.09.2022 um 22:52 +0200 schrieb Florian Micklich:
> Hi,
> just recognized that the format of my list was totally unreadable
> inthe emailI added my list in a propper way here [1]
> Florian
> [1] https://issues.apache.org/jira/browse/STREAMPIPES-581
>
> Am Mittwoch, dem 14.09.2022 um 16:52 +0200 schrieb Florian Micklich:
> > Hi all,I made a rouge structure about a possible
> > documentation. • Introduction: ◦ Overview ◦ Basic
> > Elements (formerStreamPipes Concept) ◦ Time in StreamPipes
> > (Why timestamp isimportant in the data?) ◦ Semantic in
> > StreamPipes ◦OtherTopics? ◦ Installation
> > ▪ BasicInstallation ▪ Docker Deployment
> > ▪ K8s/K3s ▪ Use SSL ▪
> > Security ◦GettingHelp • Overview (Basic GUI Explanation
> > and Introduction for eachGUIpoint and why it is implemented (short
> > description text) for exampleDashboard why Dashboard, what you can
> > do. Basically little more deepdiver than Basic Elements section.
> > Section with Most of theScreenshots, So easy to maintain due
> > changes) ◦ Login Page andBasics ◦ Welcome
> > Page ◦ Pipelines ◦Connect ◦ Dashboard ◦
> > Data Explorer ◦ Apps(deprecated?) ◦ Install Pipeline
> > Elements ◦ AssetManagement ◦ File Management ◦
> > Configuration & UserPrefences ◦ Notification • Data
> > Source Elements(Adapter) ◦ Why Adapter. Connect to the Real
> > World ◦WhatKind of Sources (Data Set vs Data Stream /
> > Categories) ◦ BasicAdapter Setup in 4 Steps (at the
> > moment) ▪ Step 1Settings ▪ Step 2
> > Select Format •JSONwith Description for Array
> > Field / GeoJSON / Single Object / ArraywithShort Data
> > Examples • XML •CSV
> > • Image ▪ Step 3
> > configurefields • Different
> > between Measurement,DimensionHeader • Add
> > Nested / Add static Value / AddTimestamp • Edit
> > fields (Big Thinks.Also) • Preview
> > ▪ Step4 startadapters • Settings /
> > Description (why Importantand where can I find this info later
> > on?) •Remove Duplicates / Reduce event rates
> > /Persist events ◦ConnectAdapters Overview
> > ▪ Apache Kafka with ExampleSources ▪ Apache
> > Pulsar ▪ FileSet ▪ File
> > Stream ▪ ... ◦ConnectAdapters IIOT
> > Overview ▪ ... ◦ How ToAdapter ▪
> > Setup Adapter Example 1 ▪SetupHTTPS Adapter ◦
> > Technical Concept of Adapters (DevPart???) •Data Processors •
> > Data Sinks • Create Pipeline •Dashboard • AssetsThe
> > working example will be in the section HowTo and some
> > exampleresources in each adapter description.What do you
> > think?First step would be to check if it possible to move the files
> > so thatthe structure is not destroyed and the simply start with the
> > adapters:)I made already a rebase between dev and STREAMPIPES-581
> > branch, duesome changes in the dev branch.FlorianAm Montag, dem
> > 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder:
> > > Hi Florian,I am not sure if we will find many available public
> > > data sources,especially for PLCs. But maybe we find some docker
> > > containers thatcan be used for the tutorial. If we do not find
> > > some, maybe we canprovide them in our registry.We could also use
> > > them for theautomatedtests. That would be great, because
> > > currently those tests are notintegrated in the nightly run.I
> > > think for most users it is already helpful to have a
> > > workingexample for the different adapters, so they do not
> > > necessarily haveto try out the examples.As you said we should
> > > collect all the information somewhere. Shouldwe use thwiki for
> > > that?Cheers,Philipp________________________________Von: Florian
> > > Micklich<micklich@apache.org
> > > > Gesendet: Samstag, August 27, 2022 1:20 AMAn:
> > > dev@streampipes.apache.org
> > > <dev@streampipes.apache.org
> > > > Betreff: Re:
> > > [DISCUSS] Documentation - Description for Adapters and
> > > ProcessingElementsHi,perfekt, I will create this branch. I will
> > > think about apossiblestructure over the weekend. I let you
> > > know.How are youtesting and using the adapters and wich one are
> > > you usingmost? Doyouhave some URL sources you always for
> > > developing processor testingand what is the typical workflow to
> > > add them?Maybe we can start tocollect them in some way so I can
> > > add them later inthedocumentation.GreetingsFlorianAm Donnerstag,
> > > dem 25.08.2022 um 19:38 +0000 schrieb DominikRiemer:
> > > > Sounds good!We usually use only the issue name as
> > > > branchname(STREAMPIPES-581) without the "feature" prefix, but
> > > > both istotallyfine!Dominik-----Original Message-----From:
> > > > Florian Micklich <micklich@apache.org
> > > >
> > > > > Sent: Wednesday, August 24, 2022 9:44 PMTo:
> > > >
> > > > dev@streampipes.apache.org
> > > >
> > > > Subject: Re: [DISCUSS] Documentation - Description for
> > > > AdaptersandProcessing ElementsHi Dominik,thanks for the answer
> > > > and beforeIask more questions,I'll simply start :-)Therefore I
> > > > created aticket
> > > > https://issues.apache.org/jira/browse/STREAMPIPES-581
> > > >
> > > > I will also create a feature branch feature/STREAMPIPES-581
> > > > orwhatis here best practise naming convention?GreetingFlorian
> > > > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb
> > > > DominikRiemer:
> > > > > Hi Florian,happy to help! 1. There is a header in
> > > > > eachmarkdownfilewith an ID (e.g. in the 01_try-
> > > > > installation.md).Thisidcorrespondsto the id in the
> > > > > sidebar.json2. Yes, all imagesareinthe staticfolder 3. Well,
> > > > > they are not really auto-autogenerated😉 There is amaven
> > > > > plugin which extracts the markdownfiles ande.g.,
> > > > > createsentries for the sidebar.json and modifiesthe
> > > > > imagelinks so thatthey work on the website, but it's
> > > > > notsomethingautomated in Githubactions or so, although it
> > > > > would beawesome tosupport this4. Youdon't need to manually
> > > > > add theversion, this ismanaged byDocusaurus. The markdown
> > > > > files in thedocumentationfolder alwaysrefer to the next
> > > > > unreleased version.When you startthe website, youcan click on
> > > > > the version (0.69.0should be thedefault) and thenselect
> > > > > "master" which points youtothe unreleasedversion. Aftereach
> > > > > release, we run the "ds-version" command andthen
> > > > > Docusauruschecks all files in the documentationfolder
> > > > > againstthe files fromthe previous release and creates anew
> > > > > docs version.E.g., see olderdocumentation files at [1].
> > > > > Incase you want tobackport changes tothe documentation pages
> > > > > toprevious versions,these files can also bemodified, but
> > > > > that'srather rare.5. Goodpoint, I _think_ this shouldbe
> > > > > possiblewithout any problems, atleast it's worth a try 😉Hope
> > > > > thishelps!CheersDominik[1]
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----From: Florian Micklich <
> > > > > micklich@apache.org
> > > > >
> > > > > > Sent: Wednesday, August 24, 2022 7:59 PMTo:
> > > > >
> > > > > dev@streampipes.apache.org
> > > > >
> > > > > Subject: Re: [DISCUSS] Documentation - Description
> > > > > forAdaptersandProcessing ElementsHi,ok, I started docusaurus
> > > > > fromsourceDominik has previouslymentioned and did a build &
> > > > > runwebsite likementioned in theReadme.Next time I am
> > > > > even faster,because I willstart the "BuildDocumentation"
> > > > > from thedocumentation/websitefolder, as it itclearly
> > > > > mentioned in theReadme :-D Before I start,I am trying
> > > > > tounderstand thestructureof this documentation and Ihave
> > > > > somequestions.1.So everythingisorganised by thesidebar.jsonSo
> > > > > the try-installation [1] islinkedto the 01_try-
> > > > > installation.md file[2]. How is this working ifthe names
> > > > > doesnot match?2. I see that all images are stored inthe
> > > > > static websitepath here[3]. Is this correct?3. Do Iunderstand
> > > > > it correctly thatall filesin the PE folder [4] areauto
> > > > > generated and will beautogenerated next time again? SoifI
> > > > > change something here,will thisbe overwritten the
> > > > > nexttime?4.Is it OK if I add the 0.70version inthe version
> > > > > filehere[5]?5. Is it OK to structure thedocuments insome more
> > > > > sup-folders? We could discuss the sub-foldernames here
> > > > > ofcourse butIthink that would make the documentationlittle
> > > > > biteasier tonavigate for the authors. This includes
> > > > > alsopictures instep2.Greetings Florian[1]
> > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2]
> > > > >
> > > > >
> > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]<https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md%5b3%5d>
> > > > >
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]<https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img%5b4%5d>
> > > > >
> > > > > https://github.com
> > > > >
> > > > > /apache/incubator-streampipes-
> > > > > website/tree/dev/documentation/docs/pe[5]
> > > > > https://github.com/apache/inc
> > > > > ubator-streampipes-
> > > > > website/blob/dev/documentation/website/versions.jsonAm
> > > > > Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb
> > > > > PhilippZehnder:
> > > > > > Hi together,it would be great to have more
> > > > > > contentinthedocumentation.I am alsoin favor of using
> > > > > > docusaurus.Itusesmarkdown, which leaves us thepossibility
> > > > > > to move toanothertoolingat some point in the future ifwe
> > > > > > want.@Florianpleasewrite if yousee something that should be
> > > > > > updatedorchanged
> > > > > > inthecurrentversion.Cheers,Philipp_________________________
> > > > > > _______Von:DominikRiemer <dominik.riemer@bytefabrik.ai
> > > > > > > Gesendet: Mittwoch,August
> > > > > > 24,202211:47 AMAn: dev@streampipes.apache.org
> > > > > >
> > > > > > <dev@streampipes.apache.org
> > > > > > > Betreff: RE: [DISCUSS]
> > > > > > Documentation-Description for Adapters
> > > > > > andProcessingElementsHiFlorian,Cool!Thedocumentation on the
> > > > > > website is based onDocusauruswhichsupportsthings like docs
> > > > > > versioning.Docusaurususesmarkdown files,so werely on
> > > > > > markdown for thedocumentation.Thereis a 'website' folderat
> > > > > > [1] where you canfind the npm scriptstostart docusaurus and
> > > > > > a'docs' folderwhichincludesthedocumentation markdown files
> > > > > > forthe next releaseversion(selectg"next" in the docs
> > > > > > website). Aftera newrelease,we run a"npm ds-version" which
> > > > > > then detects changesin themarkdown filesand createsa new
> > > > > > release docsversion.Thedocuments in the "docs"folder can be
> > > > > > freelymodifiedandadded.That's a good place wherewe can add
> > > > > > new tutorials.The"pe"foldercontains thedocumentation of
> > > > > > pipeline elements.Thesearecurrentlygeneratedfrom the
> > > > > > StreamPipes source code usingthestreampipes-maven-plugin
> > > > > > (as some rewriting is needed,e.g.,to fixthe pathsofimages).
> > > > > > This process is far from perfectandneedssome manualwork.It
> > > > > > would be nice to have some
> > > > > > Githubactionsscriptwhichautomatically updates the
> > > > > > Documentation,butthat's abitmoreadvanced task I'd say 😉If
> > > > > > you want to makechanges tothestructure, this can be done
> > > > > > inthe"sidebars.json"file [3],whichis also auto-versioned by
> > > > > > Docusaurus.[1]
> > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website
> > > > > >
> > > > > > [2]
> > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs
> > > > > >
> > > > > > [3]
> > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json
> > > > > >
> > > > > >
> > > > > > -----Original Message-----From: Florian Micklich <
> > > > > > micklich@apache.org
> > > > > >
> > > > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo:
> > > > > >
> > > > > > dev@streampipes.apache.org
> > > > > >
> > > > > > Subject: Re: [DISCUSS] Documentation -
> > > > > > DescriptionforAdaptersandProcessing ElementsHello,I would
> > > > > > also prefer ashortIn-Appdocumentation and
> > > > > > largerdocumentation "outside".Personally, Iprefer to read
> > > > > > pdf documentsso I can takeadditionalnotes. ButHTML is also
> > > > > > good for a quickonlinesearch.We must keepin mindwhat kind
> > > > > > of documentation wewanttocreate and should notmixthem.
> > > > > > Also, cross-references betweentheindividual"chapters"should
> > > > > > be avoided. This makes
> > > > > > iteasierifsomethingshouldchange. 1. Quick
> > > > > > starterdocumentation. 2.SetupDocumentation 3.TechnicalD
> > > > > > ocumentation 4. Usermanualdocumentation with
> > > > > > bestpractiseIthink we should hit theroad andtake a look
> > > > > > along the way :)Iknow the in appdocumentation is inmarkdown
> > > > > > so far?Iwouldrecommend to write thedocumentation
> > > > > > inasciidoc [1] [2]This couldbe set up withasciidoctorj [3].
> > > > > > Thismakes it easytoincludeeverything in Gitand also keep
> > > > > > thedocumentationalongside thesource code. So ifsomething
> > > > > > isrewritten, thedeveloper can alsochange thedocumentation.
> > > > > > Makesnewscreenshotsand deletes oldones.3] Thecreation of a
> > > > > > PDF orHTML file canbedone withGithubAction. With every
> > > > > > commitor with everyrelease. Theresults can bepushed to the
> > > > > > StreamPipeswebsite, Iguess?What doyouthink?Florian[1]
> > > > > > https://docs.asciidoctor.org/asciidoc/latest/[2]<https://docs.asciidoctor.org/asciidoc/latest/%5b2%5d>
> > > > > >
> > > > > > German Source
> > > > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]<https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor%5b3%5d>
> > > > > >
> > > > > > https://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.2022 um 07:07 +0000 schrieb Dominik
> > > > > > Riemer:
> > > > > > > Hi,yes, we can do both: In-app
> > > > > > > documentationwhichonlycoverstheconfiguration and a short
> > > > > > > description oftheadapterandpipelineelement and additional
> > > > > > > step-by-stepguides/tutorialswithnicescreenshots that are
> > > > > > > availableintheonlinedocumentation. Wecanhave such step-
> > > > > > > by-step guidesfor themostfrequently used
> > > > > > > adapters.@Florian, do you wanttowork onsuch aguide on
> > > > > > > thewebsite? I knowyou had some
> > > > > > > reallygooddocumentationideas somewhile ago
> > > > > > > 😉CheersDominik-----OriginalMessage-----From: Philipp
> > > > > > > Zehnder <philipp.zehnder@bytefabrik.ai
> > > > > > > > Sent:Sunday, August
> > > > > > > 21,20221:07PMTo: dev@streampipes.apache.org
> > > > > > >
> > > > > > > Subject: [DISCUSS] Documentation
> > > > > > > -DescriptionforAdaptersandProcessing Elements Hi all,
> > > > > > > hereisa new threadtodiscusstheadapter documentation.Here
> > > > > > > is alinkto thepreviousdiscussionin therelease thread
> > > > > > > [1].@Dominik Iknow thattheregistration isdifferent, but
> > > > > > > since alladaptersalready havethethree
> > > > > > > files(documentation.md,
> > > > > > > icon.png,andstring.en),shouldn’tit bepossible to also
> > > > > > > display thedescriptionin theUI.How are wedoingcurrently
> > > > > > > for theicons?Because they areloaded dynamicallyaswell. I
> > > > > > > think themaindifference is thatthe icons
> > > > > > > areloadeddirectly from theworkercontainer and notfrom
> > > > > > > thebackend.Couldn’t we do itsimilarly forthedocumentation
> > > > > > > file?(Pleasecorrect me if Iamwrong)Regardingthe
> > > > > > > description on theWebsite.I think itwouldbegreat if we
> > > > > > > couldgenerate this basedon themarkdown filesforeach
> > > > > > > adapter.The documentation.md
> > > > > > > filesmostlydescribetheconfigurationparameters of adapters
> > > > > > > andprocessors.Ithinkitwould be great tohave additional
> > > > > > > exampleshow to usethem.(e.g.describe how a CSVfile is
> > > > > > > uploaded or how aPLCisconnected).Should we include
> > > > > > > suchexamplesintothedocumentation.md file orshould there
> > > > > > > beanotherlocation forthis?Any thoughts onthat?@Florian
> > > > > > > youwrote that you would liketo workon thedocumentation.If
> > > > > > > youneed anything to start
> > > > > > > with,pleaseletusknow.Cheers,Philipp[1]
> > > > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg
> > > > > > >
The other adapters are divided into 2 groups.
Re: [DISCUSS] Documentation - Description for Adapters and Processing Elements
Posted by Florian Micklich <mi...@apache.org>.
Hi all,
I sorted the Adapters a little bit, to start the documentation
The following adapters are mentioned on the website, but are not
included in my installation.
Are they outdated or somehow lost?
MySQL Set Adapter
MySql Stream Adapter
NSW Traffic Cameras
The other adapters are divided into 2 groups.
Is something missing or does it need to be corrected in some way?
Some adapters do not belong to any category.
connect-adapters-iiot
NAME | TYPE | CATEGORIES
Apache Kafka | Data Stream
Apache Pulsar | Data Stream
File Set | Data Set
File Stream | Data Stream
HTTP Server | Data Stream
HTTP Set | Data Set
HTTP Stream | Data Stream
InfluxDB Set Adapter | Data Set
InfluxDB Stream Adapter | Data Stream
Machine Data Simulator | Data Stream | Testing &
Debugging
MQTT | Data Stream
OPC UA | Data Stream | Generic
Adapters, Production & Manufactoring
PLC4X MODBUS | Data Stream | Production &
Manufactoring
PLC4X S7 | Data Stream | Production &
Manufactoring
ROS Bridge | Data Stream | Production &
Manufactoring
connect-adapters
NAME | TYPE | CATEGORIES
Coindesk Bitcoin Price | Data Stream | Finance
Flic MQTT | Data Stream | Energy
GDELT | Data Stream
IEX Cloud News | Data Stream | Finance, News
IEX Cloud Stock Quotes | Data Stream | Finance
Image Upload (Set) | Data Set
Image Upload (Stream) | Data Stream
ISS Location | Data Stream | Open Data
NETIO http JSON | Data Stream | Energy
NETIO MQTT M2M | Data Stream | Energy
Random Data Simulator (Set) | Data Set | Testing &
Debugging
Random Data Simulator (Stream) | Data Stream | Testing &
Debugging
Slack | Data Stream | Social Media
TI Sensor Tag | Data Stream | Open Data
Wikipedia Edits | Data Stream | Social Media,
Open Data
Wikipedia New Articles | Data Stream | Social Media,
Open Data
Greetings
Florian
Am Mittwoch, dem 14.09.2022 um 22:52 +0200 schrieb Florian Micklich:
> Hi,
> just recognized that the format of my list was totally unreadable
> inthe emailI added my list in a propper way here [1]
> Florian
> [1] https://issues.apache.org/jira/browse/STREAMPIPES-581
>
> Am Mittwoch, dem 14.09.2022 um 16:52 +0200 schrieb Florian Micklich:
> > Hi all,I made a rouge structure about a possible
> > documentation. • Introduction: ◦ Overview ◦ Basic
> > Elements (formerStreamPipes Concept) ◦ Time in StreamPipes
> > (Why timestamp isimportant in the data?) ◦ Semantic in
> > StreamPipes ◦OtherTopics? ◦ Installation
> > ▪ BasicInstallation ▪ Docker Deployment
> > ▪ K8s/K3s ▪ Use SSL ▪
> > Security ◦GettingHelp • Overview (Basic GUI Explanation
> > and Introduction for eachGUIpoint and why it is implemented (short
> > description text) for exampleDashboard why Dashboard, what you can
> > do. Basically little more deepdiver than Basic Elements section.
> > Section with Most of theScreenshots, So easy to maintain due
> > changes) ◦ Login Page andBasics ◦ Welcome
> > Page ◦ Pipelines ◦Connect ◦ Dashboard ◦
> > Data Explorer ◦ Apps(deprecated?) ◦ Install Pipeline
> > Elements ◦ AssetManagement ◦ File Management ◦
> > Configuration & UserPrefences ◦ Notification • Data
> > Source Elements(Adapter) ◦ Why Adapter. Connect to the Real
> > World ◦WhatKind of Sources (Data Set vs Data Stream /
> > Categories) ◦ BasicAdapter Setup in 4 Steps (at the
> > moment) ▪ Step 1Settings ▪ Step 2
> > Select Format •JSONwith Description for Array
> > Field / GeoJSON / Single Object / ArraywithShort Data
> > Examples • XML •CSV
> > • Image ▪ Step 3
> > configurefields • Different
> > between Measurement,DimensionHeader • Add
> > Nested / Add static Value / AddTimestamp • Edit
> > fields (Big Thinks.Also) • Preview
> > ▪ Step4 startadapters • Settings /
> > Description (why Importantand where can I find this info later
> > on?) •Remove Duplicates / Reduce event rates
> > /Persist events ◦ConnectAdapters Overview
> > ▪ Apache Kafka with ExampleSources ▪ Apache
> > Pulsar ▪ FileSet ▪ File
> > Stream ▪ ... ◦ConnectAdapters IIOT
> > Overview ▪ ... ◦ How ToAdapter ▪
> > Setup Adapter Example 1 ▪SetupHTTPS Adapter ◦
> > Technical Concept of Adapters (DevPart???) •Data Processors •
> > Data Sinks • Create Pipeline •Dashboard • AssetsThe
> > working example will be in the section HowTo and some
> > exampleresources in each adapter description.What do you
> > think?First step would be to check if it possible to move the files
> > so thatthe structure is not destroyed and the simply start with the
> > adapters:)I made already a rebase between dev and STREAMPIPES-581
> > branch, duesome changes in the dev branch.FlorianAm Montag, dem
> > 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder:
> > > Hi Florian,I am not sure if we will find many available public
> > > data sources,especially for PLCs. But maybe we find some docker
> > > containers thatcan be used for the tutorial. If we do not find
> > > some, maybe we canprovide them in our registry.We could also use
> > > them for theautomatedtests. That would be great, because
> > > currently those tests are notintegrated in the nightly run.I
> > > think for most users it is already helpful to have a
> > > workingexample for the different adapters, so they do not
> > > necessarily haveto try out the examples.As you said we should
> > > collect all the information somewhere. Shouldwe use thwiki for
> > > that?Cheers,Philipp________________________________Von: Florian
> > > Micklich<micklich@apache.org
> > > > Gesendet: Samstag, August 27, 2022 1:20 AMAn:
> > > dev@streampipes.apache.org
> > > <dev@streampipes.apache.org
> > > > Betreff: Re:
> > > [DISCUSS] Documentation - Description for Adapters and
> > > ProcessingElementsHi,perfekt, I will create this branch. I will
> > > think about apossiblestructure over the weekend. I let you
> > > know.How are youtesting and using the adapters and wich one are
> > > you usingmost? Doyouhave some URL sources you always for
> > > developing processor testingand what is the typical workflow to
> > > add them?Maybe we can start tocollect them in some way so I can
> > > add them later inthedocumentation.GreetingsFlorianAm Donnerstag,
> > > dem 25.08.2022 um 19:38 +0000 schrieb DominikRiemer:
> > > > Sounds good!We usually use only the issue name as
> > > > branchname(STREAMPIPES-581) without the "feature" prefix, but
> > > > both istotallyfine!Dominik-----Original Message-----From:
> > > > Florian Micklich <micklich@apache.org
> > > >
> > > > > Sent: Wednesday, August 24, 2022 9:44 PMTo:
> > > >
> > > > dev@streampipes.apache.org
> > > >
> > > > Subject: Re: [DISCUSS] Documentation - Description for
> > > > AdaptersandProcessing ElementsHi Dominik,thanks for the answer
> > > > and beforeIask more questions,I'll simply start :-)Therefore I
> > > > created aticket
> > > > https://issues.apache.org/jira/browse/STREAMPIPES-581
> > > >
> > > > I will also create a feature branch feature/STREAMPIPES-581
> > > > orwhatis here best practise naming convention?GreetingFlorian
> > > > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb
> > > > DominikRiemer:
> > > > > Hi Florian,happy to help! 1. There is a header in
> > > > > eachmarkdownfilewith an ID (e.g. in the 01_try-
> > > > > installation.md).Thisidcorrespondsto the id in the
> > > > > sidebar.json2. Yes, all imagesareinthe staticfolder 3. Well,
> > > > > they are not really auto-autogenerated😉 There is amaven
> > > > > plugin which extracts the markdownfiles ande.g.,
> > > > > createsentries for the sidebar.json and modifiesthe
> > > > > imagelinks so thatthey work on the website, but it's
> > > > > notsomethingautomated in Githubactions or so, although it
> > > > > would beawesome tosupport this4. Youdon't need to manually
> > > > > add theversion, this ismanaged byDocusaurus. The markdown
> > > > > files in thedocumentationfolder alwaysrefer to the next
> > > > > unreleased version.When you startthe website, youcan click on
> > > > > the version (0.69.0should be thedefault) and thenselect
> > > > > "master" which points youtothe unreleasedversion. Aftereach
> > > > > release, we run the "ds-version" command andthen
> > > > > Docusauruschecks all files in the documentationfolder
> > > > > againstthe files fromthe previous release and creates anew
> > > > > docs version.E.g., see olderdocumentation files at [1].
> > > > > Incase you want tobackport changes tothe documentation pages
> > > > > toprevious versions,these files can also bemodified, but
> > > > > that'srather rare.5. Goodpoint, I _think_ this shouldbe
> > > > > possiblewithout any problems, atleast it's worth a try 😉Hope
> > > > > thishelps!CheersDominik[1]
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----From: Florian Micklich <
> > > > > micklich@apache.org
> > > > >
> > > > > > Sent: Wednesday, August 24, 2022 7:59 PMTo:
> > > > >
> > > > > dev@streampipes.apache.org
> > > > >
> > > > > Subject: Re: [DISCUSS] Documentation - Description
> > > > > forAdaptersandProcessing ElementsHi,ok, I started docusaurus
> > > > > fromsourceDominik has previouslymentioned and did a build &
> > > > > runwebsite likementioned in theReadme.Next time I am
> > > > > even faster,because I willstart the "BuildDocumentation"
> > > > > from thedocumentation/websitefolder, as it itclearly
> > > > > mentioned in theReadme :-D Before I start,I am trying
> > > > > tounderstand thestructureof this documentation and Ihave
> > > > > somequestions.1.So everythingisorganised by thesidebar.jsonSo
> > > > > the try-installation [1] islinkedto the 01_try-
> > > > > installation.md file[2]. How is this working ifthe names
> > > > > doesnot match?2. I see that all images are stored inthe
> > > > > static websitepath here[3]. Is this correct?3. Do Iunderstand
> > > > > it correctly thatall filesin the PE folder [4] areauto
> > > > > generated and will beautogenerated next time again? SoifI
> > > > > change something here,will thisbe overwritten the
> > > > > nexttime?4.Is it OK if I add the 0.70version inthe version
> > > > > filehere[5]?5. Is it OK to structure thedocuments insome more
> > > > > sup-folders? We could discuss the sub-foldernames here
> > > > > ofcourse butIthink that would make the documentationlittle
> > > > > biteasier tonavigate for the authors. This includes
> > > > > alsopictures instep2.Greetings Florian[1]
> > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2]
> > > > >
> > > > >
> > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]
> > > > >
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]
> > > > >
> > > > > https://github.com
> > > > >
> > > > > /apache/incubator-streampipes-
> > > > > website/tree/dev/documentation/docs/pe[5]
> > > > > https://github.com/apache/inc
> > > > > ubator-streampipes-
> > > > > website/blob/dev/documentation/website/versions.jsonAm
> > > > > Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb
> > > > > PhilippZehnder:
> > > > > > Hi together,it would be great to have more
> > > > > > contentinthedocumentation.I am alsoin favor of using
> > > > > > docusaurus.Itusesmarkdown, which leaves us thepossibility
> > > > > > to move toanothertoolingat some point in the future ifwe
> > > > > > want.@Florianpleasewrite if yousee something that should be
> > > > > > updatedorchanged
> > > > > > inthecurrentversion.Cheers,Philipp_________________________
> > > > > > _______Von:DominikRiemer <dominik.riemer@bytefabrik.ai
> > > > > > > Gesendet: Mittwoch,August
> > > > > > 24,202211:47 AMAn: dev@streampipes.apache.org
> > > > > >
> > > > > > <dev@streampipes.apache.org
> > > > > > > Betreff: RE: [DISCUSS]
> > > > > > Documentation-Description for Adapters
> > > > > > andProcessingElementsHiFlorian,Cool!Thedocumentation on the
> > > > > > website is based onDocusauruswhichsupportsthings like docs
> > > > > > versioning.Docusaurususesmarkdown files,so werely on
> > > > > > markdown for thedocumentation.Thereis a 'website' folderat
> > > > > > [1] where you canfind the npm scriptstostart docusaurus and
> > > > > > a'docs' folderwhichincludesthedocumentation markdown files
> > > > > > forthe next releaseversion(selectg"next" in the docs
> > > > > > website). Aftera newrelease,we run a"npm ds-version" which
> > > > > > then detects changesin themarkdown filesand createsa new
> > > > > > release docsversion.Thedocuments in the "docs"folder can be
> > > > > > freelymodifiedandadded.That's a good place wherewe can add
> > > > > > new tutorials.The"pe"foldercontains thedocumentation of
> > > > > > pipeline elements.Thesearecurrentlygeneratedfrom the
> > > > > > StreamPipes source code usingthestreampipes-maven-plugin
> > > > > > (as some rewriting is needed,e.g.,to fixthe pathsofimages).
> > > > > > This process is far from perfectandneedssome manualwork.It
> > > > > > would be nice to have some
> > > > > > Githubactionsscriptwhichautomatically updates the
> > > > > > Documentation,butthat's abitmoreadvanced task I'd say 😉If
> > > > > > you want to makechanges tothestructure, this can be done
> > > > > > inthe"sidebars.json"file [3],whichis also auto-versioned by
> > > > > > Docusaurus.[1]
> > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website
> > > > > >
> > > > > > [2]
> > > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs
> > > > > >
> > > > > > [3]
> > > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json
> > > > > >
> > > > > >
> > > > > > -----Original Message-----From: Florian Micklich <
> > > > > > micklich@apache.org
> > > > > >
> > > > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo:
> > > > > >
> > > > > > dev@streampipes.apache.org
> > > > > >
> > > > > > Subject: Re: [DISCUSS] Documentation -
> > > > > > DescriptionforAdaptersandProcessing ElementsHello,I would
> > > > > > also prefer ashortIn-Appdocumentation and
> > > > > > largerdocumentation "outside".Personally, Iprefer to read
> > > > > > pdf documentsso I can takeadditionalnotes. ButHTML is also
> > > > > > good for a quickonlinesearch.We must keepin mindwhat kind
> > > > > > of documentation wewanttocreate and should notmixthem.
> > > > > > Also, cross-references betweentheindividual"chapters"should
> > > > > > be avoided. This makes
> > > > > > iteasierifsomethingshouldchange. 1. Quick
> > > > > > starterdocumentation. 2.SetupDocumentation 3.TechnicalD
> > > > > > ocumentation 4. Usermanualdocumentation with
> > > > > > bestpractiseIthink we should hit theroad andtake a look
> > > > > > along the way :)Iknow the in appdocumentation is inmarkdown
> > > > > > so far?Iwouldrecommend to write thedocumentation
> > > > > > inasciidoc [1] [2]This couldbe set up withasciidoctorj [3].
> > > > > > Thismakes it easytoincludeeverything in Gitand also keep
> > > > > > thedocumentationalongside thesource code. So ifsomething
> > > > > > isrewritten, thedeveloper can alsochange thedocumentation.
> > > > > > Makesnewscreenshotsand deletes oldones.3] Thecreation of a
> > > > > > PDF orHTML file canbedone withGithubAction. With every
> > > > > > commitor with everyrelease. Theresults can bepushed to the
> > > > > > StreamPipeswebsite, Iguess?What doyouthink?Florian[1]
> > > > > > https://docs.asciidoctor.org/asciidoc/latest/[2]
> > > > > >
> > > > > > German Source
> > > > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]
> > > > > >
> > > > > > https://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.2022 um 07:07 +0000 schrieb Dominik
> > > > > > Riemer:
> > > > > > > Hi,yes, we can do both: In-app
> > > > > > > documentationwhichonlycoverstheconfiguration and a short
> > > > > > > description oftheadapterandpipelineelement and additional
> > > > > > > step-by-stepguides/tutorialswithnicescreenshots that are
> > > > > > > availableintheonlinedocumentation. Wecanhave such step-
> > > > > > > by-step guidesfor themostfrequently used
> > > > > > > adapters.@Florian, do you wanttowork onsuch aguide on
> > > > > > > thewebsite? I knowyou had some
> > > > > > > reallygooddocumentationideas somewhile ago
> > > > > > > 😉CheersDominik-----OriginalMessage-----From: Philipp
> > > > > > > Zehnder <philipp.zehnder@bytefabrik.ai
> > > > > > > > Sent:Sunday, August
> > > > > > > 21,20221:07PMTo: dev@streampipes.apache.org
> > > > > > >
> > > > > > > Subject: [DISCUSS] Documentation
> > > > > > > -DescriptionforAdaptersandProcessing Elements Hi all,
> > > > > > > hereisa new threadtodiscusstheadapter documentation.Here
> > > > > > > is alinkto thepreviousdiscussionin therelease thread
> > > > > > > [1].@Dominik Iknow thattheregistration isdifferent, but
> > > > > > > since alladaptersalready havethethree
> > > > > > > files(documentation.md,
> > > > > > > icon.png,andstring.en),shouldn’tit bepossible to also
> > > > > > > display thedescriptionin theUI.How are wedoingcurrently
> > > > > > > for theicons?Because they areloaded dynamicallyaswell. I
> > > > > > > think themaindifference is thatthe icons
> > > > > > > areloadeddirectly from theworkercontainer and notfrom
> > > > > > > thebackend.Couldn’t we do itsimilarly forthedocumentation
> > > > > > > file?(Pleasecorrect me if Iamwrong)Regardingthe
> > > > > > > description on theWebsite.I think itwouldbegreat if we
> > > > > > > couldgenerate this basedon themarkdown filesforeach
> > > > > > > adapter.The documentation.md
> > > > > > > filesmostlydescribetheconfigurationparameters of adapters
> > > > > > > andprocessors.Ithinkitwould be great tohave additional
> > > > > > > exampleshow to usethem.(e.g.describe how a CSVfile is
> > > > > > > uploaded or how aPLCisconnected).Should we include
> > > > > > > suchexamplesintothedocumentation.md file orshould there
> > > > > > > beanotherlocation forthis?Any thoughts onthat?@Florian
> > > > > > > youwrote that you would liketo workon thedocumentation.If
> > > > > > > youneed anything to start
> > > > > > > with,pleaseletusknow.Cheers,Philipp[1]
> > > > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg
> > > > > > >
The other adapters are divided into 2 groups.
Re: [DISCUSS] Documentation - Description for Adapters and Processing Elements
Posted by Florian Micklich <mi...@apache.org>.
Hi,
just recognized that the format of my list was totally unreadable in
the email
I added my list in a propper way here [1]
Florian
[1] https://issues.apache.org/jira/browse/STREAMPIPES-581
Am Mittwoch, dem 14.09.2022 um 16:52 +0200 schrieb Florian Micklich:
> Hi all,
> I made a rouge structure about a possible documentation.
> • Introduction: ◦ Overview ◦ Basic Elements (former
> StreamPipes Concept) ◦ Time in StreamPipes (Why timestamp is
> important in the data?) ◦ Semantic in StreamPipes ◦
> Other
> Topics? ◦ Installation ▪ Basic
> Installation ▪ Docker Deployment ▪ K8s
> /
> K3s ▪ Use SSL ▪ Security ◦
> Getting
> Help • Overview (Basic GUI Explanation and Introduction for each
> GUI
> point and why it is implemented (short description text) for example
> Dashboard why Dashboard, what you can do. Basically little more deep
> diver than Basic Elements section. Section with Most of the
> Screenshots, So easy to maintain due changes) ◦ Login Page and
> Basics ◦ Welcome Page ◦ Pipelines ◦
> Connect ◦ Dashboard ◦ Data Explorer ◦ Apps
> (deprecated?) ◦ Install Pipeline Elements ◦ Asset
> Management ◦ File Management ◦ Configuration & User
> Prefences ◦ Notification • Data Source Elements
> (Adapter) ◦ Why Adapter. Connect to the Real World ◦
> What
> Kind of Sources (Data Set vs Data Stream / Categories) ◦ Basic
> Adapter Setup in 4 Steps (at the moment) ▪ Step 1
> Settings ▪ Step 2 Select Format •
> JSON
> with Description for Array Field / GeoJSON / Single Object / Array
> with
> Short Data Examples • XML •
> CSV
> • Image ▪ Step 3 configure
> fields • Different between Measurement,
> Dimension
> Header • Add Nested / Add static Value / Add
> Timestamp • Edit fields (Big Thinks.
> Also) • Preview ▪ Step4 start
> adapters • Settings / Description (why Important
> and where can I find this info later on?) •
> Remove Duplicates / Reduce event rates /Persist events ◦
> Connect
> Adapters Overview ▪ Apache Kafka with Example
> Sources ▪ Apache Pulsar ▪ File
> Set ▪ File Stream ▪ ... ◦
> Connect
> Adapters IIOT Overview ▪ ... ◦ How To
> Adapter ▪ Setup Adapter Example 1 ▪
> Setup
> HTTPS Adapter ◦ Technical Concept of Adapters (Dev
> Part???) •
> Data Processors • Data Sinks • Create Pipeline •
> Dashboard • Assets
> The working example will be in the section HowTo and some example
> resources in each adapter description.What do you think?
> First step would be to check if it possible to move the files so that
> the structure is not destroyed and the simply start with the adapters
> :)I made already a rebase between dev and STREAMPIPES-581 branch, due
> some changes in the dev branch.
> Florian
> Am Montag, dem 29.08.2022 um 01:07 +0000 schrieb Philipp Zehnder:
> > Hi Florian,
> > I am not sure if we will find many available public data sources,
> > especially for PLCs. But maybe we find some docker containers that
> > can be used for the tutorial. If we do not find some, maybe we can
> > provide them in our registry.We could also use them for the
> > automated
> > tests. That would be great, because currently those tests are not
> > integrated in the nightly run.
> > I think for most users it is already helpful to have a working
> > example for the different adapters, so they do not necessarily have
> > to try out the examples.
> > As you said we should collect all the information somewhere. Should
> > we use thwiki for that?
> > Cheers,Philipp________________________________Von: Florian Micklich
> > <
> > micklich@apache.org
> > >Gesendet: Samstag, August 27, 2022 1:20 AMAn:
> > dev@streampipes.apache.org
> > <
> > dev@streampipes.apache.org
> > >Betreff: Re:
> > [DISCUSS] Documentation - Description for Adapters and Processing
> > Elements
> > Hi,perfekt, I will create this branch. I will think about a
> > possiblestructure over the weekend. I let you know.How are you
> > testing and using the adapters and wich one are you usingmost? Do
> > you
> > have some URL sources you always for developing processor testing
> > and what is the typical workflow to add them?Maybe we can start to
> > collect them in some way so I can add them later in
> > thedocumentation.GreetingsFlorian
> > Am Donnerstag, dem 25.08.2022 um 19:38 +0000 schrieb Dominik
> > Riemer:
> > > Sounds good!We usually use only the issue name as branch
> > > name(STREAMPIPES-581) without the "feature" prefix, but both is
> > > totallyfine!Dominik
> > > -----Original Message-----From: Florian Micklich <
> > > micklich@apache.org
> > >
> > > > Sent: Wednesday, August 24, 2022 9:44 PMTo:
> > >
> > > dev@streampipes.apache.org
> > >
> > > Subject: Re: [DISCUSS] Documentation - Description for Adapters
> > > andProcessing ElementsHi Dominik,thanks for the answer and before
> > > I
> > > ask more questions,I'll simply start :-)Therefore I created a
> > > ticket
> > > https://issues.apache.org/jira/browse/STREAMPIPES-581
> > >
> > > I will also create a feature branch feature/STREAMPIPES-581 or
> > > whatis here best practise naming convention?GreetingFlorian
> > >
> > > Am Mittwoch, dem 24.08.2022 um 19:07 +0000 schrieb Dominik
> > > Riemer:
> > > > Hi Florian,happy to help! 1. There is a header in each
> > > > markdownfilewith an ID (e.g. in the 01_try-installation.md).
> > > > This
> > > > idcorrespondsto the id in the sidebar.json2. Yes, all images
> > > > are
> > > > inthe staticfolder 3. Well, they are not really auto-auto
> > > > generated😉 There is amaven plugin which extracts the markdown
> > > > files ande.g., createsentries for the sidebar.json and modifies
> > > > the imagelinks so thatthey work on the website, but it's not
> > > > somethingautomated in Githubactions or so, although it would be
> > > > awesome tosupport this4. Youdon't need to manually add the
> > > > version, this ismanaged byDocusaurus. The markdown files in the
> > > > documentationfolder alwaysrefer to the next unreleased version.
> > > > When you startthe website, youcan click on the version (0.69.0
> > > > should be thedefault) and thenselect "master" which points you
> > > > to
> > > > the unreleasedversion. Aftereach release, we run the "ds-
> > > > version"
> > > > command andthen Docusauruschecks all files in the documentation
> > > > folder againstthe files fromthe previous release and creates a
> > > > new docs version.E.g., see olderdocumentation files at [1]. In
> > > > case you want tobackport changes tothe documentation pages to
> > > > previous versions,these files can also bemodified, but that's
> > > > rather rare.5. Goodpoint, I _think_ this shouldbe possible
> > > > without any problems, atleast it's worth a try 😉Hope this
> > > > helps!CheersDominik[1]
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/versioned_docs
> > > >
> > > >
> > > >
> > > > -----Original Message-----From: Florian Micklich <
> > > > micklich@apache.org
> > > >
> > > > > Sent: Wednesday, August 24, 2022 7:59 PMTo:
> > > >
> > > > dev@streampipes.apache.org
> > > >
> > > > Subject: Re: [DISCUSS] Documentation - Description for
> > > > AdaptersandProcessing ElementsHi,ok, I started docusaurus from
> > > > sourceDominik has previouslymentioned and did a build & run
> > > > website likementioned in theReadme.Next time I am even faster,
> > > > because I willstart the "BuildDocumentation" from the
> > > > documentation/websitefolder, as it itclearly mentioned in the
> > > > Readme :-D Before I start,I am trying tounderstand the
> > > > structure
> > > > of this documentation and Ihave somequestions.1.So everything
> > > > is
> > > > organised by thesidebar.jsonSo the try-installation [1] is
> > > > linked
> > > > to the 01_try-installation.md file[2]. How is this working if
> > > > the names doesnot match?2. I see that all images are stored in
> > > > the static websitepath here[3]. Is this correct?3. Do I
> > > > understand it correctly thatall filesin the PE folder [4] are
> > > > auto generated and will beautogenerated next time again? So
> > > > if
> > > > I change something here,will thisbe overwritten the next
> > > > time?4.Is it OK if I add the 0.70version inthe version file
> > > > here
> > > > [5]?5. Is it OK to structure thedocuments insome more sup-
> > > > folders? We could discuss the sub-foldernames here ofcourse but
> > > > I
> > > > think that would make the documentationlittle biteasier to
> > > > navigate for the authors. This includes alsopictures instep
> > > > 2.Greetings Florian[1]
> > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json#L5[2]
> > > >
> > > >
> > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/docs/01_try-installation.md[3]
> > > >
> > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website/static/img[4]
> > > >
> > > > https://github.com
> > > >
> > > > /apache/incubator-streampipes-
> > > > website/tree/dev/documentation/docs/pe[5]
> > > > https://github.com/apache/inc
> > > > ubator-streampipes-
> > > > website/blob/dev/documentation/website/versions.json
> > > > Am Mittwoch, dem 24.08.2022 um 12:32 +0000 schrieb Philipp
> > > > Zehnder:
> > > > > Hi together,it would be great to have more content
> > > > > inthedocumentation.I am alsoin favor of using docusaurus.
> > > > > Itusesmarkdown, which leaves us thepossibility to move to
> > > > > anothertoolingat some point in the future ifwe want.@Florian
> > > > > pleasewrite if yousee something that should be updated
> > > > > orchanged in
> > > > > thecurrentversion.Cheers,Philipp_____________________________
> > > > > ___Von:DominikRiemer <
> > > > > dominik.riemer@bytefabrik.ai
> > > > > >Gesendet: Mittwoch,August
> > > > > 24,202211:47 AMAn:
> > > > > dev@streampipes.apache.org
> > > > >
> > > > > <
> > > > > dev@streampipes.apache.org
> > > > > >Betreff: RE: [DISCUSS]
> > > > > Documentation-Description for Adapters andProcessing
> > > > > ElementsHi
> > > > > Florian,Cool!Thedocumentation on the website is based on
> > > > > Docusauruswhichsupportsthings like docs versioning.
> > > > > Docusaurus
> > > > > usesmarkdown files,so werely on markdown for the
> > > > > documentation.Thereis a 'website' folderat [1] where you can
> > > > > find the npm scriptstostart docusaurus and a'docs' folder
> > > > > which
> > > > > includesthedocumentation markdown files forthe next release
> > > > > version(selectg"next" in the docs website). Aftera new
> > > > > release,
> > > > > we run a"npm ds-version" which then detects changesin the
> > > > > markdown filesand createsa new release docs
> > > > > version.Thedocuments in the "docs"folder can be freely
> > > > > modified
> > > > > andadded.That's a good place wherewe can add new tutorials.
> > > > > The
> > > > > "pe"foldercontains thedocumentation of pipeline elements.
> > > > > These
> > > > > arecurrentlygeneratedfrom the StreamPipes source code using
> > > > > thestreampipes-maven-plugin (as some rewriting is needed,
> > > > > e.g.,
> > > > > to fixthe pathsofimages). This process is far from perfect
> > > > > and
> > > > > needssome manualwork.It would be nice to have some Github
> > > > > actionsscriptwhichautomatically updates the Documentation,
> > > > > but
> > > > > that's abitmoreadvanced task I'd say 😉If you want to make
> > > > > changes tothestructure, this can be done in
> > > > > the"sidebars.json"
> > > > > file [3],whichis also auto-versioned by Docusaurus.[1]
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/website
> > > > >
> > > > > [2]
> > > > > https://github.com/apache/incubator-streampipes-website/tree/dev/documentation/docs
> > > > >
> > > > > [3]
> > > > > https://github.com/apache/incubator-streampipes-website/blob/dev/documentation/website/sidebars.json
> > > > >
> > > > >
> > > > > -----Original Message-----From: Florian Micklich <
> > > > > micklich@apache.org
> > > > >
> > > > > > Sent: Wednesday, August 24, 2022 10:13 AMTo:
> > > > >
> > > > > dev@streampipes.apache.org
> > > > >
> > > > > Subject: Re: [DISCUSS] Documentation - Description
> > > > > forAdaptersandProcessing ElementsHello,I would also prefer a
> > > > > shortIn-Appdocumentation and largerdocumentation "outside"
> > > > > .Personally, Iprefer to read pdf documentsso I can take
> > > > > additionalnotes. ButHTML is also good for a quickonline
> > > > > search.We must keepin mindwhat kind of documentation we
> > > > > wantto
> > > > > create and should notmixthem. Also, cross-references between
> > > > > theindividual"chapters"should be avoided. This makes it
> > > > > easier
> > > > > ifsomethingshouldchange. 1. Quick starter
> > > > > documentation. 2.SetupDocumentation 3.Technical
> > > > > Documentation 4. Usermanualdocumentation with bestpractiseI
> > > > > think we should hit theroad andtake a look along the way :)
> > > > > Iknow the in appdocumentation is inmarkdown so far?I
> > > > > wouldrecommend to write thedocumentation inasciidoc [1] [2]
> > > > > This couldbe set up withasciidoctorj [3]. Thismakes it easy
> > > > > to
> > > > > includeeverything in Gitand also keep thedocumentation
> > > > > alongside thesource code. So ifsomething isrewritten, the
> > > > > developer can alsochange thedocumentation. Makesnew
> > > > > screenshots
> > > > > and deletes oldones.3] Thecreation of a PDF orHTML file can
> > > > > be
> > > > > done withGithubAction. With every commitor with every
> > > > > release. Theresults can bepushed to the StreamPipeswebsite, I
> > > > > guess?What doyouthink?Florian[1]
> > > > > https://docs.asciidoctor.org/asciidoc/latest/[2]
> > > > >
> > > > > German Source
> > > > > https://blog.ordix.de/docs-as-code-dokumentation-mit-asciidoctor[3]
> > > > >
> > > > > htt
> > > > > ps://github.com/asciidoctor/asciidoctorjAmMittwoch,dem24.08.2
> > > > > 02
> > > > > 2 um 07:07 +0000 schrieb Dominik Riemer:
> > > > > > Hi,yes, we can do both: In-app documentation
> > > > > > whichonlycoverstheconfiguration and a short description of
> > > > > > theadapterandpipelineelement and additional step-by-
> > > > > > stepguides/tutorialswithnicescreenshots that are available
> > > > > > in
> > > > > > theonlinedocumentation. Wecanhave such step-by-step guides
> > > > > > for themostfrequently used adapters.@Florian, do you want
> > > > > > to
> > > > > > work onsuch aguide on thewebsite? I knowyou had some really
> > > > > > gooddocumentationideas somewhile ago 😉CheersDominik-----
> > > > > > OriginalMessage-----From: Philipp Zehnder <
> > > > > > philipp.zehnder@bytefabrik.ai
> > > > > > > Sent:Sunday, August
> > > > > > 21,20221:07PMTo:
> > > > > > dev@streampipes.apache.org
> > > > > >
> > > > > > Subject: [DISCUSS] Documentation -
> > > > > > DescriptionforAdaptersandProcessing Elements Hi all, here
> > > > > > is
> > > > > > a new threadtodiscusstheadapter documentation.Here is a
> > > > > > link
> > > > > > to thepreviousdiscussionin therelease thread [1].@Dominik I
> > > > > > know thattheregistration isdifferent, but since alladapters
> > > > > > already havethethree files(documentation.md, icon.png,
> > > > > > andstring.en),shouldn’tit bepossible to also display the
> > > > > > descriptionin theUI.How are wedoingcurrently for the
> > > > > > icons?Because they areloaded dynamicallyaswell. I think the
> > > > > > maindifference is thatthe icons areloadeddirectly from the
> > > > > > workercontainer and notfrom thebackend.Couldn’t we do it
> > > > > > similarly forthedocumentation file?(Pleasecorrect me if I
> > > > > > am
> > > > > > wrong)Regardingthe description on theWebsite.I think it
> > > > > > would
> > > > > > begreat if we couldgenerate this basedon themarkdown files
> > > > > > foreach adapter.The documentation.md filesmostlydescribe
> > > > > > theconfigurationparameters of adapters andprocessors.
> > > > > > Ithink
> > > > > > itwould be great tohave additional exampleshow to use
> > > > > > them.(e.g.describe how a CSVfile is uploaded or how aPLC
> > > > > > isconnected).Should we include suchexamples
> > > > > > intothedocumentation.md file orshould there be
> > > > > > anotherlocation forthis?Any thoughts onthat?@Florian you
> > > > > > wrote that you would liketo workon thedocumentation.If you
> > > > > > need anything to start with,please
> > > > > > letusknow.Cheers,Philipp[1]
> > > > > > https://lists.apache.org/thread/29w586db1btqo0ps1rgo3fscvt3tvrzg
> > > > > >