You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by Martin Trat <Tr...@fzi.de> on 2022/01/14 13:40:59 UTC
AW: No pipeline-elements available
Hi Philipp,
Thanks for your hint – building beforehand did the job for me…
I can now actually see and probably use a bunch of processor and sink pipeline elements (in pipeline editor and via install pipeline elements).
However, if I try to use adapters (I would like to use messages from a MQTT broker as data source) I cannot seem to get any (when trying both cli (env set to full) and compose-based (docker-compose.full.yaml) installation). For this, I click on connect and neither Data Marketplace nor My Adapters lets me set up an MQTT (or any other) data source.
What could be the problem here?
Thank you and best
Martin
Von: Philipp Zehnder <ze...@apache.org>
Gesendet: Montag, 27. Dezember 2021 10:11
An: users@streampipes.apache.org
Betreff: Re: No pipeline-elements available
Hi Martin,
thanks for writing us and sorry for the confusion about the development setup.
Recently, we merged our repositories into one to ease the development [1] and did not update all the documentation yet.
However, we plan to do this before the next release.
I updated the description how to setup a development environment for new processing elements [2].
I hope this helps for the setup and that the error is gone when you use the CLI to start up StreamPipes.
Please let me know when you have any problems or when something is not clear. You can also contact us if you find documentation that is no longer up to date so that we can update it.
Philipp
[1] https://github.com/apache/incubator-streampipes
[2] https://cwiki.apache.org/confluence/display/STREAMPIPES/Extensions
Von: Martin Trat < <ma...@fzi.de> Trat@fzi.de>
Antworten an: < <ma...@streampipes.apache.org> users@streampipes.apache.org>
Datum: Freitag, 24. Dezember 2021 um 14:39
An: "users@streampipes.apache.org <ma...@streampipes.apache.org> " <users@streampipes.apache.org <ma...@streampipes.apache.org> >
Betreff: No pipeline-elements available
Dear StreamPipes Community,
Installing StreamPipes 0.68.0 via release download from GitHub worked for me.
However, in order to develop custom processors, I would like to use a recent snapshot state. I cloned c6527b9 from the dev branch of incubating-streampipes repo, stopped all containers, pruned images, volumes and containers and attempted a compose-based installation (using docker-compose.full.yml). I followed the rather sparse documentation from the website (documentation version selected “master” under “Pre-release versions”) and the repository’s readme under installer/compose.
Browsing my then running UI instance, I cannot download/install/see any pipeline elements (0 available).
Do I need to execute any installer scripts, set environment variables or do anything the like? Do I need another repo for pipeline elements?
Also, I consulted with a partnering company, contributing to StreamPipes. They also use a snapshot installation and can still draw pipeline elements (I also tried pulling the commit they are using, but to no avail). It therefore seems counter intuitive when proposing the following error to be relevant for missing pipeline elements. However, they confirm their pipeline-elements-all-flink container being continuously stuck in restarting state. The container shows the following log on repeat:
```
12:00:05.079 SP [main] INFO o.a.s.s.consul.ConsulProvider - Checking if consul is available...
12:00:05.094 SP [main] INFO o.a.s.s.consul.ConsulProvider - Successfully connected to Consul
Exception in thread "main" java.lang.NullPointerException: id
at java.util.Objects.requireNonNull(Objects.java:228)
at com.orbitz.consul.model.agent.ImmutableRegistration$Builder.id(ImmutableRegistration.java:847)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.createRegistrationBody(SpConsulServiceDiscovery.java:120)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.registerService(SpConsulServiceDiscovery.java:49)
at org.apache.streampipes.container.base.StreamPipesServiceBase.registerService(StreamPipesServiceBase.java:66)
at org.apache.streampipes.container.base.StreamPipesServiceBase.startStreamPipesService(StreamPipesServiceBase.java:42)
at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:67)
at org.apache.streampipes.container.standalone.init.StandaloneModelSubmitter.init(StandaloneModelSubmitter.java:57)
at org.apache.streampipes.pe.flink.AllFlinkPipelineElementsInit.main(AllFlinkPipelineElementsInit.java:105)
12:01:07,931 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/streampipes-processing-element-container.jar!/logback.xml]
12:01:07,969 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@f97bd761 <ma...@f97bd761> - URL [jar:file:/streampipes-processing-element-container.jar!/logback.xml] is not of type file
12:01:08,155 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:01:08,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:01:08,170 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.streampipes] to INFO
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.lightcouch] to ERROR
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka] to ERROR
12:01:08,281 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
12:01:08,282 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
12:01:08,282 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:01:08,286 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@75dd924 <ma...@75dd924> - Registering current configuration as safe fallback point
```
Some possibly relevant information:
* Ubuntu 20.04
* Software requirements from documentation fulfilled
* Latest v1 docker-compose v 1.29.2
* docker v 20.10.10
* Reasonably powerfully sized VM meeting hardware requirements
* Deviations from default
* Running UI at host port 81
Thank you very much for helping me with this!
Best
Martin
.........................................................
Martin Trat, M.Sc.
Wissenschaftlicher Mitarbeiter | Research Associate
Intelligent Systems and Production Engineering (ISPE)
Process & Data Management in Engineering (PDE)
FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14
76131 Karlsruhe, Germany
Tel.: +49 721 9654-509
trat@fzi.de <ma...@fzi.de> | www.fzi.de/mitarbeiter/trat <http://www.fzi.de/mitarbeiter/trat> | https://linkedin.com/in/martin-trat <https://linkedin.com/in/martin-trat-372599139>
www.fzi.de <http://www.fzi.de/> | www.twitter.com/FZI_official <http://www.twitter.com/FZI_official> | www.facebook.com/FZI.Official <http://www.facebook.com/FZI.Official> | www.youtube.com/FZIchannel <http://www.youtube.com/FZIchannel>
.........................................................
FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
.........................................................
Re: No pipeline-elements available
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Martin,
so if I understand you correctly the docker-compose.yml in the root directory of the project works, but the docker-compose.yml files in the installer and the CLI does not work, right?
I guess the problem is due to some configurations that are not removed when you change the environment. Because the only difference of the docker-compose files is that the one in the root directory builds the docker images on your host and the docker-compose files in the installer uses the images that are provided on docker hub.
Did you remove the volumes (e.g. docker volume prune) and networks (e.g. docker networks prune) before switching the environment? (Please be careful with those commands, when you have other services running on this host. They will remove all docker volumes).
You can also check in the logs what the IP address is that is used for the services:
e.g. 16:32:23.889 SP [main] INFO o.a.s.commons.networking.Networking - Using IP from provided environment variable SP_HOST: 192.168.1.25
Somehow the URL for the backend is set to null in the adapters service, this is why they are not able to register themselves.
Philipp
Von: Martin Trat <Tr...@fzi.de>
Antworten an: <us...@streampipes.apache.org>
Datum: Montag, 17. Januar 2022 um 15:28
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: AW: No pipeline-elements available
Hi Philipp,
I have an update on this.
When using docker-compose from the incubator-streampipes root folder, I manage to use said adapters!
When I attempted this via cli or docker-compose(.full.yml) from the installer sub folders, I could not get adapters running. In case you are interested in this, please see a permanently recurring snipped from the logs below.
Best
Martin
connect-adapters_1 | 13:18:22.720 SP [main] ERROR o.a.s.c.c.w.m.MasterRestClient - Could not register adapter at url - is a 'StreamPipes Core' service running?
connect-adapters_1 | org.apache.streampipes.commons.exceptions.SpRuntimeException: null
connect-adapters_1 | at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:107)
connect-adapters_1 | at org.apache.streampipes.client.api.AbstractClientApi.post(AbstractClientApi.java:43)
connect-adapters_1 | at org.apache.streampipes.client.api.AdminApi.registerAdapters(AdminApi.java:33)
connect-adapters_1 | at org.apache.streampipes.connect.container.worker.management.MasterRestClient.register(MasterRestClient.java:37)
connect-adapters_1 | at org.apache.streampipes.connect.container.worker.init.ConnectWorkerRegistrationService.registerWorker(ConnectWorkerRegistrationService.java:34)
connect-adapters_1 | at org.apache.streampipes.container.extensions.ExtensionsModelSubmitter.afterServiceRegistered(ExtensionsModelSubmitter.java:53)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:73)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:52)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:42)
connect-adapters_1 | at org.apache.streampipes.connect.ConnectAdapterInit.main(ConnectAdapterInit.java:42)
connect-adapters_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
connect-adapters_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
connect-adapters_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
connect-adapters_1 | at java.lang.reflect.Method.invoke(Method.java:498)
connect-adapters_1 | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
connect-adapters_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
connect-adapters_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
connect-adapters_1 | at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
connect-adapters_1 | 13:18:22.721 SP [main] INFO o.a.s.c.c.w.i.ConnectWorkerRegistrationService - Retrying in 5 seconds
connect-adapters-iiot_1 | 13:18:24.739 SP [main] ERROR o.a.s.c.c.w.m.MasterRestClient - Could not register adapter at url - is a 'StreamPipes Core' service running?
connect-adapters-iiot_1 | org.apache.streampipes.commons.exceptions.SpRuntimeException: null
connect-adapters-iiot_1 | at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:107)
connect-adapters-iiot_1 | at org.apache.streampipes.client.api.AbstractClientApi.post(AbstractClientApi.java:43)
connect-adapters-iiot_1 | at org.apache.streampipes.client.api.AdminApi.registerAdapters(AdminApi.java:33)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.container.worker.management.MasterRestClient.register(MasterRestClient.java:37)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.container.worker.init.ConnectWorkerRegistrationService.registerWorker(ConnectWorkerRegistrationService.java:34)
connect-adapters-iiot_1 | at org.apache.streampipes.container.extensions.ExtensionsModelSubmitter.afterServiceRegistered(ExtensionsModelSubmitter.java:53)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:73)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:52)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:42)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.iiot.ConnectAdapterIiotInit.main(ConnectAdapterIiotInit.java:42)
connect-adapters-iiot_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
connect-adapters-iiot_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
connect-adapters-iiot_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
connect-adapters-iiot_1 | at java.lang.reflect.Method.invoke(Method.java:498)
connect-adapters-iiot_1 | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
connect-adapters-iiot_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
connect-adapters-iiot_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
connect-adapters-iiot_1 | at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
connect-adapters-iiot_1 | 13:18:24.743 SP [main] INFO o.a.s.c.c.w.i.ConnectWorkerRegistrationService - Retrying in 5 seconds
Von: Martin Trat <Tr...@fzi.de>
Gesendet: Freitag, 14. Januar 2022 14:41
An: users@streampipes.apache.org
Betreff: AW: No pipeline-elements available
Hi Philipp,
Thanks for your hint – building beforehand did the job for me…
I can now actually see and probably use a bunch of processor and sink pipeline elements (in pipeline editor and via install pipeline elements).
However, if I try to use adapters (I would like to use messages from a MQTT broker as data source) I cannot seem to get any (when trying both cli (env set to full) and compose-based (docker-compose.full.yaml) installation). For this, I click on connect and neither Data Marketplace nor My Adapters lets me set up an MQTT (or any other) data source.
What could be the problem here?
Thank you and best
Martin
Von: Philipp Zehnder <ze...@apache.org>
Gesendet: Montag, 27. Dezember 2021 10:11
An: users@streampipes.apache.org
Betreff: Re: No pipeline-elements available
Hi Martin,
thanks for writing us and sorry for the confusion about the development setup.
Recently, we merged our repositories into one to ease the development [1] and did not update all the documentation yet.
However, we plan to do this before the next release.
I updated the description how to setup a development environment for new processing elements [2].
I hope this helps for the setup and that the error is gone when you use the CLI to start up StreamPipes.
Please let me know when you have any problems or when something is not clear. You can also contact us if you find documentation that is no longer up to date so that we can update it.
Philipp
[1] https://github.com/apache/incubator-streampipes
[2] https://cwiki.apache.org/confluence/display/STREAMPIPES/Extensions
Von: Martin Trat <Tr...@fzi.de>
Antworten an: <us...@streampipes.apache.org>
Datum: Freitag, 24. Dezember 2021 um 14:39
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: No pipeline-elements available
Dear StreamPipes Community,
Installing StreamPipes 0.68.0 via release download from GitHub worked for me.
However, in order to develop custom processors, I would like to use a recent snapshot state. I cloned c6527b9 from the dev branch of incubating-streampipes repo, stopped all containers, pruned images, volumes and containers and attempted a compose-based installation (using docker-compose.full.yml). I followed the rather sparse documentation from the website (documentation version selected “master” under “Pre-release versions”) and the repository’s readme under installer/compose.
Browsing my then running UI instance, I cannot download/install/see any pipeline elements (0 available).
Do I need to execute any installer scripts, set environment variables or do anything the like? Do I need another repo for pipeline elements?
Also, I consulted with a partnering company, contributing to StreamPipes. They also use a snapshot installation and can still draw pipeline elements (I also tried pulling the commit they are using, but to no avail). It therefore seems counter intuitive when proposing the following error to be relevant for missing pipeline elements. However, they confirm their pipeline-elements-all-flink container being continuously stuck in restarting state. The container shows the following log on repeat:
```
12:00:05.079 SP [main] INFO o.a.s.s.consul.ConsulProvider - Checking if consul is available...
12:00:05.094 SP [main] INFO o.a.s.s.consul.ConsulProvider - Successfully connected to Consul
Exception in thread "main" java.lang.NullPointerException: id
at java.util.Objects.requireNonNull(Objects.java:228)
at com.orbitz.consul.model.agent.ImmutableRegistration$Builder.id(ImmutableRegistration.java:847)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.createRegistrationBody(SpConsulServiceDiscovery.java:120)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.registerService(SpConsulServiceDiscovery.java:49)
at org.apache.streampipes.container.base.StreamPipesServiceBase.registerService(StreamPipesServiceBase.java:66)
at org.apache.streampipes.container.base.StreamPipesServiceBase.startStreamPipesService(StreamPipesServiceBase.java:42)
at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:67)
at org.apache.streampipes.container.standalone.init.StandaloneModelSubmitter.init(StandaloneModelSubmitter.java:57)
at org.apache.streampipes.pe.flink.AllFlinkPipelineElementsInit.main(AllFlinkPipelineElementsInit.java:105)
12:01:07,931 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/streampipes-processing-element-container.jar!/logback.xml]
12:01:07,969 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@f97bd761 - URL [jar:file:/streampipes-processing-element-container.jar!/logback.xml] is not of type file
12:01:08,155 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:01:08,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:01:08,170 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.streampipes] to INFO
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.lightcouch] to ERROR
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka] to ERROR
12:01:08,281 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
12:01:08,282 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
12:01:08,282 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:01:08,286 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@75dd924 - Registering current configuration as safe fallback point
```
Some possibly relevant information:
Ubuntu 20.04
Software requirements from documentation fulfilled
Latest v1 docker-compose v 1.29.2
docker v 20.10.10
Reasonably powerfully sized VM meeting hardware requirements
Deviations from default
Running UI at host port 81
Thank you very much for helping me with this!
Best
Martin
.........................................................
Martin Trat, M.Sc.
Wissenschaftlicher Mitarbeiter | Research Associate
Intelligent Systems and Production Engineering (ISPE)
Process & Data Management in Engineering (PDE)
FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14
76131 Karlsruhe, Germany
Tel.: +49 721 9654-509
trat@fzi.de | www.fzi.de/mitarbeiter/trat | https://linkedin.com/in/martin-trat
www.fzi.de | www.twitter.com/FZI_official | www.facebook.com/FZI.Official | www.youtube.com/FZIchannel
.........................................................
FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
.........................................................
AW: No pipeline-elements available
Posted by Martin Trat <Tr...@fzi.de>.
Hi Philipp,
I have an update on this.
When using docker-compose from the incubator-streampipes root folder, I manage to use said adapters!
When I attempted this via cli or docker-compose(.full.yml) from the installer sub folders, I could not get adapters running. In case you are interested in this, please see a permanently recurring snipped from the logs below.
Best
Martin
connect-adapters_1 | 13:18:22.720 SP [main] ERROR o.a.s.c.c.w.m.MasterRestClient - Could not register adapter at url - is a 'StreamPipes Core' service running?
connect-adapters_1 | org.apache.streampipes.commons.exceptions.SpRuntimeException: null
connect-adapters_1 | at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:107)
connect-adapters_1 | at org.apache.streampipes.client.api.AbstractClientApi.post(AbstractClientApi.java:43)
connect-adapters_1 | at org.apache.streampipes.client.api.AdminApi.registerAdapters(AdminApi.java:33)
connect-adapters_1 | at org.apache.streampipes.connect.container.worker.management.MasterRestClient.register(MasterRestClient.java:37)
connect-adapters_1 | at org.apache.streampipes.connect.container.worker.init.ConnectWorkerRegistrationService.registerWorker(ConnectWorkerRegistrationService.java:34)
connect-adapters_1 | at org.apache.streampipes.container.extensions.ExtensionsModelSubmitter.afterServiceRegistered(ExtensionsModelSubmitter.java:53)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:73)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:52)
connect-adapters_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:42)
connect-adapters_1 | at org.apache.streampipes.connect.ConnectAdapterInit.main(ConnectAdapterInit.java:42)
connect-adapters_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
connect-adapters_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
connect-adapters_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
connect-adapters_1 | at java.lang.reflect.Method.invoke(Method.java:498)
connect-adapters_1 | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
connect-adapters_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
connect-adapters_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
connect-adapters_1 | at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
connect-adapters_1 | 13:18:22.721 SP [main] INFO o.a.s.c.c.w.i.ConnectWorkerRegistrationService - Retrying in 5 seconds
connect-adapters-iiot_1 | 13:18:24.739 SP [main] ERROR o.a.s.c.c.w.m.MasterRestClient - Could not register adapter at url - is a 'StreamPipes Core' service running?
connect-adapters-iiot_1 | org.apache.streampipes.commons.exceptions.SpRuntimeException: null
connect-adapters-iiot_1 | at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:107)
connect-adapters-iiot_1 | at org.apache.streampipes.client.api.AbstractClientApi.post(AbstractClientApi.java:43)
connect-adapters-iiot_1 | at org.apache.streampipes.client.api.AdminApi.registerAdapters(AdminApi.java:33)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.container.worker.management.MasterRestClient.register(MasterRestClient.java:37)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.container.worker.init.ConnectWorkerRegistrationService.registerWorker(ConnectWorkerRegistrationService.java:34)
connect-adapters-iiot_1 | at org.apache.streampipes.container.extensions.ExtensionsModelSubmitter.afterServiceRegistered(ExtensionsModelSubmitter.java:53)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:73)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:52)
connect-adapters-iiot_1 | at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:42)
connect-adapters-iiot_1 | at org.apache.streampipes.connect.iiot.ConnectAdapterIiotInit.main(ConnectAdapterIiotInit.java:42)
connect-adapters-iiot_1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
connect-adapters-iiot_1 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
connect-adapters-iiot_1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
connect-adapters-iiot_1 | at java.lang.reflect.Method.invoke(Method.java:498)
connect-adapters-iiot_1 | at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
connect-adapters-iiot_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
connect-adapters-iiot_1 | at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
connect-adapters-iiot_1 | at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
connect-adapters-iiot_1 | 13:18:24.743 SP [main] INFO o.a.s.c.c.w.i.ConnectWorkerRegistrationService - Retrying in 5 seconds
Von: Martin Trat <Tr...@fzi.de>
Gesendet: Freitag, 14. Januar 2022 14:41
An: users@streampipes.apache.org
Betreff: AW: No pipeline-elements available
Hi Philipp,
Thanks for your hint – building beforehand did the job for me…
I can now actually see and probably use a bunch of processor and sink pipeline elements (in pipeline editor and via install pipeline elements).
However, if I try to use adapters (I would like to use messages from a MQTT broker as data source) I cannot seem to get any (when trying both cli (env set to full) and compose-based (docker-compose.full.yaml) installation). For this, I click on connect and neither Data Marketplace nor My Adapters lets me set up an MQTT (or any other) data source.
What could be the problem here?
Thank you and best
Martin
Von: Philipp Zehnder <zehnder@apache.org <ma...@apache.org> >
Gesendet: Montag, 27. Dezember 2021 10:11
An: users@streampipes.apache.org <ma...@streampipes.apache.org>
Betreff: Re: No pipeline-elements available
Hi Martin,
thanks for writing us and sorry for the confusion about the development setup.
Recently, we merged our repositories into one to ease the development [1] and did not update all the documentation yet.
However, we plan to do this before the next release.
I updated the description how to setup a development environment for new processing elements [2].
I hope this helps for the setup and that the error is gone when you use the CLI to start up StreamPipes.
Please let me know when you have any problems or when something is not clear. You can also contact us if you find documentation that is no longer up to date so that we can update it.
Philipp
[1] https://github.com/apache/incubator-streampipes
[2] https://cwiki.apache.org/confluence/display/STREAMPIPES/Extensions
Von: Martin Trat < <ma...@fzi.de> Trat@fzi.de>
Antworten an: < <ma...@streampipes.apache.org> users@streampipes.apache.org>
Datum: Freitag, 24. Dezember 2021 um 14:39
An: "users@streampipes.apache.org <ma...@streampipes.apache.org> " <users@streampipes.apache.org <ma...@streampipes.apache.org> >
Betreff: No pipeline-elements available
Dear StreamPipes Community,
Installing StreamPipes 0.68.0 via release download from GitHub worked for me.
However, in order to develop custom processors, I would like to use a recent snapshot state. I cloned c6527b9 from the dev branch of incubating-streampipes repo, stopped all containers, pruned images, volumes and containers and attempted a compose-based installation (using docker-compose.full.yml). I followed the rather sparse documentation from the website (documentation version selected “master” under “Pre-release versions”) and the repository’s readme under installer/compose.
Browsing my then running UI instance, I cannot download/install/see any pipeline elements (0 available).
Do I need to execute any installer scripts, set environment variables or do anything the like? Do I need another repo for pipeline elements?
Also, I consulted with a partnering company, contributing to StreamPipes. They also use a snapshot installation and can still draw pipeline elements (I also tried pulling the commit they are using, but to no avail). It therefore seems counter intuitive when proposing the following error to be relevant for missing pipeline elements. However, they confirm their pipeline-elements-all-flink container being continuously stuck in restarting state. The container shows the following log on repeat:
```
12:00:05.079 SP [main] INFO o.a.s.s.consul.ConsulProvider - Checking if consul is available...
12:00:05.094 SP [main] INFO o.a.s.s.consul.ConsulProvider - Successfully connected to Consul
Exception in thread "main" java.lang.NullPointerException: id
at java.util.Objects.requireNonNull(Objects.java:228)
at com.orbitz.consul.model.agent.ImmutableRegistration$Builder.id(ImmutableRegistration.java:847)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.createRegistrationBody(SpConsulServiceDiscovery.java:120)
at org.apache.streampipes.svcdiscovery.consul.SpConsulServiceDiscovery.registerService(SpConsulServiceDiscovery.java:49)
at org.apache.streampipes.container.base.StreamPipesServiceBase.registerService(StreamPipesServiceBase.java:66)
at org.apache.streampipes.container.base.StreamPipesServiceBase.startStreamPipesService(StreamPipesServiceBase.java:42)
at org.apache.streampipes.service.extensions.base.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:67)
at org.apache.streampipes.container.standalone.init.StandaloneModelSubmitter.init(StandaloneModelSubmitter.java:57)
at org.apache.streampipes.pe.flink.AllFlinkPipelineElementsInit.main(AllFlinkPipelineElementsInit.java:105)
12:01:07,931 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
12:01:07,932 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/streampipes-processing-element-container.jar!/logback.xml]
12:01:07,969 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@f97bd761 <ma...@f97bd761> - URL [jar:file:/streampipes-processing-element-container.jar!/logback.xml] is not of type file
12:01:08,155 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:01:08,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:01:08,170 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
12:01:08,279 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.streampipes] to INFO
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.lightcouch] to ERROR
12:01:08,279 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka] to ERROR
12:01:08,281 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
12:01:08,282 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
12:01:08,282 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:01:08,286 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@75dd924 <ma...@75dd924> - Registering current configuration as safe fallback point
```
Some possibly relevant information:
* Ubuntu 20.04
* Software requirements from documentation fulfilled
* Latest v1 docker-compose v 1.29.2
* docker v 20.10.10
* Reasonably powerfully sized VM meeting hardware requirements
* Deviations from default
* Running UI at host port 81
Thank you very much for helping me with this!
Best
Martin
.........................................................
Martin Trat, M.Sc.
Wissenschaftlicher Mitarbeiter | Research Associate
Intelligent Systems and Production Engineering (ISPE)
Process & Data Management in Engineering (PDE)
FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14
76131 Karlsruhe, Germany
Tel.: +49 721 9654-509
trat@fzi.de <ma...@fzi.de> | www.fzi.de/mitarbeiter/trat <http://www.fzi.de/mitarbeiter/trat> | https://linkedin.com/in/martin-trat <https://linkedin.com/in/martin-trat-372599139>
www.fzi.de <http://www.fzi.de/> | www.twitter.com/FZI_official <http://www.twitter.com/FZI_official> | www.facebook.com/FZI.Official <http://www.facebook.com/FZI.Official> | www.youtube.com/FZIchannel <http://www.youtube.com/FZIchannel>
.........................................................
FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
.........................................................