You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by Robert Edward Herding <ro...@yandex.com> on 2021/09/01 07:25:52 UTC
Re: Creating a Processor
Hello Patrick,
Yes, it is working, you solved the problem :) !!!
Thank you very much for the last days. I have learned a lot.
I have to confess that this setup now doesn't look difficult at all (now),
actually it is quite simple.
But working with docker and other system locally together can be a bit
annoying.
Thanks again and lets start build components.
I wish all the best for StreamPipes team.
Best regards,
Robert Edward Hearding
31.08.2021, 10:19, "Patrick Wiener" <wi...@apache.org>:
> From your backend logs it seems that it tries to fetch element descriptions
from a service referenced by the name „geofence-tutorial“. Assuming that this
is the name of your processor which is running in your IDE on your machine
this won’t work as „geofence-tutorial“ cannot be resolved from within Docker.
>
> If you remove and delete all volumes also for Consul, and restart everythign
this shouldn’t be a problem anymore, as then the processors registers with the
IP address of your machine.
>
> It can be that you initially registered it via „geofence-tutorial“. See the
Config.java class
>
> config.register(ConfigKeys.HOST, "geofence-tutorial", "Data processor
host“);
>
> It defaults to „geofence-tutorial“ if you not provide an env variable of
ConfigKeys.HOST (SP_HOST). This is what you probably did the second time when
starting, i.e., SP_HOST=192.168.x.x
>
> However, upon initial startup the processor falsely registered under the
address „geofence-tutorial“ for the corresponding service id which we do not
overwrite on later restarts.
>
> To cut the long story short, please run from clean slate and remove all
containers and volumes, set SP_HOST to your IP once you start of the service
from your IDE and it should work. There shouldn’t
>
> be any references to a service under the address „geofence-tutorial“
anymore.
>
> A short note: the whole service definition and discovery is completely
reconstructed for the next release which will hopefully be much more easy to
handle and totally get rid of these potential issues.
>
> Hope this helps :)
>
> Best
>
> Patrick
>
>> Am 31.08.2021 um 15:51 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
>>
>> Hello Patrick,
>>
>> I have attached the log file from Consul and Backend services.
>>
>> Thanks
>>
>> Robert
>>
>> 31.08.2021, 08:54, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)>:
>>
>>> Hi Robert,
>>>
>>> could you please go and check the backend service logs? Maybe it provides
more details on why the installation fails.
>>>
>>> Patrick
>>>
>>>> Am 31.08.2021 um 14:49 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
>>>>
>>>> Hi Patrick and Dominik,
>>>>
>>>> Thanks for all the patience. I really appreciate the help.
>>>>
>>>> Yes I am always deleting the volume to have a fresh start. But I still
get the red dot.
>>>>
>>>> Then I tried what Dominik said and now I have a beautiful DataProcessor
Geofencing component :) .
>>>>
>>>> But when I tried to install it:
>>>>
>>>> #### Installing 1 of 1 (Geofencing)...error
>>>>
>>>> **More Details:**
>>>>
>>>> _geofence-tutorial: System error_
>>>>
>>>> Also again I have got this warning (not sure if this is the problem, but
it seems related as it only shows when its time to import the component)
>>>>
>>>> PM org.glassfish.jersey.server.wadl.WadlFeature configure
>>>>
>>>> WARNING: JAXBContext implementation could not be found. WADL feature is
disabled.
>>>>
>>>> I will fix this warning and review the program for some mistype (just to
be sure it is perfectly ok) and I hope coming back with good news.
>>>>
>>>> If you guys have any more ideas please let me know.
>>>>
>>>> Best regard,
>>>>
>>>> Robert Edward
>>>>
>>>> 31.08.2021, 07:13, "Dominik Riemer"
<[riemer@apache.org](mailto:riemer@apache.org)>:
>>>>
>>>>> Hi Robert,
>
> on some development setups the locally started extensions service is not
detected as a healthy service by Consul and thus not added to the list of
pipeline elements in the UI.
> You can also try to click on "Manage Endpoints" and add the URL of the
extensions service manually (just add the URL such as
[http://192.168.1.12:8092](http://192.168.1.12:8092/)). Maybe that's also
worth a try ;-)
>
> Dominik
>
> On 2021/08/31 10:25:06, Patrick Wiener
<[wiener@apache.org](mailto:wiener@apache.org)> wrote:
>>>>>
>>>>>> Hi Robert,
>
> did you also reset everything to a clean slate using the CLI? There might
be some misconfigurations due to your initial settings using „localhost“ that
might not get overwritten.
> Hopefully, if you perform the following steps we get you sorted:
>
> Remove StreamPipes and perform cleanup (don’t forget the -v flag as this is
required to delete the docker volumes): ./streampipes down -v
> Restart StreamPipes: ./streampipes up -d
> Finish setup process in the SP UI
> Start processor from IDE with your 192.168.x.x IP
> Check the SP UI at the „Configuration“ > „Pipeline Element Configuration“
tab and validate the green dot for your new processor service
> Go to „Install Pipeline Elements“ and check if you can install the new
processor
>
> Best
> Patrick
>
> > Am 31.08.2021 um 11:18 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
> >
> > Hello Patrick,
> >
> > Thanks once more for your answer, very interesting.
> > I have set my IP as you said.
> > Now I can access the RDF description, only if I set the IP address in the
URL without namespaces, like
"<http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing>"
> > But still doesn't import the processor
> > Funny thing I can access the RDF page also with the docker IP
> >
> > Reading the logs I also found an warning, only when accessing the RDF
page:
> >
> > AM org.glassfish.jersey.server.wadl.WadlFeature configure
> > WARNING: JAXBContext implementation could not be found. WADL feature is
disabled.
> >
> > As you said "manage endpoints" is not connecting to Consul(inside
docker), while RDF description page is(outside docker).
> > One question; It is possible to create an external Consul service(in the
cloud) and import from there? That would be the easier/faster way ?
> >
> > Best regards,
> > Robert
> >
> >
> >
> > 30.08.2021, 03:27, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)>:
> > Hi Robert,
> >
> > good to hear that we got it to register.
> >
> > The second problem seems to be due to the fact that the service health
check on the processor is failing given that you set the url to localhost:8092
for SP_HOST.
> > Thats also why you see the red dot in the overview.
> >
> > The reason for this basically is due to the hybrid dev environment with
parts of your setup running in the IDE (the processor) and the other parts
inside Docker (here Consul for
> > service discovery and service health check). While connections from your
Docker Host (your Debian machine) work on Consul over localhost:8500 (outside
-> inside Docker) the
> > issue occurs when services like Consul try to communicate to other
services running outside Docker, here on your Debian host, especially via
„localhost“. The reason for this is
> > that Docker containers run isolated in their own namespace and are not
aware of each other. While docker to docker connections work due to some
convenient help from Docker
> > using service DNS names (for containers within the same network), it
fails for development purposes (with some exceptions). Hence, when using
„localhost“ Consul thinks the service
> > is also running within the same container as Consul is, which is false,
as it is running on your machine in your IDE without Docker.
> >
> > So given your setup on a linux machine, what I would recommend for
development is to not use „localhost“ but the actual IP of your linux machine
(see ifconfig). Docker containers
> > can communicate to the NIC's (e.g. eth0) IP address. However, this only
works on linux as Docker runs natively. Yet it is worthy to note that if you
regularly change your network environments
> > which may lead to different IP addresses assigned to this NIC, the
service health check would start failing again, as the service was originally
registered with another IP.
> > We currently work on improving the whole service discovery/service
management part to hopefully resolve all these issues within the current dev
branch which is part of the next release
> > (0.69.0).
> >
> > As of now, see if it helps using the machine’s IP address (e.g.,
192.168.x.x) instead of localhost.
> >
> > Best
> > Patrick
> >
> > Am 29.08.2021 um 17:26 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)
<[mailto:robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>>:
> >
> > Hello Patrick,
> >
> > Thank you very much, your explanation helped a lot.
> > I was running the compose docker file, not the cli enviroment.
> > So I created a pipeline element template with cli, and it worked
perfectly, pretty straightforward.
> > Now I have a registered service at Consul.
> >
> > After that I tried to import the processor in Install Pipeline Elements,
with the url given at <http://localhost:8092/> <<http://localhost:8092/>> (I
had to change the port).
> > The Install Pipeline Elements shows no elements (it keeps searching) but
I can see "geofence tutorial" in Pipeline Element Configuration with a red
dot, for sure it is not working but it can see the container.
> > Can you please give me a north on this?
> >
> > Yours sincerely,
> >
> > Robert
> >
> >
> >
> >
> >
> >
> > 28.08.2021, 10:37, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)
<[mailto:wiener@apache.org](mailto:wiener@apache.org)>>:
> > Hi Robert,
> >
> > first of all I’d like to welcome you to the list.
> >
> > I assume you already started the developer environment such that Consul
Docker container is running?
> > You can use the StreamPipes CLI to easily configure a development
environment. You can find relevant parts from here [1].
> >
> > A little bit of background: we leverage Consul for service discovery such
that all extending services, e.g. processors etc
> > first register at Consul from where the StreamPipes core picks up service
endpoints of respective extensions.
> > Hence, currently also for development purposes it is necessary to set of
a minimal development environment using the CLI and
> > a corresponding environment template.
> >
> > Can you also provide an excerpt of „docker ps -a“?
> >
> > Cheers
> > Patrick
> >
> > [1] <https://streampipes.apache.org/docs/docs/extend-setup.html>
<<https://streampipes.apache.org/docs/docs/extend-setup.html>>
> >
> > Am 27.08.2021 um 23:19 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)
<[mailto:robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>>:
> >
> > Hello folks,
> >
> > I am a developer located in South Africa.
> > I have been testing StreamPipes for a few weeks now and I have decided to
build, a processor entity for testing and developing for a future industrial
use.
> > Said that I start following the tutorial
<https://streampipes.apache.org/docs/docs/extend-tutorial-data-
processors.html> <<https://streampipes.apache.org/docs/docs/extend-tutorial-
data-processors.html>>.
> > The project is building perfectly, but when it gets to the point of
registering items in Consul it just loops with the message.
> >
> > 22:41:53.553 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 22:41:53.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> > 22:41:54.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 22:41:54.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> > 20:41:55.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 20:41:55.556 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> >
> > I am really not used to work with Consul, so have been really hard to
understand why it can't find it.
> > I tried to change the SP_HOST in the run configuration.
> >
> > localhost
> > 0.0.0.0
> > 127.0.0.1
> > 192.168.1.1
> > 192.168.1.12
> > host.docker.internal
> >
> > Below the docker IP and docker network.
> >
> > udp 0 0 192.168.1.12:bootpc host.docker.inte:bootps ESTABLISHED
> > unix 3 [ ] STREAM CONNECTED 701942
/run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 692953
/run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 703571
/run/snap.docker/containerd/containerd.sock
> >
> >
> > NETWORK ID NAME DRIVER SCOPE
> > 305ea4177bdb bridge bridge local
> > 8c6df02eeef9 host host local
> > ead9e6d74aa1 none null local
> >
> > As you can see I am running linux Debian.
> > Any help would be greatly appreciated.
> >
> > Yours sincerely,
> >
> > Robert
> >
>
>
>>
>> <backend.log><consul.log>
Re: Creating a Processor
Posted by Robert Edward Herding <ro...@yandex.com>.
Hello Patrick,
Yes, I will send you my notes from all process, from installing streampipes,
create an object and uploading to kubernets.
For sure is other view of the process, a user view.
But I have to say that is pretty friendly once you got the hand of it.
I have repeated the process now a few times without flaws.
One of my clients needs control around 180 CNC machines.
Most of them are pretty new, but others(the big ones) has an adapted
communication system.
So, some will comunicate via mqtt broker and others via RF, LOra or cable.
My idea is to create multiple adapters, one processor and one sink.
And this setup seems to be perfect for streamPipes.
Thanks again
01.09.2021, 03:46, "Patrick Wiener" <wi...@apache.org>:
> Awesome :)
>
> If you have any idea what could be further improved to help the onboarding
process just reach out to us.
>
> Out of curiosity, did you just wanted to play around with SP or did you have
a specific use case in mind?
>
> Best
>
> Patrick
>
>> Am 01.09.2021 um 09:25 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
>>
>> Hello Patrick,
>>
>> Yes, it is working, you solved the problem :) !!!
>>
>> Thank you very much for the last days. I have learned a lot.
>>
>> I have to confess that this setup now doesn't look difficult at all (now),
actually it is quite simple.
>>
>> But working with docker and other system locally together can be a bit
annoying.
>>
>> Thanks again and lets start build components.
>>
>> I wish all the best for StreamPipes team.
>>
>> Best regards,
>>
>> Robert Edward Hearding
>>
>> 31.08.2021, 10:19, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)>:
>>
>>> From your backend logs it seems that it tries to fetch element
descriptions from a service referenced by the name „geofence-tutorial“.
Assuming that this is the name of your processor which is running in your IDE
on your machine this won’t work as „geofence-tutorial“ cannot be resolved from
within Docker.
>>>
>>> If you remove and delete all volumes also for Consul, and restart
everythign this shouldn’t be a problem anymore, as then the processors
registers with the IP address of your machine.
>>>
>>> It can be that you initially registered it via „geofence-tutorial“. See
the Config.java class
>>>
>>> config.register(ConfigKeys.HOST, "geofence-tutorial", "Data processor
host“);
>>>
>>> It defaults to „geofence-tutorial“ if you not provide an env variable of
ConfigKeys.HOST (SP_HOST). This is what you probably did the second time when
starting, i.e., SP_HOST=192.168.x.x
>>>
>>> However, upon initial startup the processor falsely registered under the
address „geofence-tutorial“ for the corresponding service id which we do not
overwrite on later restarts.
>>>
>>> To cut the long story short, please run from clean slate and remove all
containers and volumes, set SP_HOST to your IP once you start of the service
from your IDE and it should work. There shouldn’t
>>>
>>> be any references to a service under the address „geofence-tutorial“
anymore.
>>>
>>> A short note: the whole service definition and discovery is completely
reconstructed for the next release which will hopefully be much more easy to
handle and totally get rid of these potential issues.
>>>
>>> Hope this helps :)
>>>
>>> Best
>>>
>>> Patrick
>>>
>>>> Am 31.08.2021 um 15:51 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
>>>>
>>>> Hello Patrick,
>>>>
>>>> I have attached the log file from Consul and Backend services.
>>>>
>>>> Thanks
>>>>
>>>> Robert
>>>>
>>>> 31.08.2021, 08:54, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)>:
>>>>
>>>>> Hi Robert,
>>>>>
>>>>> could you please go and check the backend service logs? Maybe it
provides more details on why the installation fails.
>>>>>
>>>>> Patrick
>>>>>
>>>>>> Am 31.08.2021 um 14:49 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
>>>>>>
>>>>>> Hi Patrick and Dominik,
>>>>>>
>>>>>> Thanks for all the patience. I really appreciate the help.
>>>>>>
>>>>>> Yes I am always deleting the volume to have a fresh start. But I still
get the red dot.
>>>>>>
>>>>>> Then I tried what Dominik said and now I have a beautiful DataProcessor
Geofencing component :) .
>>>>>>
>>>>>> But when I tried to install it:
>>>>>>
>>>>>> #### Installing 1 of 1 (Geofencing)...error
>>>>>>
>>>>>> **More Details:**
>>>>>>
>>>>>> _geofence-tutorial: System error_
>>>>>>
>>>>>> Also again I have got this warning (not sure if this is the problem,
but it seems related as it only shows when its time to import the component)
>>>>>>
>>>>>> PM org.glassfish.jersey.server.wadl.WadlFeature configure
>>>>>>
>>>>>> WARNING: JAXBContext implementation could not be found. WADL feature is
disabled.
>>>>>>
>>>>>> I will fix this warning and review the program for some mistype (just
to be sure it is perfectly ok) and I hope coming back with good news.
>>>>>>
>>>>>> If you guys have any more ideas please let me know.
>>>>>>
>>>>>> Best regard,
>>>>>>
>>>>>> Robert Edward
>>>>>>
>>>>>> 31.08.2021, 07:13, "Dominik Riemer"
<[riemer@apache.org](mailto:riemer@apache.org)>:
>>>>>>
>>>>>>> Hi Robert,
>
> on some development setups the locally started extensions service is not
detected as a healthy service by Consul and thus not added to the list of
pipeline elements in the UI.
> You can also try to click on "Manage Endpoints" and add the URL of the
extensions service manually (just add the URL such as
[http://192.168.1.12:8092](http://192.168.1.12:8092/)). Maybe that's also
worth a try ;-)
>
> Dominik
>
> On 2021/08/31 10:25:06, Patrick Wiener
<[wiener@apache.org](mailto:wiener@apache.org)> wrote:
>>>>>>>
>>>>>>>> Hi Robert,
>
> did you also reset everything to a clean slate using the CLI? There might
be some misconfigurations due to your initial settings using „localhost“ that
might not get overwritten.
> Hopefully, if you perform the following steps we get you sorted:
>
> Remove StreamPipes and perform cleanup (don’t forget the -v flag as this is
required to delete the docker volumes): ./streampipes down -v
> Restart StreamPipes: ./streampipes up -d
> Finish setup process in the SP UI
> Start processor from IDE with your 192.168.x.x IP
> Check the SP UI at the „Configuration“ > „Pipeline Element Configuration“
tab and validate the green dot for your new processor service
> Go to „Install Pipeline Elements“ and check if you can install the new
processor
>
> Best
> Patrick
>
> > Am 31.08.2021 um 11:18 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>:
> >
> > Hello Patrick,
> >
> > Thanks once more for your answer, very interesting.
> > I have set my IP as you said.
> > Now I can access the RDF description, only if I set the IP address in the
URL without namespaces, like
"<http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing>"
> > But still doesn't import the processor
> > Funny thing I can access the RDF page also with the docker IP
> >
> > Reading the logs I also found an warning, only when accessing the RDF
page:
> >
> > AM org.glassfish.jersey.server.wadl.WadlFeature configure
> > WARNING: JAXBContext implementation could not be found. WADL feature is
disabled.
> >
> > As you said "manage endpoints" is not connecting to Consul(inside
docker), while RDF description page is(outside docker).
> > One question; It is possible to create an external Consul service(in the
cloud) and import from there? That would be the easier/faster way ?
> >
> > Best regards,
> > Robert
> >
> >
> >
> > 30.08.2021, 03:27, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)>:
> > Hi Robert,
> >
> > good to hear that we got it to register.
> >
> > The second problem seems to be due to the fact that the service health
check on the processor is failing given that you set the url to localhost:8092
for SP_HOST.
> > Thats also why you see the red dot in the overview.
> >
> > The reason for this basically is due to the hybrid dev environment with
parts of your setup running in the IDE (the processor) and the other parts
inside Docker (here Consul for
> > service discovery and service health check). While connections from your
Docker Host (your Debian machine) work on Consul over localhost:8500 (outside
-> inside Docker) the
> > issue occurs when services like Consul try to communicate to other
services running outside Docker, here on your Debian host, especially via
„localhost“. The reason for this is
> > that Docker containers run isolated in their own namespace and are not
aware of each other. While docker to docker connections work due to some
convenient help from Docker
> > using service DNS names (for containers within the same network), it
fails for development purposes (with some exceptions). Hence, when using
„localhost“ Consul thinks the service
> > is also running within the same container as Consul is, which is false,
as it is running on your machine in your IDE without Docker.
> >
> > So given your setup on a linux machine, what I would recommend for
development is to not use „localhost“ but the actual IP of your linux machine
(see ifconfig). Docker containers
> > can communicate to the NIC's (e.g. eth0) IP address. However, this only
works on linux as Docker runs natively. Yet it is worthy to note that if you
regularly change your network environments
> > which may lead to different IP addresses assigned to this NIC, the
service health check would start failing again, as the service was originally
registered with another IP.
> > We currently work on improving the whole service discovery/service
management part to hopefully resolve all these issues within the current dev
branch which is part of the next release
> > (0.69.0).
> >
> > As of now, see if it helps using the machine’s IP address (e.g.,
192.168.x.x) instead of localhost.
> >
> > Best
> > Patrick
> >
> > Am 29.08.2021 um 17:26 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)
<[mailto:robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>>:
> >
> > Hello Patrick,
> >
> > Thank you very much, your explanation helped a lot.
> > I was running the compose docker file, not the cli enviroment.
> > So I created a pipeline element template with cli, and it worked
perfectly, pretty straightforward.
> > Now I have a registered service at Consul.
> >
> > After that I tried to import the processor in Install Pipeline Elements,
with the url given at <http://localhost:8092/> <<http://localhost:8092/>> (I
had to change the port).
> > The Install Pipeline Elements shows no elements (it keeps searching) but
I can see "geofence tutorial" in Pipeline Element Configuration with a red
dot, for sure it is not working but it can see the container.
> > Can you please give me a north on this?
> >
> > Yours sincerely,
> >
> > Robert
> >
> >
> >
> >
> >
> >
> > 28.08.2021, 10:37, "Patrick Wiener"
<[wiener@apache.org](mailto:wiener@apache.org)
<[mailto:wiener@apache.org](mailto:wiener@apache.org)>>:
> > Hi Robert,
> >
> > first of all I’d like to welcome you to the list.
> >
> > I assume you already started the developer environment such that Consul
Docker container is running?
> > You can use the StreamPipes CLI to easily configure a development
environment. You can find relevant parts from here [1].
> >
> > A little bit of background: we leverage Consul for service discovery such
that all extending services, e.g. processors etc
> > first register at Consul from where the StreamPipes core picks up service
endpoints of respective extensions.
> > Hence, currently also for development purposes it is necessary to set of
a minimal development environment using the CLI and
> > a corresponding environment template.
> >
> > Can you also provide an excerpt of „docker ps -a“?
> >
> > Cheers
> > Patrick
> >
> > [1] <https://streampipes.apache.org/docs/docs/extend-setup.html>
<<https://streampipes.apache.org/docs/docs/extend-setup.html>>
> >
> > Am 27.08.2021 um 23:19 schrieb Robert Edward Herding
<[robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)
<[mailto:robertedward.herding@yandex.com](mailto:robertedward.herding@yandex.com)>>:
> >
> > Hello folks,
> >
> > I am a developer located in South Africa.
> > I have been testing StreamPipes for a few weeks now and I have decided to
build, a processor entity for testing and developing for a future industrial
use.
> > Said that I start following the tutorial
<https://streampipes.apache.org/docs/docs/extend-tutorial-data-
processors.html> <<https://streampipes.apache.org/docs/docs/extend-tutorial-
data-processors.html>>.
> > The project is building perfectly, but when it gets to the point of
registering items in Consul it just loops with the message.
> >
> > 22:41:53.553 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 22:41:53.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> > 22:41:54.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 22:41:54.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> > 20:41:55.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying
to connect to Consul to register config items
> > 20:41:55.556 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying
in 1 second
> >
> > I am really not used to work with Consul, so have been really hard to
understand why it can't find it.
> > I tried to change the SP_HOST in the run configuration.
> >
> > localhost
> > 0.0.0.0
> > 127.0.0.1
> > 192.168.1.1
> > 192.168.1.12
> > host.docker.internal
> >
> > Below the docker IP and docker network.
> >
> > udp 0 0 192.168.1.12:bootpc host.docker.inte:bootps ESTABLISHED
> > unix 3 [ ] STREAM CONNECTED 701942
/run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 692953
/run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 703571
/run/snap.docker/containerd/containerd.sock
> >
> >
> > NETWORK ID NAME DRIVER SCOPE
> > 305ea4177bdb bridge bridge local
> > 8c6df02eeef9 host host local
> > ead9e6d74aa1 none null local
> >
> > As you can see I am running linux Debian.
> > Any help would be greatly appreciated.
> >
> > Yours sincerely,
> >
> > Robert
> >
>
>
>>>>
>>>> <backend.log><consul.log>
Re: Creating a Processor
Posted by Patrick Wiener <wi...@apache.org>.
Awesome :)
If you have any idea what could be further improved to help the onboarding process just reach out to us.
Out of curiosity, did you just wanted to play around with SP or did you have a specific use case in mind?
Best
Patrick
> Am 01.09.2021 um 09:25 schrieb Robert Edward Herding <ro...@yandex.com>:
>
> Hello Patrick,
>
> Yes, it is working, you solved the problem :) !!!
> Thank you very much for the last days. I have learned a lot.
>
> I have to confess that this setup now doesn't look difficult at all (now), actually it is quite simple.
> But working with docker and other system locally together can be a bit annoying.
>
> Thanks again and lets start build components.
>
> I wish all the best for StreamPipes team.
>
> Best regards,
> Robert Edward Hearding
>
>
>
>
> 31.08.2021, 10:19, "Patrick Wiener" <wi...@apache.org>:
> From your backend logs it seems that it tries to fetch element descriptions from a service referenced by the name „geofence-tutorial“. Assuming that this is the name of your processor which is running in your IDE on your machine this won’t work as „geofence-tutorial“ cannot be resolved from within Docker.
>
> If you remove and delete all volumes also for Consul, and restart everythign this shouldn’t be a problem anymore, as then the processors registers with the IP address of your machine.
> It can be that you initially registered it via „geofence-tutorial“. See the Config.java class
>
> config.register(ConfigKeys.HOST, "geofence-tutorial", "Data processor host“);
>
> It defaults to „geofence-tutorial“ if you not provide an env variable of ConfigKeys.HOST (SP_HOST). This is what you probably did the second time when starting, i.e., SP_HOST=192.168.x.x
> However, upon initial startup the processor falsely registered under the address „geofence-tutorial“ for the corresponding service id which we do not overwrite on later restarts.
>
> To cut the long story short, please run from clean slate and remove all containers and volumes, set SP_HOST to your IP once you start of the service from your IDE and it should work. There shouldn’t
> be any references to a service under the address „geofence-tutorial“ anymore.
>
> A short note: the whole service definition and discovery is completely reconstructed for the next release which will hopefully be much more easy to handle and totally get rid of these potential issues.
>
> Hope this helps :)
>
> Best
> Patrick
>
> Am 31.08.2021 um 15:51 schrieb Robert Edward Herding <robertedward.herding@yandex.com <ma...@yandex.com>>:
>
> Hello Patrick,
>
> I have attached the log file from Consul and Backend services.
>
> Thanks
> Robert
>
> 31.08.2021, 08:54, "Patrick Wiener" <wiener@apache.org <ma...@apache.org>>:
> Hi Robert,
>
> could you please go and check the backend service logs? Maybe it provides more details on why the installation fails.
>
> Patrick
>
> Am 31.08.2021 um 14:49 schrieb Robert Edward Herding <robertedward.herding@yandex.com <ma...@yandex.com>>:
>
> Hi Patrick and Dominik,
>
> Thanks for all the patience. I really appreciate the help.
>
> Yes I am always deleting the volume to have a fresh start. But I still get the red dot.
> Then I tried what Dominik said and now I have a beautiful DataProcessor Geofencing component :) .
> But when I tried to install it:
>
> Installing 1 of 1 (Geofencing)...error
>
> More Details:
> geofence-tutorial: System error
>
> Also again I have got this warning (not sure if this is the problem, but it seems related as it only shows when its time to import the component)
>
> PM org.glassfish.jersey.server.wadl.WadlFeature configure
> WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
>
> I will fix this warning and review the program for some mistype (just to be sure it is perfectly ok) and I hope coming back with good news.
> If you guys have any more ideas please let me know.
>
>
> Best regard,
> Robert Edward
>
> 31.08.2021, 07:13, "Dominik Riemer" <riemer@apache.org <ma...@apache.org>>:
> Hi Robert,
>
> on some development setups the locally started extensions service is not detected as a healthy service by Consul and thus not added to the list of pipeline elements in the UI.
> You can also try to click on "Manage Endpoints" and add the URL of the extensions service manually (just add the URL such as http://192.168.1.12:8092 <http://192.168.1.12:8092/>). Maybe that's also worth a try ;-)
>
> Dominik
>
> On 2021/08/31 10:25:06, Patrick Wiener <wiener@apache.org <ma...@apache.org>> wrote:
>
> Hi Robert,
>
> did you also reset everything to a clean slate using the CLI? There might be some misconfigurations due to your initial settings using „localhost“ that might not get overwritten.
> Hopefully, if you perform the following steps we get you sorted:
>
> Remove StreamPipes and perform cleanup (don’t forget the -v flag as this is required to delete the docker volumes): ./streampipes down -v
> Restart StreamPipes: ./streampipes up -d
> Finish setup process in the SP UI
> Start processor from IDE with your 192.168.x.x IP
> Check the SP UI at the „Configuration“ > „Pipeline Element Configuration“ tab and validate the green dot for your new processor service
> Go to „Install Pipeline Elements“ and check if you can install the new processor
>
> Best
> Patrick
>
> > Am 31.08.2021 um 11:18 schrieb Robert Edward Herding <robertedward.herding@yandex.com <ma...@yandex.com>>:
> >
> > Hello Patrick,
> >
> > Thanks once more for your answer, very interesting.
> > I have set my IP as you said.
> > Now I can access the RDF description, only if I set the IP address in the URL without namespaces, like "http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing <http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing>"
> > But still doesn't import the processor
> > Funny thing I can access the RDF page also with the docker IP
> >
> > Reading the logs I also found an warning, only when accessing the RDF page:
> >
> > AM org.glassfish.jersey.server.wadl.WadlFeature configure
> > WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
> >
> > As you said "manage endpoints" is not connecting to Consul(inside docker), while RDF description page is(outside docker).
> > One question; It is possible to create an external Consul service(in the cloud) and import from there? That would be the easier/faster way ?
> >
> > Best regards,
> > Robert
> >
> >
> >
> > 30.08.2021, 03:27, "Patrick Wiener" <wiener@apache.org <ma...@apache.org>>:
> > Hi Robert,
> >
> > good to hear that we got it to register.
> >
> > The second problem seems to be due to the fact that the service health check on the processor is failing given that you set the url to localhost:8092 for SP_HOST.
> > Thats also why you see the red dot in the overview.
> >
> > The reason for this basically is due to the hybrid dev environment with parts of your setup running in the IDE (the processor) and the other parts inside Docker (here Consul for
> > service discovery and service health check). While connections from your Docker Host (your Debian machine) work on Consul over localhost:8500 (outside -> inside Docker) the
> > issue occurs when services like Consul try to communicate to other services running outside Docker, here on your Debian host, especially via „localhost“. The reason for this is
> > that Docker containers run isolated in their own namespace and are not aware of each other. While docker to docker connections work due to some convenient help from Docker
> > using service DNS names (for containers within the same network), it fails for development purposes (with some exceptions). Hence, when using „localhost“ Consul thinks the service
> > is also running within the same container as Consul is, which is false, as it is running on your machine in your IDE without Docker.
> >
> > So given your setup on a linux machine, what I would recommend for development is to not use „localhost“ but the actual IP of your linux machine (see ifconfig). Docker containers
> > can communicate to the NIC's (e.g. eth0) IP address. However, this only works on linux as Docker runs natively. Yet it is worthy to note that if you regularly change your network environments
> > which may lead to different IP addresses assigned to this NIC, the service health check would start failing again, as the service was originally registered with another IP.
> > We currently work on improving the whole service discovery/service management part to hopefully resolve all these issues within the current dev branch which is part of the next release
> > (0.69.0).
> >
> > As of now, see if it helps using the machine’s IP address (e.g., 192.168.x.x) instead of localhost.
> >
> > Best
> > Patrick
> >
> > Am 29.08.2021 um 17:26 schrieb Robert Edward Herding <robertedward.herding@yandex.com <ma...@yandex.com> <mailto:robertedward.herding@yandex.com <ma...@yandex.com>>>:
> >
> > Hello Patrick,
> >
> > Thank you very much, your explanation helped a lot.
> > I was running the compose docker file, not the cli enviroment.
> > So I created a pipeline element template with cli, and it worked perfectly, pretty straightforward.
> > Now I have a registered service at Consul.
> >
> > After that I tried to import the processor in Install Pipeline Elements, with the url given at http://localhost:8092/ <http://localhost:8092/> <http://localhost:8092/ <http://localhost:8092/>> (I had to change the port).
> > The Install Pipeline Elements shows no elements (it keeps searching) but I can see "geofence tutorial" in Pipeline Element Configuration with a red dot, for sure it is not working but it can see the container.
> > Can you please give me a north on this?
> >
> > Yours sincerely,
> >
> > Robert
> >
> >
> >
> >
> >
> >
> > 28.08.2021, 10:37, "Patrick Wiener" <wiener@apache.org <ma...@apache.org> <mailto:wiener@apache.org <ma...@apache.org>>>:
> > Hi Robert,
> >
> > first of all I’d like to welcome you to the list.
> >
> > I assume you already started the developer environment such that Consul Docker container is running?
> > You can use the StreamPipes CLI to easily configure a development environment. You can find relevant parts from here [1].
> >
> > A little bit of background: we leverage Consul for service discovery such that all extending services, e.g. processors etc
> > first register at Consul from where the StreamPipes core picks up service endpoints of respective extensions.
> > Hence, currently also for development purposes it is necessary to set of a minimal development environment using the CLI and
> > a corresponding environment template.
> >
> > Can you also provide an excerpt of „docker ps -a“?
> >
> > Cheers
> > Patrick
> >
> > [1] https://streampipes.apache.org/docs/docs/extend-setup.html <https://streampipes.apache.org/docs/docs/extend-setup.html> <https://streampipes.apache.org/docs/docs/extend-setup.html <https://streampipes.apache.org/docs/docs/extend-setup.html>>
> >
> > Am 27.08.2021 um 23:19 schrieb Robert Edward Herding <robertedward.herding@yandex.com <ma...@yandex.com> <mailto:robertedward.herding@yandex.com <ma...@yandex.com>>>:
> >
> > Hello folks,
> >
> > I am a developer located in South Africa.
> > I have been testing StreamPipes for a few weeks now and I have decided to build, a processor entity for testing and developing for a future industrial use.
> > Said that I start following the tutorial https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html <https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html> <https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html <https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html>>.
> > The project is building perfectly, but when it gets to the point of registering items in Consul it just loops with the message.
> >
> > 22:41:53.553 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
> > 22:41:53.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
> > 22:41:54.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
> > 22:41:54.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
> > 20:41:55.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
> > 20:41:55.556 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
> >
> > I am really not used to work with Consul, so have been really hard to understand why it can't find it.
> > I tried to change the SP_HOST in the run configuration.
> >
> > localhost
> > 0.0.0.0
> > 127.0.0.1
> > 192.168.1.1
> > 192.168.1.12
> > host.docker.internal
> >
> > Below the docker IP and docker network.
> >
> > udp 0 0 192.168.1.12:bootpc host.docker.inte:bootps ESTABLISHED
> > unix 3 [ ] STREAM CONNECTED 701942 /run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 692953 /run/snap.docker/containerd/containerd.sock
> > unix 3 [ ] STREAM CONNECTED 703571 /run/snap.docker/containerd/containerd.sock
> >
> >
> > NETWORK ID NAME DRIVER SCOPE
> > 305ea4177bdb bridge bridge local
> > 8c6df02eeef9 host host local
> > ead9e6d74aa1 none null local
> >
> > As you can see I am running linux Debian.
> > Any help would be greatly appreciated.
> >
> > Yours sincerely,
> >
> > Robert
> >
>
>
> <backend.log><consul.log>