You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Vineet Tiwari <vi...@gmail.com> on 2021/03/02 13:06:46 UTC

Re: Intend to contribute to Streampipes

Hey Patrick,
Thanks for the clarifications. I was able to get things up and running in
my machine. I have two more questions:

1. As you mentioned for the initialization of extensions, there are various
projects to bundle them. What I gathered from the main class of these
projects, I can see that extensions-all-jvm contains all elements (data
set, stream, processor and sink). The others bundles contain subsets of
these elements (For example: pipeline-elements-all-jvm contains all
elements except data streams and data sets). To run the whole application,
I need to run the extensions-all-jvm so that all the pipeline elements are
initialized. Is my understanding correct?
2. When I navigate to the connect page of the application in UI, I see that
backend tries to get icons for the elements in UI. To successfully get the
icons, I need to run the connect-adapters. What exactly is the role of
connect-adapters here apart from icons fetching.

Thanks,
Vineet Tiwari

On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wi...@apache.org> wrote:

> Hi Vineet,
>
> There are a few points here that will hopefully help you to get your dev
> setup running.
>
> Env variables
> We do recommend using IntelliJ as your IDE as we
> provide IntelliJ specific run configurations as part of the Github
> projects that will configure
> the services with appropriate env variables, e.g. for the core [1] and
> backend [2]. However,
> you can also run from Eclipse but you need to add these env variables
> manually to Eclipse.
>
> General Git project setup
> We currently differentiate between core and extensions git projects. The
> core contains core
> functionality for pipeline management, connect master, serialization,
> messaging wrapper,
> runtime wrapper etc. and two services that are also part of StreamPipes:
>
> - backend service (contains core functionality, e.g. pipeline management,
> connect master)
> - ui service
>
> It is worthy to note that we integrated the connect master service (was a
> standalone service before)
> in the backend service to simplify the overall architecture. This is why
> you did not find a main
> class for it anymore on the current dev branch.
>
> Besides, there exists the extensions project that contains all concrete
> connect adapters as well
> as pipeline elements. We can bundle them in various ways, e.g.
>
> - by type (streampipes-processors-filters-jvm)
> - by wrapper (streampipes-pipeline-elements-all-jvm)
> - all extensions by wrapper (streampipes-extensions-all-jvm)
>
> Dev environment
> We recommend checking out both projects and integrating them in a
> multi-project structure
> in your IDE.
>
> StreamPipes relies on a few external services, e.g. Consul, Couchdb,
> Kafka. To easify the
> development please use the StreamPipes CLI option to start your dev
> environment [3].
>
> Here, there are various environments configurations depending on wether
> you want to develop:
>
> - core only -> backend
> - extensions only -> pipeline-element
> - core + extensions -> basic
>
> Go through the provided README file to see how to use the CLI. Briefly, it
> would be something
> like this for starting a docker dev environment with necessary services:
>
> ./streampipes env --set basic
> ./streampipes pull
> ./streampipes up -d
>
> In your IDE, set the the env variables for the services accordingly either
> using the provided
> run configs for IntelliJ or adding env variables for the StreamPipes to
> e.g. Eclipse by using
> the IntelliJ run configs as a template.
>
> We store this configs in Consul and do currently not overwrite them once a
> service has registered.
> You can either manually delete them from the Consul UI (localhost:8500) or
> simply start from
> a clean setup if you have a misconfiguration:
>
> ./streampipes clean -v
>
> NOTE!
> If non variables are present, all services use default variables that
> represent the prod settings
> when starting the containerized environment. Thus it is important to set
> these variables
> according to your setup.
>
> Networking in dev setup local machine <—> Docker
> This mixed setup: dev in IDE + Dockerized environment causes some network
> problems since
> use Consul (running in a container) to conduct service health checks to a
> service running on
> your local machine (in your IDE). Thus, we need an IP/DNS that is
> reachable/resolvable from
> inside the Docker container.
>
> Here it gets a little tricky as Docker currently does not provide a common
> solution for all OS.
> For instance, for the streampipes-pipeline-elements-all-jvm you will see
> in the run config that
> it sets an env variable as following [4]:
>
> <env name="SP_HOST" value="host.docker.internal" />
>
> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically,
> host.docker.internal
> is a Docker resolvable address on MacOS and Windows [5]
>
> For Linux:
> Whenever you see and env variable that uses „host.docker.internal“ as a
> value, you need to
> add the IP of you local machine
>
> ——
>
> Regarding your questions:
>
> > 1. I was trying to start the backend module, which started fine, but when
> > executing from UI, it is not able to connect to consul and mq due to host
> > not found. I think I have not set the env variables correctly. I tried
> > setting them in eclipse, but the host values are being taken from the
> > consul key value store. How to change that to localhost?
> Related due to missing env variables when starting the backend main class.
> I assume, it’s trying to connect to something like consul:8500 instead of
> localhost:8500.
> Try with a clean setup, i.e., remove all docker volumes using the clean
> command
> above and setting the variables beforehand.
>
>
> > 2. There are instructions to setup master container on the confluence,
> but
> > I could not find its main class.
> This is most likely outdated, as explained above we recently integrated
> connect
> master into the backend. This is why you didn’t find a main class for the
> connect master.
>
>
> > 3. Out of all the containers in docker, I am not sure of the function and
> > code inside these two containers - pipeline-element-all and
> connect-worker.
> > Also, once started, the connect-worker keeps on trying to connect to the
> > backend api on http://backend:8030 by default. How do I change it to
> > localhost?
> For the first part, I assume with function and code you mean where to find
> it?
> As said, we provide various bundles for pipeline elements.
>
> pipeline-element-all-jvm bundles all pipeline element modules of the
> extensions
> projects that are written for the plain Java wrapper (jvm) [6] (see the
> main class).
>
> connect-worker bundles all adapters [7] (see the main class).
>
> For the second part, this is mostly related again to missing env variables
> when starting
> the service. „backend“ is the default name of the docker service in prod.
> Take a look
> at the run configs for the env variables [8]. There you’ll find the
> following entry:
>
> <env name="SP_BACKEND_HOST" value="localhost" />
>
> These env variables should be set in your IDE when starting your service.
>
>
> Hope this guide was helpful.
>
> We are currently restructuring the documentation, thus in the future we
> will add description as above
> right in the documentation.
>
>
> Patrick
>
>
> [1]
> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
> <
> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
> >
> [2]
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations
> <
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations>
>
> [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli
> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>
> [4]
> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml
> <
> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml>
>
> [5] https://docs.docker.com/docker-for-windows/networking/ <
> https://docs.docker.com/docker-for-windows/networking/>
> [6]
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm
> <
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm>
>
> [7]
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters
> <
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters>
>
> [8]
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml
> <
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml>
>
>
> > Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
> >
> > Hello,
> > I have gone through some of the materials given above (thanks for that)
> and
> > I have got a good idea on the high level architecture and design of the
> > application.
> >
> > I started to set up streampipes on my local machine as well.
> > I have the below queries:
> > 1. I was trying to start the backend module, which started fine, but when
> > executing from UI, it is not able to connect to consul and mq due to host
> > not found. I think I have not set the env variables correctly. I tried
> > setting them in eclipse, but the host values are being taken from the
> > consul key value store. How to change that to localhost?
> > 2. There are instructions to setup master container on the confluence,
> but
> > I could not find its main class.
> > 3. Out of all the containers in docker, I am not sure of the function and
> > code inside these two containers - pipeline-element-all and
> connect-worker.
> > Also, once started, the connect-worker keeps on trying to connect to the
> > backend api on http://backend:8030 by default. How do I change it to
> > localhost?
> >
> > Thanks,
> > Vineet Tiwari
> >
> > On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
> wrote:
> >
> >> Hi Vineet,
> >>
> >> welcome!
> >> If you have any questions on how to get started, we'll be glad to help.
> >>
> >> Doominik
> >>
> >> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
> >>> Hello,
> >>> My name is Vineet Tiwari and I am working as a full time software
> >>> developer. I am pretty good at Java and javascript (mostly with
> ReactJs)
> >>> languages along with basic C++ and Python.
> >>> I have a total of three and a half years of experience working as a SDE
> >> on
> >>> several projects primarily with JAVA programming language.
> >>>
> >>> I do not have any prior open source contributions. I have just started
> to
> >>> explore open-source projects and I found this project very
> interesting. I
> >>> have gone through the website as well as the confluence page and I feel
> >>> that I can surely contribute well to this project with my skills and
> >>> capabilities.
> >>>
> >>> I have subscribed to the dev and commits mailing lists and I look
> forward
> >>> to hearing from you about how I can contribute to the project.
> >>>
> >>> Thanks and regards
> >>> Vineet Tiwari
> >>>
> >>
>
>

RE: Intend to contribute to Streampipes

Posted by Dominik Riemer <ri...@apache.org>.
Hi Vineet,

sounds indeed like a serialization issue - is the new field part of the object before it is serialized in the REST interface?
In general, we use Jackson to serialize objects and there shouldn't be any additional configuration needed when using the JacksonSerializer from streampipes-serializers. Just as background information, we are slowly removing the RDF annotations from the framework and switch to plain JSON, so there shouldn't be any RDF anymore in the Core-UI communication.
If you send me a code snippet or a link to the code, I'll be happy to have a look.

Dominik

-----Original Message-----
From: Vineet Tiwari <vi...@gmail.com> 
Sent: Monday, March 8, 2021 3:35 PM
To: dev@streampipes.apache.org
Subject: Re: Intend to contribute to Streampipes

Hey Guys,
I picked up this issue
https://issues.apache.org/jira/browse/STREAMPIPES-48
and started working on it. I was able to do most of the things. I am planning to reuse the EventPropertyNested instead of EventPropertyList for the lists containing objects. In order to differentiate nested objects and nested list, I have created another boolean property inside EventPropertyNested. I am able to see the nested properties displayed in the UI.
The only issue is that I am not able to see the new property that I have added in the JSON response from the guess service. Because of this, I am not able to use it in the UI.
I believe that there is some namespace library being used to serialise java to json objects. I even tried using the RDF annotations but I could not get it to work. I think there is some fundamental info that I am missing on this RDF library. Can someone please help me with this?

Thanks,
Vineet Tiwari

On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <vi...@gmail.com> wrote:

> Thanks Patrick. It is clear now.
> Currently I am going through the backend code and understanding how 
> you are creating and running the pipeline graphs in the backend code. 
> Also I am going through some of the pipeline elements.
> Do let me know if there are any issues I can look into. Anything that 
> you think may be suitable for me to look at.
>
> Thanks,
> Vineet Tiwari
>
> On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <wi...@fzi.de> wrote:
>
>> Hi Vineet,
>>
>> nice to hear that you got it running.
>>
>> Regarding your questions:
>>
>> for 1)
>> the options are up to you - „extensions-all-jvm“ is like a „fat“ 
>> bundle that contains everything.
>> It basically is made of connect-adapter + pe-all-jvm. This is kinda 
>> experimental as we ship things in Docker images and as part of this 
>> option, we can reduce the overall system load in a single node setup 
>> due to less resource consumption for multiple JVM instances inside 
>> separate containers.
>>
>> However, for developing I’d suggest you start the adapters and 
>> pe-all-jvm separately.
>>
>> for 2)
>> the connect-adapters service does the actual data connection to 
>> external data sources, e.g. kafka, mqtt, PLC as part of StreamPipes 
>> connect. Hence, this service runs the actually configured adapter 
>> instances in separate threads. Apart from that, in the 
>> setup/registration phase, the connect-adapter service tries to 
>> register at the backend and provides an adapter description which 
>> contains all the information such as what kind of protocols etc are 
>> supported by that service. Which is stored in Couchdb.
>> At runtime, once you browse to the UI and got to the Connect adapter 
>> market place, the backend queries and endpoint of the connect-adapter 
>> service to check if is still actively running and besides retrieves 
>> the dedicated icons.
>>
>> Hope this helps.
>>
>> Patrick
>>
>>
>>
>> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hey Patrick,
>> Thanks for the clarifications. I was able to get things up and 
>> running in my machine. I have two more questions:
>>
>> 1. As you mentioned for the initialization of extensions, there are 
>> various projects to bundle them. What I gathered from the main class 
>> of these projects, I can see that extensions-all-jvm contains all 
>> elements (data set, stream, processor and sink). The others bundles 
>> contain subsets of these elements (For example: 
>> pipeline-elements-all-jvm contains all elements except data streams 
>> and data sets). To run the whole application, I need to run the 
>> extensions-all-jvm so that all the pipeline elements are initialized. 
>> Is my understanding correct?
>> 2. When I navigate to the connect page of the application in UI, I 
>> see that backend tries to get icons for the elements in UI. To 
>> successfully get the icons, I need to run the connect-adapters. What 
>> exactly is the role of connect-adapters here apart from icons 
>> fetching.
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi Vineet,
>>
>> There are a few points here that will hopefully help you to get your 
>> dev setup running.
>>
>> Env variables
>> We do recommend using IntelliJ as your IDE as we provide IntelliJ 
>> specific run configurations as part of the Github projects that will 
>> configure the services with appropriate env variables, e.g. for the 
>> core [1] and backend [2]. However, you can also run from Eclipse but 
>> you need to add these env variables manually to Eclipse.
>>
>> General Git project setup
>> We currently differentiate between core and extensions git projects. 
>> The core contains core functionality for pipeline management, connect 
>> master, serialization, messaging wrapper, runtime wrapper etc. and 
>> two services that are also part of StreamPipes:
>>
>> - backend service (contains core functionality, e.g. pipeline 
>> management, connect master)
>> - ui service
>>
>> It is worthy to note that we integrated the connect master service 
>> (was a standalone service before) in the backend service to simplify 
>> the overall architecture. This is why you did not find a main class 
>> for it anymore on the current dev branch.
>>
>> Besides, there exists the extensions project that contains all 
>> concrete connect adapters as well as pipeline elements. We can bundle 
>> them in various ways, e.g.
>>
>> - by type (streampipes-processors-filters-jvm)
>> - by wrapper (streampipes-pipeline-elements-all-jvm)
>> - all extensions by wrapper (streampipes-extensions-all-jvm)
>>
>> Dev environment
>> We recommend checking out both projects and integrating them in a 
>> multi-project structure in your IDE.
>>
>> StreamPipes relies on a few external services, e.g. Consul, Couchdb, 
>> Kafka. To easify the development please use the StreamPipes CLI 
>> option to start your dev environment [3].
>>
>> Here, there are various environments configurations depending on 
>> wether you want to develop:
>>
>> - core only -> backend
>> - extensions only -> pipeline-element
>> - core + extensions -> basic
>>
>> Go through the provided README file to see how to use the CLI. 
>> Briefly, it would be something like this for starting a docker dev 
>> environment with necessary services:
>>
>> ./streampipes env --set basic
>> ./streampipes pull
>> ./streampipes up -d
>>
>> In your IDE, set the the env variables for the services accordingly 
>> either using the provided run configs for IntelliJ or adding env 
>> variables for the StreamPipes to e.g. Eclipse by using the IntelliJ 
>> run configs as a template.
>>
>> We store this configs in Consul and do currently not overwrite them 
>> once a service has registered.
>> You can either manually delete them from the Consul UI 
>> (localhost:8500) or simply start from a clean setup if you have a 
>> misconfiguration:
>>
>> ./streampipes clean -v
>>
>> NOTE!
>> If non variables are present, all services use default variables that 
>> represent the prod settings when starting the containerized 
>> environment. Thus it is important to set these variables according to 
>> your setup.
>>
>> Networking in dev setup local machine <—> Docker This mixed setup: 
>> dev in IDE + Dockerized environment causes some network problems 
>> since use Consul (running in a container) to conduct service health 
>> checks to a service running on your local machine (in your IDE). 
>> Thus, we need an IP/DNS that is reachable/resolvable from inside the 
>> Docker container.
>>
>> Here it gets a little tricky as Docker currently does not provide a 
>> common solution for all OS.
>> For instance, for the streampipes-pipeline-elements-all-jvm you will 
>> see in the run config that it sets an env variable as following [4]:
>>
>> <env name="SP_HOST" value="host.docker.internal" />
>>
>> Here, SP_HOST refers to the IP/DNS of the service endpoint. 
>> Specifically, host.docker.internal is a Docker resolvable address on 
>> MacOS and Windows [5]
>>
>> For Linux:
>> Whenever you see and env variable that uses „host.docker.internal“ as 
>> a value, you need to add the IP of you local machine
>>
>> ——
>>
>> Regarding your questions:
>>
>> 1. I was trying to start the backend module, which started fine, but 
>> when executing from UI, it is not able to connect to consul and mq 
>> due to host not found. I think I have not set the env variables 
>> correctly. I tried setting them in eclipse, but the host values are 
>> being taken from the consul key value store. How to change that to localhost?
>>
>> Related due to missing env variables when starting the backend main class.
>> I assume, it’s trying to connect to something like consul:8500 
>> instead of localhost:8500.
>> Try with a clean setup, i.e., remove all docker volumes using the 
>> clean command above and setting the variables beforehand.
>>
>>
>> 2. There are instructions to setup master container on the 
>> confluence,
>>
>> but
>>
>> I could not find its main class.
>>
>> This is most likely outdated, as explained above we recently 
>> integrated connect master into the backend. This is why you didn’t 
>> find a main class for the connect master.
>>
>>
>> 3. Out of all the containers in docker, I am not sure of the function 
>> and code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to 
>> the backend api on http://backend:8030 by default. How do I change it 
>> to localhost?
>>
>> For the first part, I assume with function and code you mean where to 
>> find it?
>> As said, we provide various bundles for pipeline elements.
>>
>> pipeline-element-all-jvm bundles all pipeline element modules of the 
>> extensions projects that are written for the plain Java wrapper (jvm) 
>> [6] (see the main class).
>>
>> connect-worker bundles all adapters [7] (see the main class).
>>
>> For the second part, this is mostly related again to missing env 
>> variables when starting the service. „backend“ is the default name of 
>> the docker service in prod.
>> Take a look
>> at the run configs for the env variables [8]. There you’ll find the 
>> following entry:
>>
>> <env name="SP_BACKEND_HOST" value="localhost" />
>>
>> These env variables should be set in your IDE when starting your service.
>>
>>
>> Hope this guide was helpful.
>>
>> We are currently restructuring the documentation, thus in the future 
>> we will add description as above right in the documentation.
>>
>>
>> Patrick
>>
>>
>> [1]
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon
>> figurations
>> <
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon
>> figurations
>>
>>
>> [2]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.
>> idea/runConfigurations
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.
>> idea/runConfigurations
>> >
>>
>> [3]
>> https://github.com/apache/incubator-streampipes-installer/tree/dev/cl
>> i 
>> <https://github.com/apache/incubator-streampipes-installer/tree/dev/c
>> li>
>> [4]
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon
>> figurations/all_pipeline_elements_jvm.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon
>> figurations/all_pipeline_elements_jvm.xml
>> >
>>
>> [5] https://docs.docker.com/docker-for-windows/networking/ < 
>> https://docs.docker.com/docker-for-windows/networking/>
>> [6]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-pipeline-elements-all-jvm
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-pipeline-elements-all-jvm
>> >
>>
>> [7]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-connect-adapters
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-connect-adapters
>> >
>>
>> [8]
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.
>> idea/runConfigurations/connect_adapters.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.
>> idea/runConfigurations/connect_adapters.xml
>> >
>>
>>
>> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hello,
>> I have gone through some of the materials given above (thanks for 
>> that)
>>
>> and
>>
>> I have got a good idea on the high level architecture and design of 
>> the application.
>>
>> I started to set up streampipes on my local machine as well.
>> I have the below queries:
>> 1. I was trying to start the backend module, which started fine, but 
>> when executing from UI, it is not able to connect to consul and mq 
>> due to host not found. I think I have not set the env variables 
>> correctly. I tried setting them in eclipse, but the host values are 
>> being taken from the consul key value store. How to change that to localhost?
>> 2. There are instructions to setup master container on the 
>> confluence,
>>
>> but
>>
>> I could not find its main class.
>> 3. Out of all the containers in docker, I am not sure of the function 
>> and code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to 
>> the backend api on http://backend:8030 by default. How do I change it 
>> to localhost?
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
>>
>> wrote:
>>
>>
>> Hi Vineet,
>>
>> welcome!
>> If you have any questions on how to get started, we'll be glad to help.
>>
>> Doominik
>>
>> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
>>
>> Hello,
>> My name is Vineet Tiwari and I am working as a full time software 
>> developer. I am pretty good at Java and javascript (mostly with
>>
>> ReactJs)
>>
>> languages along with basic C++ and Python.
>> I have a total of three and a half years of experience working as a 
>> SDE
>>
>> on
>>
>> several projects primarily with JAVA programming language.
>>
>> I do not have any prior open source contributions. I have just 
>> started
>>
>> to
>>
>> explore open-source projects and I found this project very
>>
>> interesting. I
>>
>> have gone through the website as well as the confluence page and I 
>> feel that I can surely contribute well to this project with my skills 
>> and capabilities.
>>
>> I have subscribed to the dev and commits mailing lists and I look
>>
>> forward
>>
>> to hearing from you about how I can contribute to the project.
>>
>> Thanks and regards
>> Vineet Tiwari
>>
>>
>>


RE: Intend to contribute to Streampipes

Posted by Dominik Riemer <ri...@apache.org>.
Hi Vineet,

just two additions that might help to better understand our serialization layer:
- REST interfaces can be annotated with the @JacksonSerializer annotation which ensures that our custom serializer is used.
- We auto-generate typescript objects from models that are annotated with the @TsModel annotation, there is some information on how to update the generated TS classes in the wiki at [1].

Hope this helps!

Dominik

[1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Core-UI+Serialization


-----Original Message-----
From: Vineet Tiwari <vi...@gmail.com> 
Sent: Monday, March 8, 2021 3:35 PM
To: dev@streampipes.apache.org
Subject: Re: Intend to contribute to Streampipes

Hey Guys,
I picked up this issue
https://issues.apache.org/jira/browse/STREAMPIPES-48
and started working on it. I was able to do most of the things. I am planning to reuse the EventPropertyNested instead of EventPropertyList for the lists containing objects. In order to differentiate nested objects and nested list, I have created another boolean property inside EventPropertyNested. I am able to see the nested properties displayed in the UI.
The only issue is that I am not able to see the new property that I have added in the JSON response from the guess service. Because of this, I am not able to use it in the UI.
I believe that there is some namespace library being used to serialise java to json objects. I even tried using the RDF annotations but I could not get it to work. I think there is some fundamental info that I am missing on this RDF library. Can someone please help me with this?

Thanks,
Vineet Tiwari

On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <vi...@gmail.com> wrote:

> Thanks Patrick. It is clear now.
> Currently I am going through the backend code and understanding how 
> you are creating and running the pipeline graphs in the backend code. 
> Also I am going through some of the pipeline elements.
> Do let me know if there are any issues I can look into. Anything that 
> you think may be suitable for me to look at.
>
> Thanks,
> Vineet Tiwari
>
> On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <wi...@fzi.de> wrote:
>
>> Hi Vineet,
>>
>> nice to hear that you got it running.
>>
>> Regarding your questions:
>>
>> for 1)
>> the options are up to you - „extensions-all-jvm“ is like a „fat“ 
>> bundle that contains everything.
>> It basically is made of connect-adapter + pe-all-jvm. This is kinda 
>> experimental as we ship things in Docker images and as part of this 
>> option, we can reduce the overall system load in a single node setup 
>> due to less resource consumption for multiple JVM instances inside 
>> separate containers.
>>
>> However, for developing I’d suggest you start the adapters and 
>> pe-all-jvm separately.
>>
>> for 2)
>> the connect-adapters service does the actual data connection to 
>> external data sources, e.g. kafka, mqtt, PLC as part of StreamPipes 
>> connect. Hence, this service runs the actually configured adapter 
>> instances in separate threads. Apart from that, in the 
>> setup/registration phase, the connect-adapter service tries to 
>> register at the backend and provides an adapter description which 
>> contains all the information such as what kind of protocols etc are 
>> supported by that service. Which is stored in Couchdb.
>> At runtime, once you browse to the UI and got to the Connect adapter 
>> market place, the backend queries and endpoint of the connect-adapter 
>> service to check if is still actively running and besides retrieves 
>> the dedicated icons.
>>
>> Hope this helps.
>>
>> Patrick
>>
>>
>>
>> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hey Patrick,
>> Thanks for the clarifications. I was able to get things up and 
>> running in my machine. I have two more questions:
>>
>> 1. As you mentioned for the initialization of extensions, there are 
>> various projects to bundle them. What I gathered from the main class 
>> of these projects, I can see that extensions-all-jvm contains all 
>> elements (data set, stream, processor and sink). The others bundles 
>> contain subsets of these elements (For example: 
>> pipeline-elements-all-jvm contains all elements except data streams 
>> and data sets). To run the whole application, I need to run the 
>> extensions-all-jvm so that all the pipeline elements are initialized. 
>> Is my understanding correct?
>> 2. When I navigate to the connect page of the application in UI, I 
>> see that backend tries to get icons for the elements in UI. To 
>> successfully get the icons, I need to run the connect-adapters. What 
>> exactly is the role of connect-adapters here apart from icons 
>> fetching.
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi Vineet,
>>
>> There are a few points here that will hopefully help you to get your 
>> dev setup running.
>>
>> Env variables
>> We do recommend using IntelliJ as your IDE as we provide IntelliJ 
>> specific run configurations as part of the Github projects that will 
>> configure the services with appropriate env variables, e.g. for the 
>> core [1] and backend [2]. However, you can also run from Eclipse but 
>> you need to add these env variables manually to Eclipse.
>>
>> General Git project setup
>> We currently differentiate between core and extensions git projects. 
>> The core contains core functionality for pipeline management, connect 
>> master, serialization, messaging wrapper, runtime wrapper etc. and 
>> two services that are also part of StreamPipes:
>>
>> - backend service (contains core functionality, e.g. pipeline 
>> management, connect master)
>> - ui service
>>
>> It is worthy to note that we integrated the connect master service 
>> (was a standalone service before) in the backend service to simplify 
>> the overall architecture. This is why you did not find a main class 
>> for it anymore on the current dev branch.
>>
>> Besides, there exists the extensions project that contains all 
>> concrete connect adapters as well as pipeline elements. We can bundle 
>> them in various ways, e.g.
>>
>> - by type (streampipes-processors-filters-jvm)
>> - by wrapper (streampipes-pipeline-elements-all-jvm)
>> - all extensions by wrapper (streampipes-extensions-all-jvm)
>>
>> Dev environment
>> We recommend checking out both projects and integrating them in a 
>> multi-project structure in your IDE.
>>
>> StreamPipes relies on a few external services, e.g. Consul, Couchdb, 
>> Kafka. To easify the development please use the StreamPipes CLI 
>> option to start your dev environment [3].
>>
>> Here, there are various environments configurations depending on 
>> wether you want to develop:
>>
>> - core only -> backend
>> - extensions only -> pipeline-element
>> - core + extensions -> basic
>>
>> Go through the provided README file to see how to use the CLI. 
>> Briefly, it would be something like this for starting a docker dev 
>> environment with necessary services:
>>
>> ./streampipes env --set basic
>> ./streampipes pull
>> ./streampipes up -d
>>
>> In your IDE, set the the env variables for the services accordingly 
>> either using the provided run configs for IntelliJ or adding env 
>> variables for the StreamPipes to e.g. Eclipse by using the IntelliJ 
>> run configs as a template.
>>
>> We store this configs in Consul and do currently not overwrite them 
>> once a service has registered.
>> You can either manually delete them from the Consul UI 
>> (localhost:8500) or simply start from a clean setup if you have a 
>> misconfiguration:
>>
>> ./streampipes clean -v
>>
>> NOTE!
>> If non variables are present, all services use default variables that 
>> represent the prod settings when starting the containerized 
>> environment. Thus it is important to set these variables according to 
>> your setup.
>>
>> Networking in dev setup local machine <—> Docker This mixed setup: 
>> dev in IDE + Dockerized environment causes some network problems 
>> since use Consul (running in a container) to conduct service health 
>> checks to a service running on your local machine (in your IDE). 
>> Thus, we need an IP/DNS that is reachable/resolvable from inside the 
>> Docker container.
>>
>> Here it gets a little tricky as Docker currently does not provide a 
>> common solution for all OS.
>> For instance, for the streampipes-pipeline-elements-all-jvm you will 
>> see in the run config that it sets an env variable as following [4]:
>>
>> <env name="SP_HOST" value="host.docker.internal" />
>>
>> Here, SP_HOST refers to the IP/DNS of the service endpoint. 
>> Specifically, host.docker.internal is a Docker resolvable address on 
>> MacOS and Windows [5]
>>
>> For Linux:
>> Whenever you see and env variable that uses „host.docker.internal“ as 
>> a value, you need to add the IP of you local machine
>>
>> ——
>>
>> Regarding your questions:
>>
>> 1. I was trying to start the backend module, which started fine, but 
>> when executing from UI, it is not able to connect to consul and mq 
>> due to host not found. I think I have not set the env variables 
>> correctly. I tried setting them in eclipse, but the host values are 
>> being taken from the consul key value store. How to change that to localhost?
>>
>> Related due to missing env variables when starting the backend main class.
>> I assume, it’s trying to connect to something like consul:8500 
>> instead of localhost:8500.
>> Try with a clean setup, i.e., remove all docker volumes using the 
>> clean command above and setting the variables beforehand.
>>
>>
>> 2. There are instructions to setup master container on the 
>> confluence,
>>
>> but
>>
>> I could not find its main class.
>>
>> This is most likely outdated, as explained above we recently 
>> integrated connect master into the backend. This is why you didn’t 
>> find a main class for the connect master.
>>
>>
>> 3. Out of all the containers in docker, I am not sure of the function 
>> and code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to 
>> the backend api on http://backend:8030 by default. How do I change it 
>> to localhost?
>>
>> For the first part, I assume with function and code you mean where to 
>> find it?
>> As said, we provide various bundles for pipeline elements.
>>
>> pipeline-element-all-jvm bundles all pipeline element modules of the 
>> extensions projects that are written for the plain Java wrapper (jvm) 
>> [6] (see the main class).
>>
>> connect-worker bundles all adapters [7] (see the main class).
>>
>> For the second part, this is mostly related again to missing env 
>> variables when starting the service. „backend“ is the default name of 
>> the docker service in prod.
>> Take a look
>> at the run configs for the env variables [8]. There you’ll find the 
>> following entry:
>>
>> <env name="SP_BACKEND_HOST" value="localhost" />
>>
>> These env variables should be set in your IDE when starting your service.
>>
>>
>> Hope this guide was helpful.
>>
>> We are currently restructuring the documentation, thus in the future 
>> we will add description as above right in the documentation.
>>
>>
>> Patrick
>>
>>
>> [1]
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon
>> figurations
>> <
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon
>> figurations
>>
>>
>> [2]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.
>> idea/runConfigurations
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.
>> idea/runConfigurations
>> >
>>
>> [3]
>> https://github.com/apache/incubator-streampipes-installer/tree/dev/cl
>> i 
>> <https://github.com/apache/incubator-streampipes-installer/tree/dev/c
>> li>
>> [4]
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon
>> figurations/all_pipeline_elements_jvm.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon
>> figurations/all_pipeline_elements_jvm.xml
>> >
>>
>> [5] https://docs.docker.com/docker-for-windows/networking/ < 
>> https://docs.docker.com/docker-for-windows/networking/>
>> [6]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-pipeline-elements-all-jvm
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-pipeline-elements-all-jvm
>> >
>>
>> [7]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-connect-adapters
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s
>> treampipes-connect-adapters
>> >
>>
>> [8]
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.
>> idea/runConfigurations/connect_adapters.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.
>> idea/runConfigurations/connect_adapters.xml
>> >
>>
>>
>> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hello,
>> I have gone through some of the materials given above (thanks for 
>> that)
>>
>> and
>>
>> I have got a good idea on the high level architecture and design of 
>> the application.
>>
>> I started to set up streampipes on my local machine as well.
>> I have the below queries:
>> 1. I was trying to start the backend module, which started fine, but 
>> when executing from UI, it is not able to connect to consul and mq 
>> due to host not found. I think I have not set the env variables 
>> correctly. I tried setting them in eclipse, but the host values are 
>> being taken from the consul key value store. How to change that to localhost?
>> 2. There are instructions to setup master container on the 
>> confluence,
>>
>> but
>>
>> I could not find its main class.
>> 3. Out of all the containers in docker, I am not sure of the function 
>> and code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to 
>> the backend api on http://backend:8030 by default. How do I change it 
>> to localhost?
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
>>
>> wrote:
>>
>>
>> Hi Vineet,
>>
>> welcome!
>> If you have any questions on how to get started, we'll be glad to help.
>>
>> Doominik
>>
>> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
>>
>> Hello,
>> My name is Vineet Tiwari and I am working as a full time software 
>> developer. I am pretty good at Java and javascript (mostly with
>>
>> ReactJs)
>>
>> languages along with basic C++ and Python.
>> I have a total of three and a half years of experience working as a 
>> SDE
>>
>> on
>>
>> several projects primarily with JAVA programming language.
>>
>> I do not have any prior open source contributions. I have just 
>> started
>>
>> to
>>
>> explore open-source projects and I found this project very
>>
>> interesting. I
>>
>> have gone through the website as well as the confluence page and I 
>> feel that I can surely contribute well to this project with my skills 
>> and capabilities.
>>
>> I have subscribed to the dev and commits mailing lists and I look
>>
>> forward
>>
>> to hearing from you about how I can contribute to the project.
>>
>> Thanks and regards
>> Vineet Tiwari
>>
>>
>>


Re: Intend to contribute to Streampipes

Posted by Vineet Tiwari <vi...@gmail.com>.
Hey Guys,
I picked up this issue
https://issues.apache.org/jira/browse/STREAMPIPES-48
and started working on it. I was able to do most of the things. I am
planning to reuse the EventPropertyNested instead of EventPropertyList for
the lists containing objects. In order to differentiate nested objects and
nested list, I have created another boolean property inside
EventPropertyNested. I am able to see the nested properties displayed in
the UI.
The only issue is that I am not able to see the new property that I have
added in the JSON response from the guess service. Because of this, I am
not able to use it in the UI.
I believe that there is some namespace library being used to serialise java
to json objects. I even tried using the RDF annotations but I could not get
it to work. I think there is some fundamental info that I am missing on
this RDF library. Can someone please help me with this?

Thanks,
Vineet Tiwari

On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <vi...@gmail.com> wrote:

> Thanks Patrick. It is clear now.
> Currently I am going through the backend code and understanding how you
> are creating and running the pipeline graphs in the backend code. Also I am
> going through some of the pipeline elements.
> Do let me know if there are any issues I can look into. Anything that you
> think may be suitable for me to look at.
>
> Thanks,
> Vineet Tiwari
>
> On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <wi...@fzi.de> wrote:
>
>> Hi Vineet,
>>
>> nice to hear that you got it running.
>>
>> Regarding your questions:
>>
>> for 1)
>> the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle
>> that contains everything.
>> It basically is made of connect-adapter + pe-all-jvm. This is kinda
>> experimental as we ship
>> things in Docker images and as part of this option, we can reduce the
>> overall system load in
>> a single node setup due to less resource consumption for multiple JVM
>> instances inside separate
>> containers.
>>
>> However, for developing I’d suggest you start the adapters and pe-all-jvm
>> separately.
>>
>> for 2)
>> the connect-adapters service does the actual data connection to external
>> data sources, e.g. kafka, mqtt,
>> PLC as part of StreamPipes connect. Hence, this service runs the actually
>> configured adapter
>> instances in separate threads. Apart from that, in the setup/registration
>> phase, the connect-adapter service
>> tries to register at the backend and provides an adapter description
>> which contains all the information
>> such as what kind of protocols etc are supported by that service. Which
>> is stored in Couchdb.
>> At runtime, once you browse to the UI and got to the Connect adapter
>> market place, the backend
>> queries and endpoint of the connect-adapter service to check if is still
>> actively running and besides retrieves
>> the dedicated icons.
>>
>> Hope this helps.
>>
>> Patrick
>>
>>
>>
>> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hey Patrick,
>> Thanks for the clarifications. I was able to get things up and running in
>> my machine. I have two more questions:
>>
>> 1. As you mentioned for the initialization of extensions, there are
>> various
>> projects to bundle them. What I gathered from the main class of these
>> projects, I can see that extensions-all-jvm contains all elements (data
>> set, stream, processor and sink). The others bundles contain subsets of
>> these elements (For example: pipeline-elements-all-jvm contains all
>> elements except data streams and data sets). To run the whole application,
>> I need to run the extensions-all-jvm so that all the pipeline elements are
>> initialized. Is my understanding correct?
>> 2. When I navigate to the connect page of the application in UI, I see
>> that
>> backend tries to get icons for the elements in UI. To successfully get the
>> icons, I need to run the connect-adapters. What exactly is the role of
>> connect-adapters here apart from icons fetching.
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi Vineet,
>>
>> There are a few points here that will hopefully help you to get your dev
>> setup running.
>>
>> Env variables
>> We do recommend using IntelliJ as your IDE as we
>> provide IntelliJ specific run configurations as part of the Github
>> projects that will configure
>> the services with appropriate env variables, e.g. for the core [1] and
>> backend [2]. However,
>> you can also run from Eclipse but you need to add these env variables
>> manually to Eclipse.
>>
>> General Git project setup
>> We currently differentiate between core and extensions git projects. The
>> core contains core
>> functionality for pipeline management, connect master, serialization,
>> messaging wrapper,
>> runtime wrapper etc. and two services that are also part of StreamPipes:
>>
>> - backend service (contains core functionality, e.g. pipeline management,
>> connect master)
>> - ui service
>>
>> It is worthy to note that we integrated the connect master service (was a
>> standalone service before)
>> in the backend service to simplify the overall architecture. This is why
>> you did not find a main
>> class for it anymore on the current dev branch.
>>
>> Besides, there exists the extensions project that contains all concrete
>> connect adapters as well
>> as pipeline elements. We can bundle them in various ways, e.g.
>>
>> - by type (streampipes-processors-filters-jvm)
>> - by wrapper (streampipes-pipeline-elements-all-jvm)
>> - all extensions by wrapper (streampipes-extensions-all-jvm)
>>
>> Dev environment
>> We recommend checking out both projects and integrating them in a
>> multi-project structure
>> in your IDE.
>>
>> StreamPipes relies on a few external services, e.g. Consul, Couchdb,
>> Kafka. To easify the
>> development please use the StreamPipes CLI option to start your dev
>> environment [3].
>>
>> Here, there are various environments configurations depending on wether
>> you want to develop:
>>
>> - core only -> backend
>> - extensions only -> pipeline-element
>> - core + extensions -> basic
>>
>> Go through the provided README file to see how to use the CLI. Briefly, it
>> would be something
>> like this for starting a docker dev environment with necessary services:
>>
>> ./streampipes env --set basic
>> ./streampipes pull
>> ./streampipes up -d
>>
>> In your IDE, set the the env variables for the services accordingly either
>> using the provided
>> run configs for IntelliJ or adding env variables for the StreamPipes to
>> e.g. Eclipse by using
>> the IntelliJ run configs as a template.
>>
>> We store this configs in Consul and do currently not overwrite them once a
>> service has registered.
>> You can either manually delete them from the Consul UI (localhost:8500) or
>> simply start from
>> a clean setup if you have a misconfiguration:
>>
>> ./streampipes clean -v
>>
>> NOTE!
>> If non variables are present, all services use default variables that
>> represent the prod settings
>> when starting the containerized environment. Thus it is important to set
>> these variables
>> according to your setup.
>>
>> Networking in dev setup local machine <—> Docker
>> This mixed setup: dev in IDE + Dockerized environment causes some network
>> problems since
>> use Consul (running in a container) to conduct service health checks to a
>> service running on
>> your local machine (in your IDE). Thus, we need an IP/DNS that is
>> reachable/resolvable from
>> inside the Docker container.
>>
>> Here it gets a little tricky as Docker currently does not provide a common
>> solution for all OS.
>> For instance, for the streampipes-pipeline-elements-all-jvm you will see
>> in the run config that
>> it sets an env variable as following [4]:
>>
>> <env name="SP_HOST" value="host.docker.internal" />
>>
>> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically,
>> host.docker.internal
>> is a Docker resolvable address on MacOS and Windows [5]
>>
>> For Linux:
>> Whenever you see and env variable that uses „host.docker.internal“ as a
>> value, you need to
>> add the IP of you local machine
>>
>> ——
>>
>> Regarding your questions:
>>
>> 1. I was trying to start the backend module, which started fine, but when
>> executing from UI, it is not able to connect to consul and mq due to host
>> not found. I think I have not set the env variables correctly. I tried
>> setting them in eclipse, but the host values are being taken from the
>> consul key value store. How to change that to localhost?
>>
>> Related due to missing env variables when starting the backend main class.
>> I assume, it’s trying to connect to something like consul:8500 instead of
>> localhost:8500.
>> Try with a clean setup, i.e., remove all docker volumes using the clean
>> command
>> above and setting the variables beforehand.
>>
>>
>> 2. There are instructions to setup master container on the confluence,
>>
>> but
>>
>> I could not find its main class.
>>
>> This is most likely outdated, as explained above we recently integrated
>> connect
>> master into the backend. This is why you didn’t find a main class for the
>> connect master.
>>
>>
>> 3. Out of all the containers in docker, I am not sure of the function and
>> code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to the
>> backend api on http://backend:8030 by default. How do I change it to
>> localhost?
>>
>> For the first part, I assume with function and code you mean where to find
>> it?
>> As said, we provide various bundles for pipeline elements.
>>
>> pipeline-element-all-jvm bundles all pipeline element modules of the
>> extensions
>> projects that are written for the plain Java wrapper (jvm) [6] (see the
>> main class).
>>
>> connect-worker bundles all adapters [7] (see the main class).
>>
>> For the second part, this is mostly related again to missing env variables
>> when starting
>> the service. „backend“ is the default name of the docker service in prod.
>> Take a look
>> at the run configs for the env variables [8]. There you’ll find the
>> following entry:
>>
>> <env name="SP_BACKEND_HOST" value="localhost" />
>>
>> These env variables should be set in your IDE when starting your service.
>>
>>
>> Hope this guide was helpful.
>>
>> We are currently restructuring the documentation, thus in the future we
>> will add description as above
>> right in the documentation.
>>
>>
>> Patrick
>>
>>
>> [1]
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
>> <
>>
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
>>
>>
>> [2]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations
>> >
>>
>> [3]
>> https://github.com/apache/incubator-streampipes-installer/tree/dev/cli
>> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>
>> [4]
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml
>> >
>>
>> [5] https://docs.docker.com/docker-for-windows/networking/ <
>> https://docs.docker.com/docker-for-windows/networking/>
>> [6]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm
>> >
>>
>> [7]
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters
>> >
>>
>> [8]
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml
>> <
>>
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml
>> >
>>
>>
>> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
>>
>> Hello,
>> I have gone through some of the materials given above (thanks for that)
>>
>> and
>>
>> I have got a good idea on the high level architecture and design of the
>> application.
>>
>> I started to set up streampipes on my local machine as well.
>> I have the below queries:
>> 1. I was trying to start the backend module, which started fine, but when
>> executing from UI, it is not able to connect to consul and mq due to host
>> not found. I think I have not set the env variables correctly. I tried
>> setting them in eclipse, but the host values are being taken from the
>> consul key value store. How to change that to localhost?
>> 2. There are instructions to setup master container on the confluence,
>>
>> but
>>
>> I could not find its main class.
>> 3. Out of all the containers in docker, I am not sure of the function and
>> code inside these two containers - pipeline-element-all and
>>
>> connect-worker.
>>
>> Also, once started, the connect-worker keeps on trying to connect to the
>> backend api on http://backend:8030 by default. How do I change it to
>> localhost?
>>
>> Thanks,
>> Vineet Tiwari
>>
>> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
>>
>> wrote:
>>
>>
>> Hi Vineet,
>>
>> welcome!
>> If you have any questions on how to get started, we'll be glad to help.
>>
>> Doominik
>>
>> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
>>
>> Hello,
>> My name is Vineet Tiwari and I am working as a full time software
>> developer. I am pretty good at Java and javascript (mostly with
>>
>> ReactJs)
>>
>> languages along with basic C++ and Python.
>> I have a total of three and a half years of experience working as a SDE
>>
>> on
>>
>> several projects primarily with JAVA programming language.
>>
>> I do not have any prior open source contributions. I have just started
>>
>> to
>>
>> explore open-source projects and I found this project very
>>
>> interesting. I
>>
>> have gone through the website as well as the confluence page and I feel
>> that I can surely contribute well to this project with my skills and
>> capabilities.
>>
>> I have subscribed to the dev and commits mailing lists and I look
>>
>> forward
>>
>> to hearing from you about how I can contribute to the project.
>>
>> Thanks and regards
>> Vineet Tiwari
>>
>>
>>

Re: Intend to contribute to Streampipes

Posted by Vineet Tiwari <vi...@gmail.com>.
Thanks Patrick. It is clear now.
Currently I am going through the backend code and understanding how you are
creating and running the pipeline graphs in the backend code. Also I am
going through some of the pipeline elements.
Do let me know if there are any issues I can look into. Anything that you
think may be suitable for me to look at.

Thanks,
Vineet Tiwari

On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <wi...@fzi.de> wrote:

> Hi Vineet,
>
> nice to hear that you got it running.
>
> Regarding your questions:
>
> for 1)
> the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle
> that contains everything.
> It basically is made of connect-adapter + pe-all-jvm. This is kinda
> experimental as we ship
> things in Docker images and as part of this option, we can reduce the
> overall system load in
> a single node setup due to less resource consumption for multiple JVM
> instances inside separate
> containers.
>
> However, for developing I’d suggest you start the adapters and pe-all-jvm
> separately.
>
> for 2)
> the connect-adapters service does the actual data connection to external
> data sources, e.g. kafka, mqtt,
> PLC as part of StreamPipes connect. Hence, this service runs the actually
> configured adapter
> instances in separate threads. Apart from that, in the setup/registration
> phase, the connect-adapter service
> tries to register at the backend and provides an adapter description which
> contains all the information
> such as what kind of protocols etc are supported by that service. Which is
> stored in Couchdb.
> At runtime, once you browse to the UI and got to the Connect adapter
> market place, the backend
> queries and endpoint of the connect-adapter service to check if is still
> actively running and besides retrieves
> the dedicated icons.
>
> Hope this helps.
>
> Patrick
>
>
>
> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <vi...@gmail.com>:
>
> Hey Patrick,
> Thanks for the clarifications. I was able to get things up and running in
> my machine. I have two more questions:
>
> 1. As you mentioned for the initialization of extensions, there are various
> projects to bundle them. What I gathered from the main class of these
> projects, I can see that extensions-all-jvm contains all elements (data
> set, stream, processor and sink). The others bundles contain subsets of
> these elements (For example: pipeline-elements-all-jvm contains all
> elements except data streams and data sets). To run the whole application,
> I need to run the extensions-all-jvm so that all the pipeline elements are
> initialized. Is my understanding correct?
> 2. When I navigate to the connect page of the application in UI, I see that
> backend tries to get icons for the elements in UI. To successfully get the
> icons, I need to run the connect-adapters. What exactly is the role of
> connect-adapters here apart from icons fetching.
>
> Thanks,
> Vineet Tiwari
>
> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wi...@apache.org> wrote:
>
> Hi Vineet,
>
> There are a few points here that will hopefully help you to get your dev
> setup running.
>
> Env variables
> We do recommend using IntelliJ as your IDE as we
> provide IntelliJ specific run configurations as part of the Github
> projects that will configure
> the services with appropriate env variables, e.g. for the core [1] and
> backend [2]. However,
> you can also run from Eclipse but you need to add these env variables
> manually to Eclipse.
>
> General Git project setup
> We currently differentiate between core and extensions git projects. The
> core contains core
> functionality for pipeline management, connect master, serialization,
> messaging wrapper,
> runtime wrapper etc. and two services that are also part of StreamPipes:
>
> - backend service (contains core functionality, e.g. pipeline management,
> connect master)
> - ui service
>
> It is worthy to note that we integrated the connect master service (was a
> standalone service before)
> in the backend service to simplify the overall architecture. This is why
> you did not find a main
> class for it anymore on the current dev branch.
>
> Besides, there exists the extensions project that contains all concrete
> connect adapters as well
> as pipeline elements. We can bundle them in various ways, e.g.
>
> - by type (streampipes-processors-filters-jvm)
> - by wrapper (streampipes-pipeline-elements-all-jvm)
> - all extensions by wrapper (streampipes-extensions-all-jvm)
>
> Dev environment
> We recommend checking out both projects and integrating them in a
> multi-project structure
> in your IDE.
>
> StreamPipes relies on a few external services, e.g. Consul, Couchdb,
> Kafka. To easify the
> development please use the StreamPipes CLI option to start your dev
> environment [3].
>
> Here, there are various environments configurations depending on wether
> you want to develop:
>
> - core only -> backend
> - extensions only -> pipeline-element
> - core + extensions -> basic
>
> Go through the provided README file to see how to use the CLI. Briefly, it
> would be something
> like this for starting a docker dev environment with necessary services:
>
> ./streampipes env --set basic
> ./streampipes pull
> ./streampipes up -d
>
> In your IDE, set the the env variables for the services accordingly either
> using the provided
> run configs for IntelliJ or adding env variables for the StreamPipes to
> e.g. Eclipse by using
> the IntelliJ run configs as a template.
>
> We store this configs in Consul and do currently not overwrite them once a
> service has registered.
> You can either manually delete them from the Consul UI (localhost:8500) or
> simply start from
> a clean setup if you have a misconfiguration:
>
> ./streampipes clean -v
>
> NOTE!
> If non variables are present, all services use default variables that
> represent the prod settings
> when starting the containerized environment. Thus it is important to set
> these variables
> according to your setup.
>
> Networking in dev setup local machine <—> Docker
> This mixed setup: dev in IDE + Dockerized environment causes some network
> problems since
> use Consul (running in a container) to conduct service health checks to a
> service running on
> your local machine (in your IDE). Thus, we need an IP/DNS that is
> reachable/resolvable from
> inside the Docker container.
>
> Here it gets a little tricky as Docker currently does not provide a common
> solution for all OS.
> For instance, for the streampipes-pipeline-elements-all-jvm you will see
> in the run config that
> it sets an env variable as following [4]:
>
> <env name="SP_HOST" value="host.docker.internal" />
>
> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically,
> host.docker.internal
> is a Docker resolvable address on MacOS and Windows [5]
>
> For Linux:
> Whenever you see and env variable that uses „host.docker.internal“ as a
> value, you need to
> add the IP of you local machine
>
> ——
>
> Regarding your questions:
>
> 1. I was trying to start the backend module, which started fine, but when
> executing from UI, it is not able to connect to consul and mq due to host
> not found. I think I have not set the env variables correctly. I tried
> setting them in eclipse, but the host values are being taken from the
> consul key value store. How to change that to localhost?
>
> Related due to missing env variables when starting the backend main class.
> I assume, it’s trying to connect to something like consul:8500 instead of
> localhost:8500.
> Try with a clean setup, i.e., remove all docker volumes using the clean
> command
> above and setting the variables beforehand.
>
>
> 2. There are instructions to setup master container on the confluence,
>
> but
>
> I could not find its main class.
>
> This is most likely outdated, as explained above we recently integrated
> connect
> master into the backend. This is why you didn’t find a main class for the
> connect master.
>
>
> 3. Out of all the containers in docker, I am not sure of the function and
> code inside these two containers - pipeline-element-all and
>
> connect-worker.
>
> Also, once started, the connect-worker keeps on trying to connect to the
> backend api on http://backend:8030 by default. How do I change it to
> localhost?
>
> For the first part, I assume with function and code you mean where to find
> it?
> As said, we provide various bundles for pipeline elements.
>
> pipeline-element-all-jvm bundles all pipeline element modules of the
> extensions
> projects that are written for the plain Java wrapper (jvm) [6] (see the
> main class).
>
> connect-worker bundles all adapters [7] (see the main class).
>
> For the second part, this is mostly related again to missing env variables
> when starting
> the service. „backend“ is the default name of the docker service in prod.
> Take a look
> at the run configs for the env variables [8]. There you’ll find the
> following entry:
>
> <env name="SP_BACKEND_HOST" value="localhost" />
>
> These env variables should be set in your IDE when starting your service.
>
>
> Hope this guide was helpful.
>
> We are currently restructuring the documentation, thus in the future we
> will add description as above
> right in the documentation.
>
>
> Patrick
>
>
> [1]
>
> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
> <
>
> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
>
>
> [2]
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations
> <
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations
> >
>
> [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli
> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>
> [4]
>
> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml
> <
>
> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml
> >
>
> [5] https://docs.docker.com/docker-for-windows/networking/ <
> https://docs.docker.com/docker-for-windows/networking/>
> [6]
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm
> <
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm
> >
>
> [7]
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters
> <
>
> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters
> >
>
> [8]
>
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml
> <
>
> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml
> >
>
>
> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
>
> Hello,
> I have gone through some of the materials given above (thanks for that)
>
> and
>
> I have got a good idea on the high level architecture and design of the
> application.
>
> I started to set up streampipes on my local machine as well.
> I have the below queries:
> 1. I was trying to start the backend module, which started fine, but when
> executing from UI, it is not able to connect to consul and mq due to host
> not found. I think I have not set the env variables correctly. I tried
> setting them in eclipse, but the host values are being taken from the
> consul key value store. How to change that to localhost?
> 2. There are instructions to setup master container on the confluence,
>
> but
>
> I could not find its main class.
> 3. Out of all the containers in docker, I am not sure of the function and
> code inside these two containers - pipeline-element-all and
>
> connect-worker.
>
> Also, once started, the connect-worker keeps on trying to connect to the
> backend api on http://backend:8030 by default. How do I change it to
> localhost?
>
> Thanks,
> Vineet Tiwari
>
> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
>
> wrote:
>
>
> Hi Vineet,
>
> welcome!
> If you have any questions on how to get started, we'll be glad to help.
>
> Doominik
>
> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
>
> Hello,
> My name is Vineet Tiwari and I am working as a full time software
> developer. I am pretty good at Java and javascript (mostly with
>
> ReactJs)
>
> languages along with basic C++ and Python.
> I have a total of three and a half years of experience working as a SDE
>
> on
>
> several projects primarily with JAVA programming language.
>
> I do not have any prior open source contributions. I have just started
>
> to
>
> explore open-source projects and I found this project very
>
> interesting. I
>
> have gone through the website as well as the confluence page and I feel
> that I can surely contribute well to this project with my skills and
> capabilities.
>
> I have subscribed to the dev and commits mailing lists and I look
>
> forward
>
> to hearing from you about how I can contribute to the project.
>
> Thanks and regards
> Vineet Tiwari
>
>
>

Re: Intend to contribute to Streampipes

Posted by Patrick Wiener <wi...@fzi.de>.
Hi Vineet,

nice to hear that you got it running.

Regarding your questions:

for 1) 
the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle that contains everything.
It basically is made of connect-adapter + pe-all-jvm. This is kinda experimental as we ship
things in Docker images and as part of this option, we can reduce the overall system load in
a single node setup due to less resource consumption for multiple JVM instances inside separate
containers.

However, for developing I’d suggest you start the adapters and pe-all-jvm separately. 

for 2)
the connect-adapters service does the actual data connection to external data sources, e.g. kafka, mqtt,
PLC as part of StreamPipes connect. Hence, this service runs the actually configured adapter 
instances in separate threads. Apart from that, in the setup/registration phase, the connect-adapter service
tries to register at the backend and provides an adapter description which contains all the information
such as what kind of protocols etc are supported by that service. Which is stored in Couchdb.
At runtime, once you browse to the UI and got to the Connect adapter market place, the backend
queries and endpoint of the connect-adapter service to check if is still actively running and besides retrieves
the dedicated icons.

Hope this helps.

Patrick



> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <vi...@gmail.com>:
> 
> Hey Patrick,
> Thanks for the clarifications. I was able to get things up and running in
> my machine. I have two more questions:
> 
> 1. As you mentioned for the initialization of extensions, there are various
> projects to bundle them. What I gathered from the main class of these
> projects, I can see that extensions-all-jvm contains all elements (data
> set, stream, processor and sink). The others bundles contain subsets of
> these elements (For example: pipeline-elements-all-jvm contains all
> elements except data streams and data sets). To run the whole application,
> I need to run the extensions-all-jvm so that all the pipeline elements are
> initialized. Is my understanding correct?
> 2. When I navigate to the connect page of the application in UI, I see that
> backend tries to get icons for the elements in UI. To successfully get the
> icons, I need to run the connect-adapters. What exactly is the role of
> connect-adapters here apart from icons fetching.
> 
> Thanks,
> Vineet Tiwari
> 
> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <wiener@apache.org <ma...@apache.org>> wrote:
> 
>> Hi Vineet,
>> 
>> There are a few points here that will hopefully help you to get your dev
>> setup running.
>> 
>> Env variables
>> We do recommend using IntelliJ as your IDE as we
>> provide IntelliJ specific run configurations as part of the Github
>> projects that will configure
>> the services with appropriate env variables, e.g. for the core [1] and
>> backend [2]. However,
>> you can also run from Eclipse but you need to add these env variables
>> manually to Eclipse.
>> 
>> General Git project setup
>> We currently differentiate between core and extensions git projects. The
>> core contains core
>> functionality for pipeline management, connect master, serialization,
>> messaging wrapper,
>> runtime wrapper etc. and two services that are also part of StreamPipes:
>> 
>> - backend service (contains core functionality, e.g. pipeline management,
>> connect master)
>> - ui service
>> 
>> It is worthy to note that we integrated the connect master service (was a
>> standalone service before)
>> in the backend service to simplify the overall architecture. This is why
>> you did not find a main
>> class for it anymore on the current dev branch.
>> 
>> Besides, there exists the extensions project that contains all concrete
>> connect adapters as well
>> as pipeline elements. We can bundle them in various ways, e.g.
>> 
>> - by type (streampipes-processors-filters-jvm)
>> - by wrapper (streampipes-pipeline-elements-all-jvm)
>> - all extensions by wrapper (streampipes-extensions-all-jvm)
>> 
>> Dev environment
>> We recommend checking out both projects and integrating them in a
>> multi-project structure
>> in your IDE.
>> 
>> StreamPipes relies on a few external services, e.g. Consul, Couchdb,
>> Kafka. To easify the
>> development please use the StreamPipes CLI option to start your dev
>> environment [3].
>> 
>> Here, there are various environments configurations depending on wether
>> you want to develop:
>> 
>> - core only -> backend
>> - extensions only -> pipeline-element
>> - core + extensions -> basic
>> 
>> Go through the provided README file to see how to use the CLI. Briefly, it
>> would be something
>> like this for starting a docker dev environment with necessary services:
>> 
>> ./streampipes env --set basic
>> ./streampipes pull
>> ./streampipes up -d
>> 
>> In your IDE, set the the env variables for the services accordingly either
>> using the provided
>> run configs for IntelliJ or adding env variables for the StreamPipes to
>> e.g. Eclipse by using
>> the IntelliJ run configs as a template.
>> 
>> We store this configs in Consul and do currently not overwrite them once a
>> service has registered.
>> You can either manually delete them from the Consul UI (localhost:8500) or
>> simply start from
>> a clean setup if you have a misconfiguration:
>> 
>> ./streampipes clean -v
>> 
>> NOTE!
>> If non variables are present, all services use default variables that
>> represent the prod settings
>> when starting the containerized environment. Thus it is important to set
>> these variables
>> according to your setup.
>> 
>> Networking in dev setup local machine <—> Docker
>> This mixed setup: dev in IDE + Dockerized environment causes some network
>> problems since
>> use Consul (running in a container) to conduct service health checks to a
>> service running on
>> your local machine (in your IDE). Thus, we need an IP/DNS that is
>> reachable/resolvable from
>> inside the Docker container.
>> 
>> Here it gets a little tricky as Docker currently does not provide a common
>> solution for all OS.
>> For instance, for the streampipes-pipeline-elements-all-jvm you will see
>> in the run config that
>> it sets an env variable as following [4]:
>> 
>> <env name="SP_HOST" value="host.docker.internal" />
>> 
>> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically,
>> host.docker.internal
>> is a Docker resolvable address on MacOS and Windows [5]
>> 
>> For Linux:
>> Whenever you see and env variable that uses „host.docker.internal“ as a
>> value, you need to
>> add the IP of you local machine
>> 
>> ——
>> 
>> Regarding your questions:
>> 
>>> 1. I was trying to start the backend module, which started fine, but when
>>> executing from UI, it is not able to connect to consul and mq due to host
>>> not found. I think I have not set the env variables correctly. I tried
>>> setting them in eclipse, but the host values are being taken from the
>>> consul key value store. How to change that to localhost?
>> Related due to missing env variables when starting the backend main class.
>> I assume, it’s trying to connect to something like consul:8500 instead of
>> localhost:8500.
>> Try with a clean setup, i.e., remove all docker volumes using the clean
>> command
>> above and setting the variables beforehand.
>> 
>> 
>>> 2. There are instructions to setup master container on the confluence,
>> but
>>> I could not find its main class.
>> This is most likely outdated, as explained above we recently integrated
>> connect
>> master into the backend. This is why you didn’t find a main class for the
>> connect master.
>> 
>> 
>>> 3. Out of all the containers in docker, I am not sure of the function and
>>> code inside these two containers - pipeline-element-all and
>> connect-worker.
>>> Also, once started, the connect-worker keeps on trying to connect to the
>>> backend api on http://backend:8030 by default. How do I change it to
>>> localhost?
>> For the first part, I assume with function and code you mean where to find
>> it?
>> As said, we provide various bundles for pipeline elements.
>> 
>> pipeline-element-all-jvm bundles all pipeline element modules of the
>> extensions
>> projects that are written for the plain Java wrapper (jvm) [6] (see the
>> main class).
>> 
>> connect-worker bundles all adapters [7] (see the main class).
>> 
>> For the second part, this is mostly related again to missing env variables
>> when starting
>> the service. „backend“ is the default name of the docker service in prod.
>> Take a look
>> at the run configs for the env variables [8]. There you’ll find the
>> following entry:
>> 
>> <env name="SP_BACKEND_HOST" value="localhost" />
>> 
>> These env variables should be set in your IDE when starting your service.
>> 
>> 
>> Hope this guide was helpful.
>> 
>> We are currently restructuring the documentation, thus in the future we
>> will add description as above
>> right in the documentation.
>> 
>> 
>> Patrick
>> 
>> 
>> [1]
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations
>> <
>> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations <https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations>
>>> 
>> [2]
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations <https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations>
>> <
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations <https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations>>
>> 
>> [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>
>> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>>
>> [4]
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml <https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml>
>> <
>> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml <https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml>>
>> 
>> [5] https://docs.docker.com/docker-for-windows/networking/ <https://docs.docker.com/docker-for-windows/networking/> <
>> https://docs.docker.com/docker-for-windows/networking/ <https://docs.docker.com/docker-for-windows/networking/>>
>> [6]
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm>
>> <
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm>>
>> 
>> [7]
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters>
>> <
>> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters>>
>> 
>> [8]
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml <https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml>
>> <
>> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml <https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml>>
>> 
>> 
>>> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <vi...@gmail.com>:
>>> 
>>> Hello,
>>> I have gone through some of the materials given above (thanks for that)
>> and
>>> I have got a good idea on the high level architecture and design of the
>>> application.
>>> 
>>> I started to set up streampipes on my local machine as well.
>>> I have the below queries:
>>> 1. I was trying to start the backend module, which started fine, but when
>>> executing from UI, it is not able to connect to consul and mq due to host
>>> not found. I think I have not set the env variables correctly. I tried
>>> setting them in eclipse, but the host values are being taken from the
>>> consul key value store. How to change that to localhost?
>>> 2. There are instructions to setup master container on the confluence,
>> but
>>> I could not find its main class.
>>> 3. Out of all the containers in docker, I am not sure of the function and
>>> code inside these two containers - pipeline-element-all and
>> connect-worker.
>>> Also, once started, the connect-worker keeps on trying to connect to the
>>> backend api on http://backend:8030 by default. How do I change it to
>>> localhost?
>>> 
>>> Thanks,
>>> Vineet Tiwari
>>> 
>>> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <ri...@apache.org>
>> wrote:
>>> 
>>>> Hi Vineet,
>>>> 
>>>> welcome!
>>>> If you have any questions on how to get started, we'll be glad to help.
>>>> 
>>>> Doominik
>>>> 
>>>> On 2021/02/23 06:41:06, Vineet Tiwari <vi...@gmail.com> wrote:
>>>>> Hello,
>>>>> My name is Vineet Tiwari and I am working as a full time software
>>>>> developer. I am pretty good at Java and javascript (mostly with
>> ReactJs)
>>>>> languages along with basic C++ and Python.
>>>>> I have a total of three and a half years of experience working as a SDE
>>>> on
>>>>> several projects primarily with JAVA programming language.
>>>>> 
>>>>> I do not have any prior open source contributions. I have just started
>> to
>>>>> explore open-source projects and I found this project very
>> interesting. I
>>>>> have gone through the website as well as the confluence page and I feel
>>>>> that I can surely contribute well to this project with my skills and
>>>>> capabilities.
>>>>> 
>>>>> I have subscribed to the dev and commits mailing lists and I look
>> forward
>>>>> to hearing from you about how I can contribute to the project.
>>>>> 
>>>>> Thanks and regards
>>>>> Vineet Tiwari