You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Philipp Zehnder <ze...@apache.org> on 2021/10/02 07:23:00 UTC
[DISCUSS] Split StreamPipes Connect worker
Hi all,
please have warm welcome for Hrushikesh.
He is new to open source and has signed up to work on Issue [1].
The idea of the issue is to split up our adapters, with the goal to have a dedicated service for IIoT data sources.
I’d suggest to start with two different modules:
* streampipes-connect-adapters
* GdeltAdapter
* CoindeskBitcoinAdapter
* IexCloudNewsAdapter
* IexCloudStockAdapter
* RandomDataSetAdapter
* RandomDataStreamAdapter
* SlackAdapter
* WikipediaEditedArticlesAdapter
* WikipediaNewArticlesAdapter
* InfluxDbStreamAdapter
* InfluxDbSetAdapter
* NetioRestAdapter
* NetioMQTTAdapter
* ImageStreamAdapter
* ImageSetAdapter
* IssAdapter
* FlicMQTTAdapter
* HDFSProtocol
* PulsarProtocol
* streampipes-connect-adapters-iiot
* MySqlStreamAdapter
* MySqlSetAdapter
* MachineDataStreamAdapter
* RosBridgeAdapter
* OpcUaAdapter
* Plc4xS7Adapter
* Plc4xModbusAdapter
* FileProtocol
* HttpProtocol
* FileStreamProtocol
* KafkaProtocol
* MqttProtocol
* HttpStreamProtocol
* HttpServerProtocol
I split them up according to my experience, which might be the most relevant once for IIoT. This is only a suggestion, if there are any ideas how we can change this, feel free to write them here.
What do you think about the first module name? Are there any other suggestions?
Philipp
[1] https://issues.apache.org/jira/browse/STREAMPIPES-387
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
unfortunately, the mailing list removes attachments - can you please upload the screenshot to Jira?
Then we can check the problem. In general, the log4j warning you posted should only be a warning and not prevent the system from starting, but let's check the error message.
Dominik
On 2021/10/08 07:56:44, hrushi <hr...@gmail.com> wrote:
> Hi Philipp
>
> I am getting a log4j error that prevents the service from starting. I
> have split most of the files. Only a few test files are remaining. I
> just wanted to see if the service is up and running but the log4j is
> throwing an error. I am sharing a screenshot of the error displayed.
>
> Hrushikesh
>
>
>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi Philipp
I am getting a log4j error that prevents the service from starting. I
have split most of the files. Only a few test files are remaining. I
just wanted to see if the service is up and running but the log4j is
throwing an error. I am sharing a screenshot of the error displayed.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Hrushikesh,
do you get a warning when starting the service or do you get a log4j error that prevents the service from starting?
For the utils we should find a solution where we only have one file containing the code.
To get a first version up and running it is ok to have the files in both projects. Once everything is working, my suggestion would be to move them into the streampipes-sdk module, because they ease the modeling for Kafka and MQTT data sources.
What do the others think of that?
Philipp
> On 7. Oct 2021, at 19:09, hrushi <hr...@gmail.com> wrote:
>
> Hi Philipp
>
> I get this error while running the main class. I have gone through this thread but I don't know if I need to add a |log4j.xml| file here. <https://stackoverflow.com/questions/12532339/no-appenders-could-be-found-for-loggerlog4j>
>
> log4j:WARN No appenders could be found for logger (org.jboss.logging).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info
>
>
> Also few folders like the utils module is common for both the modules (connect-adpaters && connect-adapters-iiot). I have different copies of folder in connect-adapters and connect-adapters-iiot module. Few protocol.stream files are also common for both the modules and I have 2 separate copies of those files. Is this fine or do we need to remove file duplication?
>
> Hrushikesh
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi Philipp,
Thank you for helping me out. I want to keep contributing to streampipes
currently. The dev docs are not yet ready. It is still difficult for me
to understand the project completely as the project is too big and
contains a lot of files. I will be a bit slow in grasping a few concepts
but will try understanding and learning them. I want to fix issue,
improve my java skills and also contribute to streampipes.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi Philipp
We can start working on moving the duplicate modules and moving them
into the backend project and delete the duplicate files in the
extensions projects.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Hrushikesh,
thank you very much and congratulations on your first pull request.
I included your changes and also updated the other parts of StreamPipes, such as the installer.
I think the next step would be to identify all the classes that are now required in both modules and create a list of them.
Once we have a complete list we can move them into the backend project and delete the duplicate files in the extensions project.
@all: as previously discussed in this thread I also renamed the connect-worker-main container to connect-adapter. I also updated the installer.
I am not sure if I correctly updated the Kubernetes installer. It would be great if someone could look if everything works again.
Philipp
> On 8. Oct 2021, at 14:45, Philipp Zehnder <ze...@apache.org> wrote:
>
> Since you had several different instances running on your computer it might make sense the delete everything first and try it again.
> This way we can make sure that there is nothing left in a cache or that there are no old database entries.
>
> Therefore stop everything and run “./streampipes down” & “./streampipes clean -v” and start it again.
>
> Philipp
>
>> On 8. Oct 2021, at 13:50, hrushi <hr...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am able to view all the iiot extensions in the UI. The data for each extension is being displayed. I am not able to view just the images in the UI. The images are missing. Is there any path to set specifically for the images. I moved all the required resources from stream-pipes-connect to stream-pipes-connect-iiot.
>>
>> Hrushikesh
>>
>>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Since you had several different instances running on your computer it might make sense the delete everything first and try it again.
This way we can make sure that there is nothing left in a cache or that there are no old database entries.
Therefore stop everything and run “./streampipes down” & “./streampipes clean -v” and start it again.
Philipp
> On 8. Oct 2021, at 13:50, hrushi <hr...@gmail.com> wrote:
>
> Hi,
>
> I am able to view all the iiot extensions in the UI. The data for each extension is being displayed. I am not able to view just the images in the UI. The images are missing. Is there any path to set specifically for the images. I moved all the required resources from stream-pipes-connect to stream-pipes-connect-iiot.
>
> Hrushikesh
>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi,
I am able to view all the iiot extensions in the UI. The data for each
extension is being displayed. I am not able to view just the images in
the UI. The images are missing. Is there any path to set specifically
for the images. I moved all the required resources from
stream-pipes-connect to stream-pipes-connect-iiot.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi,
once the service is started it should register at the StreamPipes backend und you should be able to start one of the adapters over the UI.
Philipp
> On 8. Oct 2021, at 11:19, hrushi <hr...@gmail.com> wrote:
>
> Hey Dominik Riemer
>
> I changed the port to 8000 and the service gets connected to the backend. How do I check if the changed I made is valid or not. If I open the browser with port 8000, I get an Error text printed. The error text is also being printed to the unchanged project (without the iiot adapters) I cloned in another directory.
>
> Hrushikesh
>
Re: RE: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hey Dominik Riemer
I changed the port to 8000 and the service gets connected to the
backend. How do I check if the changed I made is valid or not. If I open
the browser with port 8000, I get an Error text printed. The error text
is also being printed to the unchanged project (without the iiot
adapters) I cloned in another directory.
Hrushikesh
RE: [DISCUSS] Split StreamPipes Connect worker
Posted by Dominik Riemer <ri...@apache.org>.
Thanks for uploading the screenshot!
So the error comes from a port conflict (Web server failed to start. Port 8098 is already in use).
Are you already running both adapter modules (connect-worker-main and connect-worker-iiot)? Then you can try to stop one module and restart the application or you can change the port of one application from 8098 to another port.
Dominik
-----Original Message-----
From: hrushi <hr...@gmail.com>
Sent: Friday, October 8, 2021 10:29 AM
To: dev@streampipes.apache.org
Subject: Re: [DISCUSS] Split StreamPipes Connect worker
I am using Java 11 (amazon corretto)
I cloned the entire project in a separate folder and I get the same error. The service didn't start in the IDE.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
I am using Java 11 (amazon corretto)
I cloned the entire project in a separate folder and I get the same
error. The service didn't start in the IDE.
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Which Java version are you using?
Did you have this problem before you split up the module, or did the service start in your IDE?
If it worked before, I would suggest that you compare the original pom.xml file with the new one.
When the error occurred before your refactoring, we need to find the problem in you local setup.
Philipp
> On 7. Oct 2021, at 19:09, hrushi <hr...@gmail.com> wrote:
>
> Hi Philipp
>
> I get this error while running the main class. I have gone through this thread but I don't know if I need to add a |log4j.xml| file here. <https://stackoverflow.com/questions/12532339/no-appenders-could-be-found-for-loggerlog4j>
>
> log4j:WARN No appenders could be found for logger (org.jboss.logging).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info
>
>
> Also few folders like the utils module is common for both the modules (connect-adpaters && connect-adapters-iiot). I have different copies of folder in connect-adapters and connect-adapters-iiot module. Few protocol.stream files are also common for both the modules and I have 2 separate copies of those files. Is this fine or do we need to remove file duplication?
>
> Hrushikesh
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi Philipp
I get this error while running the main class. I have gone through this
thread but I don't know if I need to add a |log4j.xml| file here.
<https://stackoverflow.com/questions/12532339/no-appenders-could-be-found-for-loggerlog4j>
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info
Also few folders like the utils module is common for both the modules
(connect-adpaters && connect-adapters-iiot). I have different copies of
folder in connect-adapters and connect-adapters-iiot module. Few
protocol.stream files are also common for both the modules and I have 2
separate copies of those files. Is this fine or do we need to remove
file duplication?
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
It depends on the files. Some, like the init class must be in both packages.
For the resource files, you must sort them into the module that uses the adapter.
Also the docker files must be in both projects to build a docker image for each.
You can test your implementation by starting the module. You can start it the same as other extension modules. When you have a local StreamPipes instance running, the adapters should automatically be registered at the backend.
Philipp
> On 6. Oct 2021, at 14:28, hrushi <hr...@gmail.com> wrote:
>
> Hey Philipp ,
>
> There are few files common to both the modules. I am not sure where to place them. Do i create a new module for files common to both the modules? If so what do I name the new module? Also after splitting the module how do I check if the iiot module is working correctly or not?
>
> Hrushikesh
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hey Philipp ,
There are few files common to both the modules. I am not sure where to
place them. Do i create a new module for files common to both the
modules? If so what do I name the new module? Also after splitting the
module how do I check if the iiot module is working correctly or not?
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Hrushikesh,
super, thank you.
To push your code into the project you must provide a pull request.
Also when you have implementation specific questions we can discuss it in the pull request in GitHub.
Philipp
> On 6. Oct 2021, at 12:56, hrushi <hr...@gmail.com> wrote:
>
> Hi
>
> I am creating a new module named streampipes-connect-adapters-iiot.
>
> I will proceed according to the design stated below.
>
> streampipes-connect-adapters
> * GdeltAdapter
> * CoindeskBitcoinAdapter
> * IexCloudNewsAdapter
> * IexCloudStockAdapter
> * RandomDataSetAdapter
> * RandomDataStreamAdapter
> * SlackAdapter
> * WikipediaEditedArticlesAdapter
> * WikipediaNewArticlesAdapter
> * ImageStreamAdapter
> * ImageSetAdapter
> * IssAdapter
> * FlicMQTTAdapter
> * HDFSProtocol
>
> streampipes-connect-adapters-iiot
> * MySqlStreamAdapter
> * MySqlSetAdapter
> * MachineDataStreamAdapter
> * RosBridgeAdapter
> * OpcUaAdapter
> * Plc4xS7Adapter
> * Plc4xModbusAdapter
> * FileProtocol
> * HttpProtocol
> * FileStreamProtocol
> * KafkaProtocol
> * MqttProtocol
> * HttpStreamProtocol
> * HttpServerProtocol
> * PulsarProtocol
> * InfluxDbStreamAdapter
> * InfluxDbSetAdapter
> * NetioRestAdapter
> * NetioMQTTAdapter
> Hrushikesh
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi
I am creating a new module named streampipes-connect-adapters-iiot.
I will proceed according to the design stated below.
streampipes-connect-adapters
* GdeltAdapter
* CoindeskBitcoinAdapter
* IexCloudNewsAdapter
* IexCloudStockAdapter
* RandomDataSetAdapter
* RandomDataStreamAdapter
* SlackAdapter
* WikipediaEditedArticlesAdapter
* WikipediaNewArticlesAdapter
* ImageStreamAdapter
* ImageSetAdapter
* IssAdapter
* FlicMQTTAdapter
* HDFSProtocol
streampipes-connect-adapters-iiot
* MySqlStreamAdapter
* MySqlSetAdapter
* MachineDataStreamAdapter
* RosBridgeAdapter
* OpcUaAdapter
* Plc4xS7Adapter
* Plc4xModbusAdapter
* FileProtocol
* HttpProtocol
* FileStreamProtocol
* KafkaProtocol
* MqttProtocol
* HttpStreamProtocol
* HttpServerProtocol
* PulsarProtocol
* InfluxDbStreamAdapter
* InfluxDbSetAdapter
* NetioRestAdapter
* NetioMQTTAdapter
Hrushikesh
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
No I do not have a better idea, I only did not completely understand what you meant.
We should definitely harmonize the names.
I am in favor of naming the packages in the extensions project the same as the resulting docker containers, since this eases debugging.
As for the new packages now, is it suitable to name them so or should we use different names?
* streampipes-connect-adapters
* streampipes-connect-adapters-iiot
Philipp
> On 6. Oct 2021, at 10:37, Patrick Wiener <wi...@apache.org> wrote:
>
> Well, in general I meant the user-facing things. Hence the naming of the modules in the extensions repo with adapters - but also the
> naming of the Docker container. This is currently also referred to as „worker-main“.
>
> In a second step however, it also makes sense to apply this to the backend modules. If its better to have the notion of „worker“ somewhere
> inside the naming scheme it should be clear and in both backend and extensions.
>
> I am more in favor of having it clean and I guess adapters are more common as this is what we talk about most of the time anyways :)
>
> But maybe you have a better idea?
>
> Patrick
>
>> Am 06.10.2021 um 10:28 schrieb Philipp Zehnder <ze...@apache.org>:
>>
>> Hi Patrick,
>>
>> I totally agree, we should harmonize the naming.
>> But what are you referring to? To the naming of the modules of the connect adapters in the worker or the modules in the backend?
>>
>> My suggestion for the modules containing the adapters would be: streampipes-connect-adapters and streampipes-connect-adapters-iiot
>> Do you have an alternative suggestion?
>>
>> Philipp
>>
>>
>>> On 6. Oct 2021, at 10:15, Patrick Wiener <wi...@apache.org> wrote:
>>>
>>> Hi,
>>>
>>> fine for me. But when refactoring, would it make sense to also align the naming because
>>> it might be not to intuitive having „workers“ denoting the architectural paradigm and „adapters“
>>> for the protocols.
>>>
>>> Pipeline element services act similarly but aren’t referred to as „workers“ for pipeline elements too.
>>>
>>> What do you think about that as there is already some refactoring going on?
>>>
>>> Patrick
>>>
>>>> Am 06.10.2021 um 09:45 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>
>>>> Hi Dominik,
>>>>
>>>> that is fine with me +1
>>>>
>>>> Philipp
>>>>
>>>>> On 6. Oct 2021, at 09:43, Dominik Riemer <ri...@apache.org> wrote:
>>>>>
>>>>> Hi Philipp,
>>>>>
>>>>> I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
>>>>> module.
>>>>> Reason is that these are also "enterprise-grade" adapters that might be
>>>>> useful for IIOT applications, so that users can deploy a single module and
>>>>> find all potentially relevant adapters for their industry.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Dominik
>>>>>
>>>>> -----Original Message-----
>>>>> From: Philipp Zehnder <ze...@apache.org>
>>>>> Sent: Wednesday, October 6, 2021 8:57 AM
>>>>> To: dev@streampipes.apache.org
>>>>> Subject: Re: [DISCUSS] Split StreamPipes Connect worker
>>>>>
>>>>> Hi,
>>>>>
>>>>> the idea of the issue is to split up the package
>>>>> streampipes-connect-adapters into two packages, one containing only the
>>>>> relevant adapters for IIoT scenarios and one with others.
>>>>> So the next step would be to create a new module, add all the adapters
>>>>> there, as listed below. As a template you can use the already existing
>>>>> module.
>>>>>
>>>>>
>>>>> * streampipes-connect-adapters
>>>>> * GdeltAdapter
>>>>> * CoindeskBitcoinAdapter
>>>>> * IexCloudNewsAdapter
>>>>> * IexCloudStockAdapter
>>>>> * RandomDataSetAdapter
>>>>> * RandomDataStreamAdapter
>>>>> * SlackAdapter
>>>>> * WikipediaEditedArticlesAdapter
>>>>> * WikipediaNewArticlesAdapter
>>>>> * InfluxDbStreamAdapter
>>>>> * InfluxDbSetAdapter
>>>>> * NetioRestAdapter
>>>>> * NetioMQTTAdapter
>>>>> * ImageStreamAdapter
>>>>> * ImageSetAdapter
>>>>> * IssAdapter
>>>>> * FlicMQTTAdapter
>>>>> * HDFSProtocol
>>>>> * PulsarProtocol
>>>>>
>>>>> * streampipes-connect-adapters-iiot
>>>>> * MySqlStreamAdapter
>>>>> * MySqlSetAdapter
>>>>> * MachineDataStreamAdapter
>>>>> * RosBridgeAdapter
>>>>> * OpcUaAdapter
>>>>> * Plc4xS7Adapter
>>>>> * Plc4xModbusAdapter
>>>>> * FileProtocol
>>>>> * HttpProtocol
>>>>> * FileStreamProtocol
>>>>> * KafkaProtocol
>>>>> * MqttProtocol
>>>>> * HttpStreamProtocol
>>>>> * HttpServerProtocol
>>>>>
>>>>> Write me if something is unclear.
>>>>>
>>>>> Philipp
>>>>>
>>>>>> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I would like to contribute and fix this issue. I would like to know the
>>>>> prerequisites for this issue so that I can go through those concepts. New to
>>>>> open source and will try to contribute.
>>>>>>
>>>>>> Thank You.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Patrick Wiener <wi...@apache.org>.
Well, in general I meant the user-facing things. Hence the naming of the modules in the extensions repo with adapters - but also the
naming of the Docker container. This is currently also referred to as „worker-main“.
In a second step however, it also makes sense to apply this to the backend modules. If its better to have the notion of „worker“ somewhere
inside the naming scheme it should be clear and in both backend and extensions.
I am more in favor of having it clean and I guess adapters are more common as this is what we talk about most of the time anyways :)
But maybe you have a better idea?
Patrick
> Am 06.10.2021 um 10:28 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Hi Patrick,
>
> I totally agree, we should harmonize the naming.
> But what are you referring to? To the naming of the modules of the connect adapters in the worker or the modules in the backend?
>
> My suggestion for the modules containing the adapters would be: streampipes-connect-adapters and streampipes-connect-adapters-iiot
> Do you have an alternative suggestion?
>
> Philipp
>
>
>> On 6. Oct 2021, at 10:15, Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi,
>>
>> fine for me. But when refactoring, would it make sense to also align the naming because
>> it might be not to intuitive having „workers“ denoting the architectural paradigm and „adapters“
>> for the protocols.
>>
>> Pipeline element services act similarly but aren’t referred to as „workers“ for pipeline elements too.
>>
>> What do you think about that as there is already some refactoring going on?
>>
>> Patrick
>>
>>> Am 06.10.2021 um 09:45 schrieb Philipp Zehnder <ze...@apache.org>:
>>>
>>> Hi Dominik,
>>>
>>> that is fine with me +1
>>>
>>> Philipp
>>>
>>>> On 6. Oct 2021, at 09:43, Dominik Riemer <ri...@apache.org> wrote:
>>>>
>>>> Hi Philipp,
>>>>
>>>> I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
>>>> module.
>>>> Reason is that these are also "enterprise-grade" adapters that might be
>>>> useful for IIOT applications, so that users can deploy a single module and
>>>> find all potentially relevant adapters for their industry.
>>>>
>>>> What do you think?
>>>>
>>>> Dominik
>>>>
>>>> -----Original Message-----
>>>> From: Philipp Zehnder <ze...@apache.org>
>>>> Sent: Wednesday, October 6, 2021 8:57 AM
>>>> To: dev@streampipes.apache.org
>>>> Subject: Re: [DISCUSS] Split StreamPipes Connect worker
>>>>
>>>> Hi,
>>>>
>>>> the idea of the issue is to split up the package
>>>> streampipes-connect-adapters into two packages, one containing only the
>>>> relevant adapters for IIoT scenarios and one with others.
>>>> So the next step would be to create a new module, add all the adapters
>>>> there, as listed below. As a template you can use the already existing
>>>> module.
>>>>
>>>>
>>>> * streampipes-connect-adapters
>>>> * GdeltAdapter
>>>> * CoindeskBitcoinAdapter
>>>> * IexCloudNewsAdapter
>>>> * IexCloudStockAdapter
>>>> * RandomDataSetAdapter
>>>> * RandomDataStreamAdapter
>>>> * SlackAdapter
>>>> * WikipediaEditedArticlesAdapter
>>>> * WikipediaNewArticlesAdapter
>>>> * InfluxDbStreamAdapter
>>>> * InfluxDbSetAdapter
>>>> * NetioRestAdapter
>>>> * NetioMQTTAdapter
>>>> * ImageStreamAdapter
>>>> * ImageSetAdapter
>>>> * IssAdapter
>>>> * FlicMQTTAdapter
>>>> * HDFSProtocol
>>>> * PulsarProtocol
>>>>
>>>> * streampipes-connect-adapters-iiot
>>>> * MySqlStreamAdapter
>>>> * MySqlSetAdapter
>>>> * MachineDataStreamAdapter
>>>> * RosBridgeAdapter
>>>> * OpcUaAdapter
>>>> * Plc4xS7Adapter
>>>> * Plc4xModbusAdapter
>>>> * FileProtocol
>>>> * HttpProtocol
>>>> * FileStreamProtocol
>>>> * KafkaProtocol
>>>> * MqttProtocol
>>>> * HttpStreamProtocol
>>>> * HttpServerProtocol
>>>>
>>>> Write me if something is unclear.
>>>>
>>>> Philipp
>>>>
>>>>> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> I would like to contribute and fix this issue. I would like to know the
>>>> prerequisites for this issue so that I can go through those concepts. New to
>>>> open source and will try to contribute.
>>>>>
>>>>> Thank You.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Patrick,
I totally agree, we should harmonize the naming.
But what are you referring to? To the naming of the modules of the connect adapters in the worker or the modules in the backend?
My suggestion for the modules containing the adapters would be: streampipes-connect-adapters and streampipes-connect-adapters-iiot
Do you have an alternative suggestion?
Philipp
> On 6. Oct 2021, at 10:15, Patrick Wiener <wi...@apache.org> wrote:
>
> Hi,
>
> fine for me. But when refactoring, would it make sense to also align the naming because
> it might be not to intuitive having „workers“ denoting the architectural paradigm and „adapters“
> for the protocols.
>
> Pipeline element services act similarly but aren’t referred to as „workers“ for pipeline elements too.
>
> What do you think about that as there is already some refactoring going on?
>
> Patrick
>
>> Am 06.10.2021 um 09:45 schrieb Philipp Zehnder <ze...@apache.org>:
>>
>> Hi Dominik,
>>
>> that is fine with me +1
>>
>> Philipp
>>
>>> On 6. Oct 2021, at 09:43, Dominik Riemer <ri...@apache.org> wrote:
>>>
>>> Hi Philipp,
>>>
>>> I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
>>> module.
>>> Reason is that these are also "enterprise-grade" adapters that might be
>>> useful for IIOT applications, so that users can deploy a single module and
>>> find all potentially relevant adapters for their industry.
>>>
>>> What do you think?
>>>
>>> Dominik
>>>
>>> -----Original Message-----
>>> From: Philipp Zehnder <ze...@apache.org>
>>> Sent: Wednesday, October 6, 2021 8:57 AM
>>> To: dev@streampipes.apache.org
>>> Subject: Re: [DISCUSS] Split StreamPipes Connect worker
>>>
>>> Hi,
>>>
>>> the idea of the issue is to split up the package
>>> streampipes-connect-adapters into two packages, one containing only the
>>> relevant adapters for IIoT scenarios and one with others.
>>> So the next step would be to create a new module, add all the adapters
>>> there, as listed below. As a template you can use the already existing
>>> module.
>>>
>>>
>>> * streampipes-connect-adapters
>>> * GdeltAdapter
>>> * CoindeskBitcoinAdapter
>>> * IexCloudNewsAdapter
>>> * IexCloudStockAdapter
>>> * RandomDataSetAdapter
>>> * RandomDataStreamAdapter
>>> * SlackAdapter
>>> * WikipediaEditedArticlesAdapter
>>> * WikipediaNewArticlesAdapter
>>> * InfluxDbStreamAdapter
>>> * InfluxDbSetAdapter
>>> * NetioRestAdapter
>>> * NetioMQTTAdapter
>>> * ImageStreamAdapter
>>> * ImageSetAdapter
>>> * IssAdapter
>>> * FlicMQTTAdapter
>>> * HDFSProtocol
>>> * PulsarProtocol
>>>
>>> * streampipes-connect-adapters-iiot
>>> * MySqlStreamAdapter
>>> * MySqlSetAdapter
>>> * MachineDataStreamAdapter
>>> * RosBridgeAdapter
>>> * OpcUaAdapter
>>> * Plc4xS7Adapter
>>> * Plc4xModbusAdapter
>>> * FileProtocol
>>> * HttpProtocol
>>> * FileStreamProtocol
>>> * KafkaProtocol
>>> * MqttProtocol
>>> * HttpStreamProtocol
>>> * HttpServerProtocol
>>>
>>> Write me if something is unclear.
>>>
>>> Philipp
>>>
>>>> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>>>>
>>>> Hi
>>>>
>>>> I would like to contribute and fix this issue. I would like to know the
>>> prerequisites for this issue so that I can go through those concepts. New to
>>> open source and will try to contribute.
>>>>
>>>> Thank You.
>>>>
>>>
>>>
>>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Patrick Wiener <wi...@apache.org>.
Hi,
fine for me. But when refactoring, would it make sense to also align the naming because
it might be not to intuitive having „workers“ denoting the architectural paradigm and „adapters“
for the protocols.
Pipeline element services act similarly but aren’t referred to as „workers“ for pipeline elements too.
What do you think about that as there is already some refactoring going on?
Patrick
> Am 06.10.2021 um 09:45 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Hi Dominik,
>
> that is fine with me +1
>
> Philipp
>
>> On 6. Oct 2021, at 09:43, Dominik Riemer <ri...@apache.org> wrote:
>>
>> Hi Philipp,
>>
>> I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
>> module.
>> Reason is that these are also "enterprise-grade" adapters that might be
>> useful for IIOT applications, so that users can deploy a single module and
>> find all potentially relevant adapters for their industry.
>>
>> What do you think?
>>
>> Dominik
>>
>> -----Original Message-----
>> From: Philipp Zehnder <ze...@apache.org>
>> Sent: Wednesday, October 6, 2021 8:57 AM
>> To: dev@streampipes.apache.org
>> Subject: Re: [DISCUSS] Split StreamPipes Connect worker
>>
>> Hi,
>>
>> the idea of the issue is to split up the package
>> streampipes-connect-adapters into two packages, one containing only the
>> relevant adapters for IIoT scenarios and one with others.
>> So the next step would be to create a new module, add all the adapters
>> there, as listed below. As a template you can use the already existing
>> module.
>>
>>
>> * streampipes-connect-adapters
>> * GdeltAdapter
>> * CoindeskBitcoinAdapter
>> * IexCloudNewsAdapter
>> * IexCloudStockAdapter
>> * RandomDataSetAdapter
>> * RandomDataStreamAdapter
>> * SlackAdapter
>> * WikipediaEditedArticlesAdapter
>> * WikipediaNewArticlesAdapter
>> * InfluxDbStreamAdapter
>> * InfluxDbSetAdapter
>> * NetioRestAdapter
>> * NetioMQTTAdapter
>> * ImageStreamAdapter
>> * ImageSetAdapter
>> * IssAdapter
>> * FlicMQTTAdapter
>> * HDFSProtocol
>> * PulsarProtocol
>>
>> * streampipes-connect-adapters-iiot
>> * MySqlStreamAdapter
>> * MySqlSetAdapter
>> * MachineDataStreamAdapter
>> * RosBridgeAdapter
>> * OpcUaAdapter
>> * Plc4xS7Adapter
>> * Plc4xModbusAdapter
>> * FileProtocol
>> * HttpProtocol
>> * FileStreamProtocol
>> * KafkaProtocol
>> * MqttProtocol
>> * HttpStreamProtocol
>> * HttpServerProtocol
>>
>> Write me if something is unclear.
>>
>> Philipp
>>
>>> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>>>
>>> Hi
>>>
>>> I would like to contribute and fix this issue. I would like to know the
>> prerequisites for this issue so that I can go through those concepts. New to
>> open source and will try to contribute.
>>>
>>> Thank You.
>>>
>>
>>
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Dominik,
that is fine with me +1
Philipp
> On 6. Oct 2021, at 09:43, Dominik Riemer <ri...@apache.org> wrote:
>
> Hi Philipp,
>
> I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
> module.
> Reason is that these are also "enterprise-grade" adapters that might be
> useful for IIOT applications, so that users can deploy a single module and
> find all potentially relevant adapters for their industry.
>
> What do you think?
>
> Dominik
>
> -----Original Message-----
> From: Philipp Zehnder <ze...@apache.org>
> Sent: Wednesday, October 6, 2021 8:57 AM
> To: dev@streampipes.apache.org
> Subject: Re: [DISCUSS] Split StreamPipes Connect worker
>
> Hi,
>
> the idea of the issue is to split up the package
> streampipes-connect-adapters into two packages, one containing only the
> relevant adapters for IIoT scenarios and one with others.
> So the next step would be to create a new module, add all the adapters
> there, as listed below. As a template you can use the already existing
> module.
>
>
> * streampipes-connect-adapters
> * GdeltAdapter
> * CoindeskBitcoinAdapter
> * IexCloudNewsAdapter
> * IexCloudStockAdapter
> * RandomDataSetAdapter
> * RandomDataStreamAdapter
> * SlackAdapter
> * WikipediaEditedArticlesAdapter
> * WikipediaNewArticlesAdapter
> * InfluxDbStreamAdapter
> * InfluxDbSetAdapter
> * NetioRestAdapter
> * NetioMQTTAdapter
> * ImageStreamAdapter
> * ImageSetAdapter
> * IssAdapter
> * FlicMQTTAdapter
> * HDFSProtocol
> * PulsarProtocol
>
> * streampipes-connect-adapters-iiot
> * MySqlStreamAdapter
> * MySqlSetAdapter
> * MachineDataStreamAdapter
> * RosBridgeAdapter
> * OpcUaAdapter
> * Plc4xS7Adapter
> * Plc4xModbusAdapter
> * FileProtocol
> * HttpProtocol
> * FileStreamProtocol
> * KafkaProtocol
> * MqttProtocol
> * HttpStreamProtocol
> * HttpServerProtocol
>
> Write me if something is unclear.
>
> Philipp
>
>> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>>
>> Hi
>>
>> I would like to contribute and fix this issue. I would like to know the
> prerequisites for this issue so that I can go through those concepts. New to
> open source and will try to contribute.
>>
>> Thank You.
>>
>
>
RE: [DISCUSS] Split StreamPipes Connect worker
Posted by Dominik Riemer <ri...@apache.org>.
Hi Philipp,
I'd propose to also move the InfluxDB, Netio and Pulsar adapters to the iiot
module.
Reason is that these are also "enterprise-grade" adapters that might be
useful for IIOT applications, so that users can deploy a single module and
find all potentially relevant adapters for their industry.
What do you think?
Dominik
-----Original Message-----
From: Philipp Zehnder <ze...@apache.org>
Sent: Wednesday, October 6, 2021 8:57 AM
To: dev@streampipes.apache.org
Subject: Re: [DISCUSS] Split StreamPipes Connect worker
Hi,
the idea of the issue is to split up the package
streampipes-connect-adapters into two packages, one containing only the
relevant adapters for IIoT scenarios and one with others.
So the next step would be to create a new module, add all the adapters
there, as listed below. As a template you can use the already existing
module.
* streampipes-connect-adapters
* GdeltAdapter
* CoindeskBitcoinAdapter
* IexCloudNewsAdapter
* IexCloudStockAdapter
* RandomDataSetAdapter
* RandomDataStreamAdapter
* SlackAdapter
* WikipediaEditedArticlesAdapter
* WikipediaNewArticlesAdapter
* InfluxDbStreamAdapter
* InfluxDbSetAdapter
* NetioRestAdapter
* NetioMQTTAdapter
* ImageStreamAdapter
* ImageSetAdapter
* IssAdapter
* FlicMQTTAdapter
* HDFSProtocol
* PulsarProtocol
* streampipes-connect-adapters-iiot
* MySqlStreamAdapter
* MySqlSetAdapter
* MachineDataStreamAdapter
* RosBridgeAdapter
* OpcUaAdapter
* Plc4xS7Adapter
* Plc4xModbusAdapter
* FileProtocol
* HttpProtocol
* FileStreamProtocol
* KafkaProtocol
* MqttProtocol
* HttpStreamProtocol
* HttpServerProtocol
Write me if something is unclear.
Philipp
> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>
> Hi
>
> I would like to contribute and fix this issue. I would like to know the
prerequisites for this issue so that I can go through those concepts. New to
open source and will try to contribute.
>
> Thank You.
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by Philipp Zehnder <ze...@apache.org>.
Hi,
the idea of the issue is to split up the package streampipes-connect-adapters into two packages, one containing only the relevant adapters for IIoT scenarios and one with others.
So the next step would be to create a new module, add all the adapters there, as listed below. As a template you can use the already existing module.
* streampipes-connect-adapters
* GdeltAdapter
* CoindeskBitcoinAdapter
* IexCloudNewsAdapter
* IexCloudStockAdapter
* RandomDataSetAdapter
* RandomDataStreamAdapter
* SlackAdapter
* WikipediaEditedArticlesAdapter
* WikipediaNewArticlesAdapter
* InfluxDbStreamAdapter
* InfluxDbSetAdapter
* NetioRestAdapter
* NetioMQTTAdapter
* ImageStreamAdapter
* ImageSetAdapter
* IssAdapter
* FlicMQTTAdapter
* HDFSProtocol
* PulsarProtocol
* streampipes-connect-adapters-iiot
* MySqlStreamAdapter
* MySqlSetAdapter
* MachineDataStreamAdapter
* RosBridgeAdapter
* OpcUaAdapter
* Plc4xS7Adapter
* Plc4xModbusAdapter
* FileProtocol
* HttpProtocol
* FileStreamProtocol
* KafkaProtocol
* MqttProtocol
* HttpStreamProtocol
* HttpServerProtocol
Write me if something is unclear.
Philipp
> On 5. Oct 2021, at 15:37, hrushi <hr...@gmail.com> wrote:
>
> Hi
>
> I would like to contribute and fix this issue. I would like to know the prerequisites for this issue so that I can go through those concepts. New to open source and will try to contribute.
>
> Thank You.
>
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi
I would like to contribute and fix this issue. I would like to know the
prerequisites for this issue so that I can go through those concepts.
New to open source and will try to contribute.
Thank You.
Re: [DISCUSS] Split StreamPipes Connect worker
Posted by hrushi <hr...@gmail.com>.
Hi,
I have the incubator-streampipes-extensions up and running. I am not
able to view any external pipelines for installation. In the connect
market place all the data is being loaded from
*incubator-streampipes-installer *and not from the extensions.
How do I proceed with this?
Thank You.
On 2021/10/02 07:23:00, Philipp Zehnder <z....@apache.org> wrote:
> Hi all,>
>
> please have warm welcome for Hrushikesh.>
> He is new to open source and has signed up to work on Issue [1].>
>
> The idea of the issue is to split up our adapters, with the goal to
have a dedicated service for IIoT data sources.>
> I’d suggest to start with two different modules:>
>
> * streampipes-connect-adapters>
> * GdeltAdapter>
> * CoindeskBitcoinAdapter>
> * IexCloudNewsAdapter>
> * IexCloudStockAdapter>
> * RandomDataSetAdapter>
> * RandomDataStreamAdapter>
> * SlackAdapter>
> * WikipediaEditedArticlesAdapter>
> * WikipediaNewArticlesAdapter>
> * InfluxDbStreamAdapter>
> * InfluxDbSetAdapter>
> * NetioRestAdapter>
> * NetioMQTTAdapter>
> * ImageStreamAdapter>
> * ImageSetAdapter>
> * IssAdapter>
> * FlicMQTTAdapter>
> * HDFSProtocol>
> * PulsarProtocol>
>
> * streampipes-connect-adapters-iiot>
> * MySqlStreamAdapter>
> * MySqlSetAdapter>
> * MachineDataStreamAdapter>
> * RosBridgeAdapter>
> * OpcUaAdapter>
> * Plc4xS7Adapter>
> * Plc4xModbusAdapter>
> * FileProtocol>
> * HttpProtocol>
> * FileStreamProtocol>
> * KafkaProtocol>
> * MqttProtocol>
> * HttpStreamProtocol>
> * HttpServerProtocol>
>
> I split them up according to my experience, which might be the most
relevant once for IIoT. This is only a suggestion, if there are any
ideas how we can change this, feel free to write them here.>
> What do you think about the first module name? Are there any other
suggestions?>
>
> Philipp>
>
> [1] https://issues.apache.org/jira/browse/STREAMPIPES-387>