You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by GitBox <gi...@apache.org> on 2023/05/08 10:47:58 UTC
[GitHub] [streampipes] RobertIndie created a discussion: Add support for Apache Pulsar as a message transport layer
GitHub user RobertIndie created a discussion: Add support for Apache Pulsar as a message transport layer
Currently, Apache StreamPipes support 4 message protocols as the message transport layer: `jms`, `kafka`, `mqtt` and `nats`. Apache Pulsar is also a commonly used messaging protocol. I propose to add support for the Apache Pulsar.
But when I looked into the code, I found it was more complex than just adding a module. We also need to add special handling logic elsewhere. Such as https://github.com/apache/streampipes/blob/e271aa245670417e68315b5e262b76a71b92eaf8/streampipes-pipeline-management/src/main/java/org/apache/streampipes/manager/matching/DataSetGroundingSelector.java#L55-L83
I am wondering if there is a relevant guide or example PR that could help me implement this.
GitHub link: https://github.com/apache/streampipes/discussions/1561
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Add support for Apache Pulsar as a message transport layer (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
The pipeline owns a copy: It includes a list of data processor/sink invocations which include all user-defined parameters and are created based on the description at the time of pipeline creation.
We could exclude the protocol from the model and assign the communication protocol based on the prioritized protocol at pipeline startup time. The drawback is that compatibility checking of protocols is lost (currently, protocol compatibility between pipeline elements is checked and a protocol is selected based on all matching protocols and the messaging protocol priority assigned in the UI).
On the other hand, it would certainly reduce complexity...
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-6028574
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
GitHub user RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Update: I have created a draft PR https://github.com/apache/streampipes/pull/1576
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5917871
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by GitBox <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Hi @RobertIndie, sounds great!
There is currently a draft (huge) PR by @tenthe which removes data sets so that it should be easier to integrate a new messaging protocol.
So it should be fine to start with a new messaging module for pulsar and then add this as supportedProtocol to the SDK.
I'll check the codebase and come back with a more detailed description!
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5837215
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
It's a bit confusing currently as the model needs to be updated from the `Install Pipeline Elements` page for each installed pipeline element after registering the protocol in the `Init`classs of the module. Does this work from a plain installation or do you get the same error?
I'll also have a look at the PR!
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5982956
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
GitHub user RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Thank @bossenti . This issue has been fixed. I'm moving the PR further.
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5979999
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
GitHub user RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
I have encountered an issue here.
If I enable the pulsar messaging layer, there will be a protocol match issue when editing the pipeline:
<img width="825" alt="image" src="https://github.com/apache/streampipes/assets/16974619/00344612-7c64-4d3f-883b-16e61c135f99">
The root cause is that the expected protocol doesn't contain the `PulsarTransportProtocol` when checking the match of the protocols:
<img width="1330" alt="image" src="https://github.com/apache/streampipes/assets/16974619/d6f99acd-1e2f-49a9-b12a-389907be604e">
After I dig into the code, I find that this `requirement` is passed from the UI:
<img width="743" alt="image" src="https://github.com/apache/streampipes/assets/16974619/f7b6186d-af81-4b1b-9d41-ad895c859737">
The `supportedGrounding.transportProtocols` doesn't include the Pulsar. I think I must have missed setting it somewhere.
Where can I add the pulsar to the `supportedGrounding.transportProtocols`?
Could someone please take a look here?
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5956097
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
cc https://github.com/apache/streampipes/discussions/1584
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5971117
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
GitHub user RobertIndie added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
> Does this work from a plain installation or do you get the same error?
I reinstall the elements and it works. It's indeed a little confusing that the element wouldn't update automatically. Is it a point that we can improve?
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-5989826
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Yes, that's definitely something we can improve. But I'm not sure how a good solution could look like. Since the description of pipeline elements is stored in the core, automatically applying changes to pipeline element configurations could break existing pipelines (e.g., when removing a configuration or so).
Woudl you like to automatically update all pipeline elements upon registration or do you have any other idea how this should work?
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-6013353
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Add support for Apache Pulsar as a message transport layer (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Add support for Apache Pulsar as a message transport layer
Does an instantiated pipeline still rely on the centrally managed descriptions or does it "own" a copy?
GitHub link: https://github.com/apache/streampipes/discussions/1561#discussioncomment-6028344
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org