You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "dominikriemer (via GitHub)" <gi...@apache.org> on 2023/05/30 15:58:04 UTC
[D] Anyone using the Flink wrapper? (streampipes)
GitHub user dominikriemer created a discussion: Anyone using the Flink wrapper?
Hi,
in `streampipes-extensions`, we have a few modules that provide pipeline elements using the Apache Flink wrapper. This was a very protoypical implementation we haven't touched for a while. The main drawback is that pipeline elements need to be explicitely developed for Flink and cannot run in the default standalone mode.
While refactoring the API, I discovered that these pipeline elements are not supposed to work anymore since we switched to Spring Boot 3 which requires Java 17, while Flink supports Java 11 as maximum version.
So I guess nobody is using Flink components? In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards. We can still keep the current implementation of the Flink runtime in case somebody wants to work on this.
What do you think?
GitHub link: https://github.com/apache/streampipes/discussions/1639
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Anyone using the Flink wrapper?
Do we want to collect feedback on this from `users@streampipes...` as well?
> In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards.
What do you mean hereby? How would such a migration look like?
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6040437
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "obermeier (via GitHub)" <gi...@apache.org>.
GitHub user obermeier deleted a comment on the discussion: Anyone using the Flink wrapper?
I also think that the existing modules are better located in a "normal" module since they do not use many flink features.
But I would not give up the Flink integration complety (Maven-Flink-Archetype).
For me Flink is the most important integration component when it comes to aggregations.
Especialy for window operations.
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6050089
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "obermeier (via GitHub)" <gi...@apache.org>.
GitHub user obermeier added a comment to the discussion: Anyone using the Flink wrapper?
Hi @dominikriemer ,
thank you!
I use Flink for some aggregations e.g for visualistation of facility [data](https://www.db-fda.s-node.de/data-overview),
clculation KPIs like the Mean Time Between Failures ...
I currenty use Java 8/11 for this jobs and hope that Flike will support Java 17 in the future ;)
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6056957
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Anyone using the Flink wrapper?
Hi @obermeier thanks for the feedback! I'd definitely leave the Flink wrapper and archetype in the codebase so that someone can pick up development in case we find someone who's interested.
Just out of curiosity, how do you currently use Flink components? I didn't manage to get them working with Java 17. If you are interested, could you have a look at the current implementation in the #1632 branch and see if it works since I wasn't even able to start this in the minicluster?
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6051271
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Anyone using the Flink wrapper?
There are few pipeline elements such as the `MeasurementUnitConverter` where we could just port the application logic to the default way of implementing pipeline elements using the standalone wrapper. Another one would be the Elasticsearch sink, which also is only available for Flink at this time.
So migration in this sense is just porting the code ;-)
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6040466
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Anyone using the Flink wrapper?
@obermeier FDA looks like a cool project ;-)
So I'll create an issue to migrate a few Flink processors to the default runtime and then integrate the remaining Flink modules into a single new module `streampipes-pipeline-elements-experimental-flink` and leave the wrapper as-is!
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6073551
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Anyone using the Flink wrapper?
Ah okay, now I got you 👍🏼
Sounds like a good idea 🙂
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6040481
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "obermeier (via GitHub)" <gi...@apache.org>.
GitHub user obermeier added a comment to the discussion: Anyone using the Flink wrapper?
> Hi, in `streampipes-extensions`, we have a few modules that provide pipeline elements using the Apache Flink wrapper. This was a very protoypical implementation we haven't touched for a while. The main drawback is that pipeline elements need to be explicitely developed for Flink and cannot run in the default standalone mode.
>
> While refactoring the API, I discovered that these pipeline elements are not supposed to work anymore since we switched to Spring Boot 3 which requires Java 17, while Flink supports Java 11 as maximum version.
>
> So I guess nobody is using Flink components? In this case, I'd propose to migrate a few Flink-based pipeline elements we would like to keep to the default implementation and remove the extensions modules afterwards. We can still keep the current implementation of the Flink runtime in case somebody wants to work on this.
>
> What do you think?
I also think that the existing modules are better located in a "normal" module since they do not use many flink features.
But I would not give up the Flink integration complety (Maven-Flink-Archetype).
For me Flink is the most important integration component when it comes to aggregations.
Especialy for window operations.
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6050103
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
Re: [D] Anyone using the Flink wrapper? (streampipes)
Posted by "obermeier (via GitHub)" <gi...@apache.org>.
GitHub user obermeier added a comment to the discussion: Anyone using the Flink wrapper?
I also think that the existing modules are better located in a "normal" module since they do not use many flink features.
But I would not give up the Flink integration complety (Maven-Flink-Archetype).
For me Flink is the most important integration component when it comes to aggregations.
Especialy for window operations.
GitHub link: https://github.com/apache/streampipes/discussions/1639#discussioncomment-6050089
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org