You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by GitBox <gi...@apache.org> on 2023/02/19 14:59:28 UTC
[GitHub] [streampipes] flomickl created a discussion: Structure of Sinks
GitHub user flomickl created a discussion: Structure of Sinks
1)
in StreamPipes there are different Sink Types
https://github.com/apache/streampipes/blob/dev/streampipes-model/src/main/java/org/apache/streampipes/model/DataSinkType.java
Should we discuss if it is time to recreate these types?
2) I noticed that there is also a 1 class variant of sinks.
So it is time to refactor the "old" 3 class sinks?
3) There is a category called database flink with an Elasticsearch sink.
Would it be better so rename this and call is distributed sinks?
and add other storage like spark or Hadoop in the future?
4) Another topic on sinks?
GitHub link: https://github.com/apache/streampipes/discussions/1314
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Structure of Sinks
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Structure of Sinks
> 2. Yes, this is the same like for processors. The three-class approach is deprecated and we need to migrate all pipeline elements to the one-class approach. This is just some work to do 😀 Maybe these are suitable tasks as good-first-issues if we provide in example on how to do?
I've created the corresponding issue: #1315
GitHub link: https://github.com/apache/streampipes/discussions/1314#discussioncomment-5026405
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Structure of Sinks
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Structure of Sinks
1) Why do want to recreate them? Do you think they are outdated?
2) Yes, this is the same like for processors. The three-class approach is deprecated and we need to migrate all pipeline elements to the one-class approach. This is just some work to do 😀 Maybe these are suitable tasks as `good-first-issues` if we provide in example on how to do?
3) I think this naming is tied to the actual runtime environment, like it is for some processor elements as well.
If we want to support further runtime types like Spark then we should create dedicated modules therefore.
As far as I know, we once supported Spark already here and there, but there was close to zero usage.
Since this increases complexity and maintenance effort significantly, we should carefully think about integration additional runtime environments.
@tenthe @dominikriemer please correct me if I'm wrong here
GitHub link: https://github.com/apache/streampipes/discussions/1314#discussioncomment-5025520
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Structure of Sinks
Posted by GitBox <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Structure of Sinks
I think the question has two parts:
- The `DataSinkType` refers to the category of the data sink (there is a similar class for processors as well). This is only used in the pipeline editor to better find pipeline elements of a certain group (select `group` in the pipeline element selection panel). We can extend the list but need to be careful when removing items due to backwards compatibility.
- We have some modules with `flink` appendix which can be only used with the Flink wrapper. Anyways, we haven't maintained the Flink wrapper for quite some time and most recent pipeline elements use the default JVM wrapper. We can rework the distributed wrappers in the future, and then I'd be in favor of reusing the same set of processors for distributed and standalone wrappers instead of having separate implementations.
GitHub link: https://github.com/apache/streampipes/discussions/1314#discussioncomment-5057381
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] flomickl edited a discussion: Structure of Sinks
Posted by GitBox <gi...@apache.org>.
GitHub user flomickl edited a discussion: Structure of Sinks
1)
in StreamPipes there are different Sink Types
https://github.com/apache/streampipes/blob/dev/streampipes-model/src/main/java/org/apache/streampipes/model/DataSinkType.java
Should we discuss if it is time to recreate these types?
2) I noticed that there is also a 1 class variant of sinks.
So it is time to refactor the "old" 3 class sinks?
3) There is a category called database flink with an Elasticsearch sink.
Would it be better so rename this and call is distributed sinks?
and add other storage like spark or Hadoop in the future?
4) Another topic on sinks?
For documentation there is a new dev section https://cwiki.apache.org/confluence/x/NppbDg
GitHub link: https://github.com/apache/streampipes/discussions/1314
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org