You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aries.apache.org by Christian Schneider <ch...@die-schneider.net> on 2017/06/26 16:12:52 UTC
Prototype for reactive streams and a messaging component abstraction
I recently looked into ways to combine messaging and streaming on OSGi.
Interestingly the best streaming solution I found for my case was
Reactor (by Pivotal) which is the core of spring 5. It works out of the
box on OSGi and only has a single dependency.
The next thing was how to combine this with messaging in a loosely
coupled way. I really like Apache Camel but I think it is not up to date
any more and also acquired a lot of weight over time (especially in
camel-core). So I was looking into providing a light weight component
API and combine it with Reactor.
The result is this project:
https://github.com/cschneider/streaming-osgi/tree/master/reactortest
This is the Component API:
https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/component/api/MComponent.java
Actually I am unsure if the converter must be part of the API but this
is the current state.
I created some POC components for Mqtt, EventAdmin and Mail.
and finally two examples:
Listen on eventadmin topic, log and forward to other topic:
https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/ExampleEventAdmin.java
Listen to mqtt, compute average over sliding window and forward to other
topic:
https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/MqttExampleComponent.java
I think there is a lot of potential in Reactor and also in messaging
components that do not couple your code to the technology.
I would be happy about any feedback on the prototype. Beware the code is
not yet split into bundles but I hope the intention is still visible.
Best
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Prototype for reactive streams and a messaging component
abstraction
Posted by Christian Schneider <ch...@die-schneider.net>.
I just worked a bit more on the component and reactive streams prototype
to split it up into bundles and also make the MComponent interface a bit
simpler:
https://github.com/cschneider/streaming-osgi
In the MComponent interface I exchanged the converter with a target
type. This makes it easier to use.
public interface MComponent {
<T> Publisher<T> from(String destination, Class<? extends T> type);
<T> Subscriber<T> to(String destination, Class<? extends T> type);
}
At the moment there is no conversion. Every component has a native type
that you must use.
For example the Mqtt component uses the type byte[], EventAdmin uses
Map<String,?>.
As an enhancement we can use the ConverterService to convert from the
native type to a target type.
Currently destination is just used for things like a topic but it could
be used like in camel and also allow to override settings.
So one interesting question is .. Can a component interface stay this
simple or are we missing some essential features.
One thing is that messaging often supports properties like jms properties.
Not sure how to handle these but I would like to avoid the very generic
message model of camel that completely defeats any type safety which is
an important part of streams.
I would be happy about any suggestions for improvements and other feedback.
Best
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Prototype for reactive streams and a messaging component
abstraction
Posted by Christian Schneider <ch...@die-schneider.net>.
My main concern with Pushstreams is adoption. It currently is an OSGi
spec that probably will not see a lot of usage outside of OSGi. Reactive
streams is already supported by a few frameworks and with Spring 5
supporting it I assume there will be lots of projects out there that use
it. It also seems the reactive streams API will be part of Java 9 which
makes it even more interesting.
I wanted to use the reactor DSL as it is quite rich .. like e.g. sliding
window support. The component framework should be independent from it
but for my examples I need to decide for a stream DSL of course. Ideally
of course there could be examples with different stream DSLs but I
wanted to start small.
Christian
On 29.06.2017 13:50, Timothy Ward wrote:
> Hi Christian,
>
> PushStreams are a fundamentally different take on the way streams are controlled, mostly in an effort to make them simpler. They do not implement the reactive streams API, although it is possible to adapt between them. You mentions that you want to be independent of a stream DSL, but also that you specifically want to use the reactor DSL. These two things seem to be at odds with one another…
>
> With Push Streams you have a Push Event Source, which can be implemented directly (it’s lambda friendly) or you can use a SimplePushEventSource.
>
> From the producing side you simply publish events as they arrive. A consumer can directly connect to a source, or you can make a PushStream from it using a PushStreamProvider. A PushStream is assembled just like a Java 8 Stream, and gets you answers at the end.
>
> Push Streams are available in the sonatype OSGi repository, and are in the latest R7 draft spec. I do suggest taking a look as I think it will save a lot of code.
>
> Regards,
>
> Tim
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Prototype for reactive streams and a messaging component
abstraction
Posted by Timothy Ward <ti...@apache.org>.
Hi Christian,
PushStreams are a fundamentally different take on the way streams are controlled, mostly in an effort to make them simpler. They do not implement the reactive streams API, although it is possible to adapt between them. You mentions that you want to be independent of a stream DSL, but also that you specifically want to use the reactor DSL. These two things seem to be at odds with one another…
With Push Streams you have a Push Event Source, which can be implemented directly (it’s lambda friendly) or you can use a SimplePushEventSource.
From the producing side you simply publish events as they arrive. A consumer can directly connect to a source, or you can make a PushStream from it using a PushStreamProvider. A PushStream is assembled just like a Java 8 Stream, and gets you answers at the end.
Push Streams are available in the sonatype OSGi repository, and are in the latest R7 draft spec. I do suggest taking a look as I think it will save a lot of code.
Regards,
Tim
> On 29 Jun 2017, at 12:04, Christian Schneider <ch...@die-schneider.net> wrote:
>
> Hi Tim,
>
> I did not look into Pushstreams in detail. Does Pushstreams also support the reactive streams API?
> My goal was to create a component API that is independent of a specific stream DSL.
>
> You mentioned that push streams are simpler to use. How would a component look like for push streams. Can such a component then also be run with the reactor DSL?
>
> For my experiments with the DSL I chose reactor as it will get a lot of attention as the core of spring 5.
>
> Christian
>
> On 29.06.2017 11:10, Timothy Ward wrote:
>> Hi Christian,
>>
>> Did you also take a look at the OSGi produced reactive libraries? PushStreams seem to be a much more elegant solution for what you’re trying to do, and would let you simplify the connectors quite a lot. I think the client examples would also be quite a lot simpler. There’s also an OSGi RFC for messaging that might be helpful to look at https://github.com/osgi/design/blob/36a3ee74db246c5a73f8d043c7172494fefee948/rfcs/rfc0229/RFC0229-MQTT.pdf.
>>
>> Regards,
>>
>> Tim
>>
>>
>>
>> On 26 Jun 2017, at 17:12, Christian Schneider <ch...@die-schneider.net>> wrote:
>>
>> I recently looked into ways to combine messaging and streaming on OSGi.
>>
>> Interestingly the best streaming solution I found for my case was Reactor (by Pivotal) which is the core of spring 5. It works out of the box on OSGi and only has a single dependency.
>>
>> The next thing was how to combine this with messaging in a loosely coupled way. I really like Apache Camel but I think it is not up to date any more and also acquired a lot of weight over time (especially in camel-core). So I was looking into providing a light weight component API and combine it with Reactor.
>>
>> The result is this project:
>>
>> https://github.com/cschneider/streaming-osgi/tree/master/reactortest
>>
>> This is the Component API: https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/component/api/MComponent.java
>> Actually I am unsure if the converter must be part of the API but this is the current state.
>>
>> I created some POC components for Mqtt, EventAdmin and Mail.
>>
>> and finally two examples:
>>
>> Listen on eventadmin topic, log and forward to other topic:
>> https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/ExampleEventAdmin.java
>>
>> Listen to mqtt, compute average over sliding window and forward to other topic:
>> https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/MqttExampleComponent.java
>>
>>
>> I think there is a lot of potential in Reactor and also in messaging components that do not couple your code to the technology.
>>
>> I would be happy about any feedback on the prototype. Beware the code is not yet split into bundles but I hope the intention is still visible.
>>
>> Best
>>
>> Christian
>>
>>
>> --
>> Christian Schneider
>> http://www.liquid-reality.de
>>
>> Open Source Architect
>> http://www.talend.com
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: Prototype for reactive streams and a messaging component
abstraction
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Tim,
I did not look into Pushstreams in detail. Does Pushstreams also support
the reactive streams API?
My goal was to create a component API that is independent of a specific
stream DSL.
You mentioned that push streams are simpler to use. How would a
component look like for push streams. Can such a component then also be
run with the reactor DSL?
For my experiments with the DSL I chose reactor as it will get a lot of
attention as the core of spring 5.
Christian
On 29.06.2017 11:10, Timothy Ward wrote:
> Hi Christian,
>
> Did you also take a look at the OSGi produced reactive libraries? PushStreams seem to be a much more elegant solution for what you’re trying to do, and would let you simplify the connectors quite a lot. I think the client examples would also be quite a lot simpler. There’s also an OSGi RFC for messaging that might be helpful to look at https://github.com/osgi/design/blob/36a3ee74db246c5a73f8d043c7172494fefee948/rfcs/rfc0229/RFC0229-MQTT.pdf.
>
> Regards,
>
> Tim
>
>
>
> On 26 Jun 2017, at 17:12, Christian Schneider <ch...@die-schneider.net>> wrote:
>
> I recently looked into ways to combine messaging and streaming on OSGi.
>
> Interestingly the best streaming solution I found for my case was Reactor (by Pivotal) which is the core of spring 5. It works out of the box on OSGi and only has a single dependency.
>
> The next thing was how to combine this with messaging in a loosely coupled way. I really like Apache Camel but I think it is not up to date any more and also acquired a lot of weight over time (especially in camel-core). So I was looking into providing a light weight component API and combine it with Reactor.
>
> The result is this project:
>
> https://github.com/cschneider/streaming-osgi/tree/master/reactortest
>
> This is the Component API: https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/component/api/MComponent.java
> Actually I am unsure if the converter must be part of the API but this is the current state.
>
> I created some POC components for Mqtt, EventAdmin and Mail.
>
> and finally two examples:
>
> Listen on eventadmin topic, log and forward to other topic:
> https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/ExampleEventAdmin.java
>
> Listen to mqtt, compute average over sliding window and forward to other topic:
> https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/MqttExampleComponent.java
>
>
> I think there is a lot of potential in Reactor and also in messaging components that do not couple your code to the technology.
>
> I would be happy about any feedback on the prototype. Beware the code is not yet split into bundles but I hope the intention is still visible.
>
> Best
>
> Christian
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Prototype for reactive streams and a messaging component
abstraction
Posted by Timothy Ward <ti...@apache.org>.
Hi Christian,
Did you also take a look at the OSGi produced reactive libraries? PushStreams seem to be a much more elegant solution for what you’re trying to do, and would let you simplify the connectors quite a lot. I think the client examples would also be quite a lot simpler. There’s also an OSGi RFC for messaging that might be helpful to look at https://github.com/osgi/design/blob/36a3ee74db246c5a73f8d043c7172494fefee948/rfcs/rfc0229/RFC0229-MQTT.pdf.
Regards,
Tim
On 26 Jun 2017, at 17:12, Christian Schneider <ch...@die-schneider.net>> wrote:
I recently looked into ways to combine messaging and streaming on OSGi.
Interestingly the best streaming solution I found for my case was Reactor (by Pivotal) which is the core of spring 5. It works out of the box on OSGi and only has a single dependency.
The next thing was how to combine this with messaging in a loosely coupled way. I really like Apache Camel but I think it is not up to date any more and also acquired a lot of weight over time (especially in camel-core). So I was looking into providing a light weight component API and combine it with Reactor.
The result is this project:
https://github.com/cschneider/streaming-osgi/tree/master/reactortest
This is the Component API: https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/component/api/MComponent.java
Actually I am unsure if the converter must be part of the API but this is the current state.
I created some POC components for Mqtt, EventAdmin and Mail.
and finally two examples:
Listen on eventadmin topic, log and forward to other topic:
https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/ExampleEventAdmin.java
Listen to mqtt, compute average over sliding window and forward to other topic:
https://github.com/cschneider/streaming-osgi/blob/master/reactortest/src/main/java/reactortest/MqttExampleComponent.java
I think there is a lot of potential in Reactor and also in messaging components that do not couple your code to the technology.
I would be happy about any feedback on the prototype. Beware the code is not yet split into bundles but I hope the intention is still visible.
Best
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Prototype for reactive streams and a messaging component abstraction
Posted by David Daniel <da...@gmail.com>.
Thank you for providing this as in a month I have a task to look at
something similar. I have been playing with how to make our application
more message based and resilient. Other good libraries I have seen and was
looking forward to playing with were ActiveMQ Artemis for messaging. I
like how it is embeddable, clusterable and standards compatible it is. I
also was looking into Pax Web's undertow implementation so I can keep all
my components that use the OSGI httpservice but take advantage of the
non-blocking handlers and message passing. Undertow got OSGI headers
pulled in from a PR last week. I know Nick posted some reactive stuff he
has been working on in Pentaho and Achim posted a vertx sample. I have
looked at OSGI push streams in the past but there was very little
documentation on it even though I found the projects code online. It would
be really great to see the OSGI community coalesce around some good
practices as I feel like everyone is doing their own thing and I don't know
enough about how to integrate functional/reactive development with
services/modular development.
On Mon, Jun 26, 2017 at 12:12 PM, Christian Schneider <
chris@die-schneider.net> wrote:
> I recently looked into ways to combine messaging and streaming on OSGi.
>
> Interestingly the best streaming solution I found for my case was Reactor
> (by Pivotal) which is the core of spring 5. It works out of the box on OSGi
> and only has a single dependency.
>
> The next thing was how to combine this with messaging in a loosely coupled
> way. I really like Apache Camel but I think it is not up to date any more
> and also acquired a lot of weight over time (especially in camel-core). So
> I was looking into providing a light weight component API and combine it
> with Reactor.
>
> The result is this project:
>
> https://github.com/cschneider/streaming-osgi/tree/master/reactortest
>
> This is the Component API: https://github.com/cschneider/
> streaming-osgi/blob/master/reactortest/src/main/java/compone
> nt/api/MComponent.java
> Actually I am unsure if the converter must be part of the API but this is
> the current state.
>
> I created some POC components for Mqtt, EventAdmin and Mail.
>
> and finally two examples:
>
> Listen on eventadmin topic, log and forward to other topic:
> https://github.com/cschneider/streaming-osgi/blob/master/rea
> ctortest/src/main/java/reactortest/ExampleEventAdmin.java
>
> Listen to mqtt, compute average over sliding window and forward to other
> topic:
> https://github.com/cschneider/streaming-osgi/blob/master/rea
> ctortest/src/main/java/reactortest/MqttExampleComponent.java
>
>
> I think there is a lot of potential in Reactor and also in messaging
> components that do not couple your code to the technology.
>
> I would be happy about any feedback on the prototype. Beware the code is
> not yet split into bundles but I hope the intention is still visible.
>
> Best
>
> Christian
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>