You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polygene.apache.org by Niclas Hedhman <ni...@hedhman.org> on 2015/06/17 05:46:29 UTC
Messaging abstraction (for 3.0)
Gang,
I have been thinking a bit about a "Messaging" abstraction, similar to the
"Storage" and "Indexing" abstractions, and looked at my own implementations
how they are done, and see if we can come up with an Extension mechanism
that is suitable for a whole range of Messaging Extensions, such as
* Restlet
* Jersey
* JMS(?)
* Kafka
* Storm (?) [1]
There are a few initial "requirements" that I have written down on my white
board;
* UnitOfWork-like - i.e. any "result messages" are not sent until
everything is completed successfully.
* "Destination" concept - the entry point for event request processing
* Hierarchical delegation - think "paths", "routing", "intercepts"
* Request Parameters - think "headers" in HTTP or other metadata
* Request Payload
* Payload objects - converted into ValueComposites through mapping
* Binary payloads - byte[] blob
* Message Receiver is created per request. Factory implementation from
user.
* Optional Response Messages - probably part of a return type.
* Usecase metadata for Request Parameters, possibly lazy deserialization.
* Message Destination injection, for outgoing messages. Possibly a new
Composite meta type.
* Visibility - some messaging may be internal to the application, using
the same abstraction. So Visibility should be used as usual.
* Capable of distinction between Load Balanced Queues, Pub/Sub and
Peer2Peer Queues.
In very, very broad strokes, we are looking at something like this (Kafka
used as an example);
___________ +----------+ ___________
O partition O--| Kafka ME |--O partition O
----------- +----------+ -----------
| ^
v |
+--------------------+ +------------+
| Qi4j Messaging SPI | <--serves--| MC Factory |
+--------------------+ +------------+
| ^ |
v | |
+----------------------+ |
| MessageComposite (MC)|<--instantiates--+
+----------------------+
This picture doesn't involve the many options required to be handled, but
should serve as a mental model to help understand where I am coming from.
Before I start looking at how this could be done, does anyone have any
input on the topic (pun intended, as Topics are part of the problem domain
somehow, but I haven't worked that out yet)
[1] Getting an abstraction right, which allows Zest/Qi4j applications to be
part of the Storm topology would be awesome.
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: Messaging abstraction (for 3.0)
Posted by Jiri Jetmar <ju...@gmail.com>.
Hi Niclas,
yes, the idea (or the hope :-) ) to use Apache Camel as a integration
framework is the fact
that the guys behind camel had some time to practise how to define a
generic API
and how to integrate the underlying messaging framework and therefore we
can use it to accelerate
our efforts and not to start from scratch..
This might be naiv, but I would give it a try - at least from the
conceptual point of view.
Great that we have some Camel gurus here.. :-)
Cheers,
Jiri
2015-06-17 16:02 GMT+02:00 Niclas Hedhman <ni...@hedhman.org>:
> A lot of interest, and that is great.
>
> Jiri; Using Camel as Glue... I assume that you are hopeful that Camel will
> provide a lot of interfacing to many different external systems and we hook
> into a single interface somewhere/somehow. That is a possibility that I
> didn't think of, and worth-while discussing more. I think we even have two
> Camel experts on the PMC, James and Jeff, and I think Jeff volunteered to
> assist if needed, or?
>
> Kent; The idea of seeing emitting Events as a side-effect is a very sharp
> observation. And, wouldn't that also suggest that we have not nailed the
> @SideEffect concept very well yet, since any side-effects originating from
> entities (i.e. a UnitOfWork) should be deferred until UoW.complete() ??
> Really interesting...
> The 2 phase commit that you are eluding to, kind of already exist in the
> current EntityStoreSPI. The ES is called with a applyChanges() (i.e.
> prepare) on to which the commit() (or cancel()) is called, once all the
> participants are ready. NOW, I think we scrapped the previous
> implementation that allowed "per Entity type" segregation of EntityStores,
> due to complexity in configuring it for the user, but the fundamentals are
> practically there. The drop was simply pragmatic choice at that time, due
> to time constraints and other things were considered more important.
>
> Sandro; Thanks for reminding me of Actor model, which also led me to
> realize that the Java 8 Stream API is highly relevant as well
>
> Paul, and I didn't know that this is being extended into Java 9 (for sure?)
> with Flow. Since I have been doing reactive programming in Java since 1997,
> I am quite happy that it is being formalized at JDK level ;-)
>
> May I suggest that we keep pondering over the many possible directions,
> requirements, wishful thinking, and other crazy ideas on the topic, and we
> find a spot in Budapest and do a half day hackathon session on ideas of
> what could and could not work from a use-case point of view?
>
> Meanwhile, let's open a Jira ticket and just throw ideas, requirements,
> wishes, thoughts, concerns and so on at it.
>
> Mind you, I don't want us to create a new messaging system or even a new
> messaging paradigm, I simply want a way to plug such into our Composite
> system as naturally (for us) as possible. We have;
>
> a. ValueComposite with Property methods == superb data transfer objects,
> easily converted from Entities or to update Entities with.
>
> b. Full control of the invocation stack, in a relatively simple way.
>
> c. We have a Usecase and UnitOfWork concept, which should be leveraged.
> The UseCase allows any additional data to be "attached" to the UnitOfWork
> and picked up by whoever is interested, without passing the data in method
> calls.
>
> We should be able to create something great.
>
> Cheers
>
> On Wed, Jun 17, 2015 at 9:26 PM, Paul Merlin <pa...@nosphere.org> wrote:
>
> > Hey,
> >
> > Sandro Martini a écrit :
> > > Hi all,
> > > sorry I still didn't have enough time to look at Zest internals ...
> > >
> > > But I think that current state of the art in messaging is Akka (both
> > > for Scala and Java): http://akka.io/
> > > there is even an (unofficial dot Net porting).
> > >
> > > Or more generally, something in the Reactive Manifesto (
> > > http://www.reactivemanifesto.org/ ) way of thinking, like ReactiveX (
> > > http://reactivex.io/ ) and related n implementation language-specific,
> > > like RxJava ( https://github.com/ReactiveX/RxJava ) could be useful to
> > > integrate in Zest or use as a base ...
> > >
> > >
> > > I have some (basic) knowledge of Akka, so if useful, tell me ...
> > >
> > > Bye,
> > > Sandro
> > In the same vein, maybe we should turn our eyeballs towards
> > http://www.reactive-streams.org/ and the Doug Lea effort to add Flow to
> > Java 9
> >
> http://gee.cs.oswego.edu/dl/jsr166/dist/docs/java/util/concurrent/Flow.html
> > .
> >
> > Cheers
> >
> > /Paul
> >
> >
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
Re: Messaging abstraction (for 3.0)
Posted by Niclas Hedhman <ni...@hedhman.org>.
The issue is here; https://issues.apache.org/jira/browse/ZEST-32
I would be happy if everyone add their thoughts, ideas....
For Sandro and other new people, in this community, no ideas are crazy
enough to be discarded out of hand. Too often, we have seen, are such ideas
either directly useful, or has an ounce of "food for thought" to grow into
something useful.
Example; "Why is 'null' allowed in method arguments??" was a bizarre idea
at the time, but led to our rule that "No it is not allowed, unless marked
by the implementation"
On Wed, Jun 17, 2015 at 10:02 PM, Niclas Hedhman <ni...@hedhman.org> wrote:
>
> A lot of interest, and that is great.
>
> Jiri; Using Camel as Glue... I assume that you are hopeful that Camel will
> provide a lot of interfacing to many different external systems and we hook
> into a single interface somewhere/somehow. That is a possibility that I
> didn't think of, and worth-while discussing more. I think we even have two
> Camel experts on the PMC, James and Jeff, and I think Jeff volunteered to
> assist if needed, or?
>
> Kent; The idea of seeing emitting Events as a side-effect is a very sharp
> observation. And, wouldn't that also suggest that we have not nailed the
> @SideEffect concept very well yet, since any side-effects originating from
> entities (i.e. a UnitOfWork) should be deferred until UoW.complete() ??
> Really interesting...
> The 2 phase commit that you are eluding to, kind of already exist in the
> current EntityStoreSPI. The ES is called with a applyChanges() (i.e.
> prepare) on to which the commit() (or cancel()) is called, once all the
> participants are ready. NOW, I think we scrapped the previous
> implementation that allowed "per Entity type" segregation of EntityStores,
> due to complexity in configuring it for the user, but the fundamentals are
> practically there. The drop was simply pragmatic choice at that time, due
> to time constraints and other things were considered more important.
>
> Sandro; Thanks for reminding me of Actor model, which also led me to
> realize that the Java 8 Stream API is highly relevant as well
>
> Paul, and I didn't know that this is being extended into Java 9 (for
> sure?) with Flow. Since I have been doing reactive programming in Java
> since 1997, I am quite happy that it is being formalized at JDK level ;-)
>
> May I suggest that we keep pondering over the many possible directions,
> requirements, wishful thinking, and other crazy ideas on the topic, and we
> find a spot in Budapest and do a half day hackathon session on ideas of
> what could and could not work from a use-case point of view?
>
> Meanwhile, let's open a Jira ticket and just throw ideas, requirements,
> wishes, thoughts, concerns and so on at it.
>
> Mind you, I don't want us to create a new messaging system or even a new
> messaging paradigm, I simply want a way to plug such into our Composite
> system as naturally (for us) as possible. We have;
>
> a. ValueComposite with Property methods == superb data transfer
> objects, easily converted from Entities or to update Entities with.
>
> b. Full control of the invocation stack, in a relatively simple way.
>
> c. We have a Usecase and UnitOfWork concept, which should be leveraged.
> The UseCase allows any additional data to be "attached" to the UnitOfWork
> and picked up by whoever is interested, without passing the data in method
> calls.
>
> We should be able to create something great.
>
> Cheers
>
> On Wed, Jun 17, 2015 at 9:26 PM, Paul Merlin <pa...@nosphere.org> wrote:
>
>> Hey,
>>
>> Sandro Martini a écrit :
>> > Hi all,
>> > sorry I still didn't have enough time to look at Zest internals ...
>> >
>> > But I think that current state of the art in messaging is Akka (both
>> > for Scala and Java): http://akka.io/
>> > there is even an (unofficial dot Net porting).
>> >
>> > Or more generally, something in the Reactive Manifesto (
>> > http://www.reactivemanifesto.org/ ) way of thinking, like ReactiveX (
>> > http://reactivex.io/ ) and related n implementation language-specific,
>> > like RxJava ( https://github.com/ReactiveX/RxJava ) could be useful to
>> > integrate in Zest or use as a base ...
>> >
>> >
>> > I have some (basic) knowledge of Akka, so if useful, tell me ...
>> >
>> > Bye,
>> > Sandro
>> In the same vein, maybe we should turn our eyeballs towards
>> http://www.reactive-streams.org/ and the Doug Lea effort to add Flow to
>> Java 9
>>
>> http://gee.cs.oswego.edu/dl/jsr166/dist/docs/java/util/concurrent/Flow.html
>> .
>>
>> Cheers
>>
>> /Paul
>>
>>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: Messaging abstraction (for 3.0)
Posted by Niclas Hedhman <ni...@hedhman.org>.
A lot of interest, and that is great.
Jiri; Using Camel as Glue... I assume that you are hopeful that Camel will
provide a lot of interfacing to many different external systems and we hook
into a single interface somewhere/somehow. That is a possibility that I
didn't think of, and worth-while discussing more. I think we even have two
Camel experts on the PMC, James and Jeff, and I think Jeff volunteered to
assist if needed, or?
Kent; The idea of seeing emitting Events as a side-effect is a very sharp
observation. And, wouldn't that also suggest that we have not nailed the
@SideEffect concept very well yet, since any side-effects originating from
entities (i.e. a UnitOfWork) should be deferred until UoW.complete() ??
Really interesting...
The 2 phase commit that you are eluding to, kind of already exist in the
current EntityStoreSPI. The ES is called with a applyChanges() (i.e.
prepare) on to which the commit() (or cancel()) is called, once all the
participants are ready. NOW, I think we scrapped the previous
implementation that allowed "per Entity type" segregation of EntityStores,
due to complexity in configuring it for the user, but the fundamentals are
practically there. The drop was simply pragmatic choice at that time, due
to time constraints and other things were considered more important.
Sandro; Thanks for reminding me of Actor model, which also led me to
realize that the Java 8 Stream API is highly relevant as well
Paul, and I didn't know that this is being extended into Java 9 (for sure?)
with Flow. Since I have been doing reactive programming in Java since 1997,
I am quite happy that it is being formalized at JDK level ;-)
May I suggest that we keep pondering over the many possible directions,
requirements, wishful thinking, and other crazy ideas on the topic, and we
find a spot in Budapest and do a half day hackathon session on ideas of
what could and could not work from a use-case point of view?
Meanwhile, let's open a Jira ticket and just throw ideas, requirements,
wishes, thoughts, concerns and so on at it.
Mind you, I don't want us to create a new messaging system or even a new
messaging paradigm, I simply want a way to plug such into our Composite
system as naturally (for us) as possible. We have;
a. ValueComposite with Property methods == superb data transfer objects,
easily converted from Entities or to update Entities with.
b. Full control of the invocation stack, in a relatively simple way.
c. We have a Usecase and UnitOfWork concept, which should be leveraged.
The UseCase allows any additional data to be "attached" to the UnitOfWork
and picked up by whoever is interested, without passing the data in method
calls.
We should be able to create something great.
Cheers
On Wed, Jun 17, 2015 at 9:26 PM, Paul Merlin <pa...@nosphere.org> wrote:
> Hey,
>
> Sandro Martini a écrit :
> > Hi all,
> > sorry I still didn't have enough time to look at Zest internals ...
> >
> > But I think that current state of the art in messaging is Akka (both
> > for Scala and Java): http://akka.io/
> > there is even an (unofficial dot Net porting).
> >
> > Or more generally, something in the Reactive Manifesto (
> > http://www.reactivemanifesto.org/ ) way of thinking, like ReactiveX (
> > http://reactivex.io/ ) and related n implementation language-specific,
> > like RxJava ( https://github.com/ReactiveX/RxJava ) could be useful to
> > integrate in Zest or use as a base ...
> >
> >
> > I have some (basic) knowledge of Akka, so if useful, tell me ...
> >
> > Bye,
> > Sandro
> In the same vein, maybe we should turn our eyeballs towards
> http://www.reactive-streams.org/ and the Doug Lea effort to add Flow to
> Java 9
> http://gee.cs.oswego.edu/dl/jsr166/dist/docs/java/util/concurrent/Flow.html
> .
>
> Cheers
>
> /Paul
>
>
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: Messaging abstraction (for 3.0)
Posted by Paul Merlin <pa...@nosphere.org>.
Hey,
Sandro Martini a écrit :
> Hi all,
> sorry I still didn't have enough time to look at Zest internals ...
>
> But I think that current state of the art in messaging is Akka (both
> for Scala and Java): http://akka.io/
> there is even an (unofficial dot Net porting).
>
> Or more generally, something in the Reactive Manifesto (
> http://www.reactivemanifesto.org/ ) way of thinking, like ReactiveX (
> http://reactivex.io/ ) and related n implementation language-specific,
> like RxJava ( https://github.com/ReactiveX/RxJava ) could be useful to
> integrate in Zest or use as a base ...
>
>
> I have some (basic) knowledge of Akka, so if useful, tell me ...
>
> Bye,
> Sandro
In the same vein, maybe we should turn our eyeballs towards
http://www.reactive-streams.org/ and the Doug Lea effort to add Flow to
Java 9
http://gee.cs.oswego.edu/dl/jsr166/dist/docs/java/util/concurrent/Flow.html.
Cheers
/Paul
Re: Messaging abstraction (for 3.0)
Posted by Sandro Martini <sa...@gmail.com>.
Hi all,
sorry I still didn't have enough time to look at Zest internals ...
But I think that current state of the art in messaging is Akka (both
for Scala and Java): http://akka.io/
there is even an (unofficial dot Net porting).
Or more generally, something in the Reactive Manifesto (
http://www.reactivemanifesto.org/ ) way of thinking, like ReactiveX (
http://reactivex.io/ ) and related n implementation language-specific,
like RxJava ( https://github.com/ReactiveX/RxJava ) could be useful to
integrate in Zest or use as a base ...
I have some (basic) knowledge of Akka, so if useful, tell me ...
Bye,
Sandro
Re: Messaging abstraction (for 3.0)
Posted by Kent Sølvsten <ke...@gmail.com>.
* UnitOfWork-like - i.e. any "result messages" are not sent until
everything is completed successfully.
Initial thought: Messages should be sent as part of completing a UnitOfWork.
Currently a UnitOfWork is delegating to a list of relevant EntityStores
updating Entities.
Maybe it is about time to think of a UnitOfWork committing to relevant
XXXStores handling XXXComposites (with XXX being either Entity or
Message) ?
This way a UnitOfWork could be regarded as similar to an XA
transactionmanger handling multiple XAResources (stores).
The need for configuring a messagesender (MessageStore?) will probably
be much similar to configuring an EntityStore.
Another analogy (not quite compatible with above) could be to regard
AplicationEvents as @SideEffect of a composite - and Messages as
@SideEffect of a UnitOfWork - but I think the
MessageStore-as-similar-to-an-entityStore is more relevant.
/Kent
Den 17-06-2015 kl. 05:46 skrev Niclas Hedhman:
> Gang,
> I have been thinking a bit about a "Messaging" abstraction, similar to the
> "Storage" and "Indexing" abstractions, and looked at my own implementations
> how they are done, and see if we can come up with an Extension mechanism
> that is suitable for a whole range of Messaging Extensions, such as
>
> * Restlet
> * Jersey
> * JMS(?)
> * Kafka
> * Storm (?) [1]
>
> There are a few initial "requirements" that I have written down on my white
> board;
>
> * UnitOfWork-like - i.e. any "result messages" are not sent until
> everything is completed successfully.
> * "Destination" concept - the entry point for event request processing
> * Hierarchical delegation - think "paths", "routing", "intercepts"
> * Request Parameters - think "headers" in HTTP or other metadata
> * Request Payload
> * Payload objects - converted into ValueComposites through mapping
> * Binary payloads - byte[] blob
> * Message Receiver is created per request. Factory implementation from
> user.
> * Optional Response Messages - probably part of a return type.
> * Usecase metadata for Request Parameters, possibly lazy deserialization.
> * Message Destination injection, for outgoing messages. Possibly a new
> Composite meta type.
> * Visibility - some messaging may be internal to the application, using
> the same abstraction. So Visibility should be used as usual.
> * Capable of distinction between Load Balanced Queues, Pub/Sub and
> Peer2Peer Queues.
>
> In very, very broad strokes, we are looking at something like this (Kafka
> used as an example);
>
>
> ___________ +----------+ ___________
> O partition O--| Kafka ME |--O partition O
> ----------- +----------+ -----------
> | ^
> v |
> +--------------------+ +------------+
> | Qi4j Messaging SPI | <--serves--| MC Factory |
> +--------------------+ +------------+
> | ^ |
> v | |
> +----------------------+ |
> | MessageComposite (MC)|<--instantiates--+
> +----------------------+
>
>
> This picture doesn't involve the many options required to be handled, but
> should serve as a mental model to help understand where I am coming from.
>
> Before I start looking at how this could be done, does anyone have any
> input on the topic (pun intended, as Topics are part of the problem domain
> somehow, but I haven't worked that out yet)
>
>
>
> [1] Getting an abstraction right, which allows Zest/Qi4j applications to be
> part of the Storm topology would be awesome.
Re: Messaging abstraction (for 3.0)
Posted by Jiri Jetmar <ju...@gmail.com>.
Hi,
messaging is indeed one of the crucial features of a IT Platform.
I once worked for a Swiss Bank that maintained over 30k of different
Queues... :-)
I guess we are talking here about a generic internal API. There is already
a
number of messaging APIs that we can take as a basis. You mentioned already
few of them.
The issue is always the level of abstraction - high abstraction level leads
to
broad support of messaging solutions, but is not easy capable to support
special feature X.
Be we can use here a similar approach of extensions, like we did e.g. for
the spatial support.
What about to use Apache Camel as a "glue" between Zest messaging API and
the concrete messaging
implementation ?
.. my two cents ..
Cheers,
jj
2015-06-17 5:46 GMT+02:00 Niclas Hedhman <ni...@hedhman.org>:
> Gang,
> I have been thinking a bit about a "Messaging" abstraction, similar to the
> "Storage" and "Indexing" abstractions, and looked at my own implementations
> how they are done, and see if we can come up with an Extension mechanism
> that is suitable for a whole range of Messaging Extensions, such as
>
> * Restlet
> * Jersey
> * JMS(?)
> * Kafka
> * Storm (?) [1]
>
> There are a few initial "requirements" that I have written down on my white
> board;
>
> * UnitOfWork-like - i.e. any "result messages" are not sent until
> everything is completed successfully.
> * "Destination" concept - the entry point for event request processing
> * Hierarchical delegation - think "paths", "routing", "intercepts"
> * Request Parameters - think "headers" in HTTP or other metadata
> * Request Payload
> * Payload objects - converted into ValueComposites through mapping
> * Binary payloads - byte[] blob
> * Message Receiver is created per request. Factory implementation from
> user.
> * Optional Response Messages - probably part of a return type.
> * Usecase metadata for Request Parameters, possibly lazy deserialization.
> * Message Destination injection, for outgoing messages. Possibly a new
> Composite meta type.
> * Visibility - some messaging may be internal to the application, using
> the same abstraction. So Visibility should be used as usual.
> * Capable of distinction between Load Balanced Queues, Pub/Sub and
> Peer2Peer Queues.
>
> In very, very broad strokes, we are looking at something like this (Kafka
> used as an example);
>
>
> ___________ +----------+ ___________
> O partition O--| Kafka ME |--O partition O
> ----------- +----------+ -----------
> | ^
> v |
> +--------------------+ +------------+
> | Qi4j Messaging SPI | <--serves--| MC Factory |
> +--------------------+ +------------+
> | ^ |
> v | |
> +----------------------+ |
> | MessageComposite (MC)|<--instantiates--+
> +----------------------+
>
>
> This picture doesn't involve the many options required to be handled, but
> should serve as a mental model to help understand where I am coming from.
>
> Before I start looking at how this could be done, does anyone have any
> input on the topic (pun intended, as Topics are part of the problem domain
> somehow, but I haven't worked that out yet)
>
>
>
> [1] Getting an abstraction right, which allows Zest/Qi4j applications to be
> part of the Storm topology would be awesome.
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>