SPI for EventDriven apps

[Niclas Hedhman <ni...@hedhman.org>]

Gang,

I have been toying with trying to run Polygene on a Flink cluster (no
success yet), and part of that is coming back to having a pluggable
Messaging/Event SPI, which is pluggable in the similar fashion to
persistence.

I am also getting a clearer point what is needed, and why we need a SPI and
not just one or more libraries for different messaging systems.

My initial (and possibly naive) thoughts are as follows;

It all comes down to UnitOfWork and the fact that Queue readers are in
their own threads.

1. A Consumer impl starts a UoW, starts a transaction on the queue*
2. Polls the Queue (potentially blocking)
3. Data read is deserialized**
4. ValueComposite is forwarded to all listeners***
5. Any kind of regular Polygene happens
6. Persistence might be invoked and part of the UoW
7. Producer impl might be called to forward a message to another queue/topic
8. Eventually returns to Consumer and the UoW.complete() happens, which
needs to handle the transaction commit to all individual parts, of
UoW.cancel() to rollback all parts.


* Does it need to start transaction on any destination queues as well?

** Several issue here;
    a. Visibility from where?
    b. Multiple types on same queue, how to resolve?
    c. Data formats not compatible with ValueComposites, such as raw binary.

*** addListener or to all Services with some tagging?


Non-trivial I would think, but not excessive by Polygene standards. :-)

WDYAT?


Cheers
-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: SPI for EventDriven apps

[Niclas Hedhman <ni...@hedhman.org>]

Ok, some initial findings about running Polygene on Flink;

This is not going to be easy. Flink by default requires the "operators"
(functions/transformations, the blocks in the stream processing chain) to
be Serializable with java.io.ObjectOutputStream. That is not going to
happen any time soon with Polygene. The alternative (I think ) is to create
a custom TaskManager, possibly even a custom JobManager, which seems like a
major undertaking, with Scala actors (Akka?) and heavy use of
java.io.Serializable. At this point in time, I am not even sure that this
is at all possible even at that level of integration.

So, I am not going to bother at this time.

That said, having a Kafka integration, as described in this thread is
something that I will attempt.


Cheers
Niclas

On Wed, Mar 28, 2018 at 6:29 PM, Niclas Hedhman <ni...@hedhman.org> wrote:

> Gang,
>
> I have been toying with trying to run Polygene on a Flink cluster (no
> success yet), and part of that is coming back to having a pluggable
> Messaging/Event SPI, which is pluggable in the similar fashion to
> persistence.
>
> I am also getting a clearer point what is needed, and why we need a SPI
> and not just one or more libraries for different messaging systems.
>
> My initial (and possibly naive) thoughts are as follows;
>
> It all comes down to UnitOfWork and the fact that Queue readers are in
> their own threads.
>
> 1. A Consumer impl starts a UoW, starts a transaction on the queue*
> 2. Polls the Queue (potentially blocking)
> 3. Data read is deserialized**
> 4. ValueComposite is forwarded to all listeners***
> 5. Any kind of regular Polygene happens
> 6. Persistence might be invoked and part of the UoW
> 7. Producer impl might be called to forward a message to another
> queue/topic
> 8. Eventually returns to Consumer and the UoW.complete() happens, which
> needs to handle the transaction commit to all individual parts, of
> UoW.cancel() to rollback all parts.
>
>
> * Does it need to start transaction on any destination queues as well?
>
> ** Several issue here;
>     a. Visibility from where?
>     b. Multiple types on same queue, how to resolve?
>     c. Data formats not compatible with ValueComposites, such as raw
> binary.
>
> *** addListener or to all Services with some tagging?
>
>
> Non-trivial I would think, but not excessive by Polygene standards. :-)
>
> WDYAT?
>
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>



-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java