A few zest 3.0 thought?

[Kent Sølvsten <ke...@gmail.com>]

Gang,

A lot of stuff has been discussed during the summer - suggestions for an
upcoming 3.0 release.
A lot of it consists of new composite types and new ways of persistence
- thus challenging the existing way of persisting stuff.

A Zest application consists of a single application with a CONFIGURABLE
number of layers
Each layer has a CONFIGURABLE number of modules.
Each module has a FIXED set of structural parts: The UOW and
implementations of some factories, each factory handling the lifecycle
of a single composite type.
The UOW stuff implements a FIXED set of lifecycle interfaces, adding
ways to query, fetch and manipulate entities (a single composite type).

Below the fixed API a few service provider interfaces allow for several
implementations, especially for storing and querying stuff.

We have already stuff challenging that model, since the native queries
(SOLR/SERQL if i recall correctly) are not a great fit.
We have the spatial query stuff, which is great, but the existing model
is not really prepared to add that.

In the pipeline we have more stuff ....

Message composites with a message store?
Aggregates with an event store?
And maybe some ORM stuff too?

The "natural" way to add that would be by adding more factories to the
UOW - and maybe some more query interfaces to the modules.
Probably not a good idea to overload those already large interfaces -
and a suggestion that has already (implicitly?) been rejected for the
native query stuff.

I think we simply *need* a way to extend the core with new stuff in a
natural way.

Could we use services for that?

What would happen if we removed
ObjectFactory/ValueBuilderFactory/TransientBuilderFactory from the
module itself, and added those as services instead?
Application code could no longer use the functionality from the module
directly, but would have to use the ServiceFinder or @Structure injections.
A bonus would be adding visibility - allowing assemblers to enforce that
some of the factories can only be used from inside the module.

Not trivial to implement, but not that bad either, I think.

Somewhere down the road, that might even suggest a way to add new
composite types. Bootstrapping TransientComposites could in some way be
regarded as syntactical sugar around configuring the
TransientBuilderFactory structural service - and maybe the model
definitions could even live inside that service instead of inside the
module?

But .....

The hairy stuff is the persistent stuff living inside the UnitOfWork!

Going down the road suggested above, the obvious way would be to
implement services such as
EntityBuilderFactory/QueryBuilderFactory/EntityFinder - and then more
services to support new persistence/composites.
The UOW can NOT depend directly on those, since it cannot know which
sort of composites are persisted. So the services will have to manage
that themselves, and need to implement a common interface for receiving
notifications before/during/after completion of a UOW. The services
could have a concern ensuring the existence of an active UOW.

I believe it CAN be done - but would require a lot of collaboration
between the UOW and those services - and a lot of the services would
have to hold maps Map<UnitOfWork,"Stuff"> for stores and pending changes.

Can it be improved?

Currently all services are singletons (at least within a module) - that
is, they have module scope. The services may implement interfaces
allowing them to get notified when the module is activated/passivated.

What if we had "UnitOfWork-scoped" services?

A service with UOW scope is having a single instance per UOW (if used) -
which is ensured by the runtime itself. No need for having a concern
ensuring an active UOW - if there is no UOW there is simply no instance!
References to a UOW-scoped service is simply a proxy delegating to the
correct implementation at runtime - (the same pattern has been used in
other frameworks, normally to delegate to a ThreadLocal instance). The
service may implement interfaces allowing to get notified when the UOW
is activated/passivated - maybe the usual Activation/Activator
interfaces can be used?

The point is that these services can be stateful - and could be used to
hold eg. pending changes (entities/messages/events/aggregates) inside a
UOW. No Maps necessary. No need for the UOW to do that book keeping. So
EntityBuilderFactory/EntityFinder/QueryBuilderFactory could be
UOW-scoped services. The services would not need to register themselves
with the UOW (that is done automatically). And the UOW would not need to
tell a lot of details on completion, since those details are remembered
by the service itself.

This can be used for persistance stuff but other ideas come to mind
 - a service could implement a logging API, but only flush all debug
messages regarding a UOW to disk in case of that UOW failing.
- another service could track usage patterns of the application sorted
by use cases.

We could have a SpatialQueryBuilderFactory as a UOW-scoped service,
using an underlying indexer
We could have a SOLRQueryBuilderFactory as a UOW-scoped service using an
underlying indexer.
We could have a MessageDispatcher as a UOW-scoped service using a JMS
implementation below
.......

Maybe we could even have services with other kind of scopes - the
obvious case is HttpSession - but i guess we are now far beyond 3.0 scope.

Am I making sense? Would it be worthwhile to attempt a prototype based
on some of these ideas?

/Kent

Re: A few zest 3.0 thought?

[Niclas Hedhman <ni...@hedhman.org>]

This is triggering another thought, which is possibly also quite radical...

If everything is UnitOfWork bound, wouldn't that also imply that any
"change" should be "captured" at invocation, seen within the UoW, but not
"done" and visible to other threads, until "UoW.complete()" ??

That would mean that the UnitOfWork acts like Software Transactional
Memory, with additionally binding in all external (file, network) activity
in this as well.

In practical terms, ValueComposites shouldn't be impacted. Entities already
has this behavior.

But then we have Services, Transients and Objects, which would need to get
the Entity state semantics fitted in them, and something implementing
StateCommitter or equivalent.

This could also solve the Value deserialization problem, since there will
then be a 'handle' on what the visibility should be, and not the current
hack of a "Finder".

WDYT?
Niclas

On Tue, Aug 25, 2015 at 9:31 PM, Niclas Hedhman <ni...@hedhman.org> wrote:

>
> Yes, you are making sense, and aligns with my thoughts.
>
> The challenge lies in the details, and why QI4j pre-1.0 got seriously
> burned, by something we don't even have a name for (call it Visibility
> migration). The UnitOfWork adjusts the visibility as the calls traverses
> the modules and layers. So, although the UoW is created high up, it needs
> to "adapt" to visibility for each composite/object asking for something.
> The mechanism for this is both very delicate, not obvious, and hard to
> reason around. It is also likely to be filled with bugs.
>
> Most of the Persistence does the right thing, but in Indexing it works for
> rdf-indexer, but IIRC it doesn't work in the sql-indexer. A lot of that is
> probably related to broken separation of concerns, in that UoW should
> "somehow" ensure that the SPI level stuff doesn't need to worry about it.
>
> And it could be that there is missing a "ViewPoint" concept, i.e. the
> location from where the Visibility is derived, and that the ViewPoint might
> be even more central to the invocation stack than belonging inside the UoW,
> since I assume UoW isn't necessarily required (OR IS IT??).
>
> Hard to discuss this on mailing list, and a whiteboard and couple of days
> of time would sure be more efficient.
>
> I more than welcome you to drive this with some stylized examples, that
> tries to take into consideration what happens during persistence, querying
> and other future "services" in such a world.
>
>
> Cheers
> Niclas
>
> On Tue, Aug 25, 2015 at 7:29 AM, Kent Sølvsten <ke...@gmail.com>
> wrote:
>
>> Gang,
>>
>> A lot of stuff has been discussed during the summer - suggestions for an
>> upcoming 3.0 release.
>> A lot of it consists of new composite types and new ways of persistence
>> - thus challenging the existing way of persisting stuff.
>>
>> A Zest application consists of a single application with a CONFIGURABLE
>> number of layers
>> Each layer has a CONFIGURABLE number of modules.
>> Each module has a FIXED set of structural parts: The UOW and
>> implementations of some factories, each factory handling the lifecycle
>> of a single composite type.
>> The UOW stuff implements a FIXED set of lifecycle interfaces, adding
>> ways to query, fetch and manipulate entities (a single composite type).
>>
>> Below the fixed API a few service provider interfaces allow for several
>> implementations, especially for storing and querying stuff.
>>
>> We have already stuff challenging that model, since the native queries
>> (SOLR/SERQL if i recall correctly) are not a great fit.
>> We have the spatial query stuff, which is great, but the existing model
>> is not really prepared to add that.
>>
>> In the pipeline we have more stuff ....
>>
>> Message composites with a message store?
>> Aggregates with an event store?
>> And maybe some ORM stuff too?
>>
>> The "natural" way to add that would be by adding more factories to the
>> UOW - and maybe some more query interfaces to the modules.
>> Probably not a good idea to overload those already large interfaces -
>> and a suggestion that has already (implicitly?) been rejected for the
>> native query stuff.
>>
>> I think we simply *need* a way to extend the core with new stuff in a
>> natural way.
>>
>> Could we use services for that?
>>
>> What would happen if we removed
>> ObjectFactory/ValueBuilderFactory/TransientBuilderFactory from the
>> module itself, and added those as services instead?
>> Application code could no longer use the functionality from the module
>> directly, but would have to use the ServiceFinder or @Structure
>> injections.
>> A bonus would be adding visibility - allowing assemblers to enforce that
>> some of the factories can only be used from inside the module.
>>
>> Not trivial to implement, but not that bad either, I think.
>>
>> Somewhere down the road, that might even suggest a way to add new
>> composite types. Bootstrapping TransientComposites could in some way be
>> regarded as syntactical sugar around configuring the
>> TransientBuilderFactory structural service - and maybe the model
>> definitions could even live inside that service instead of inside the
>> module?
>>
>> But .....
>>
>> The hairy stuff is the persistent stuff living inside the UnitOfWork!
>>
>> Going down the road suggested above, the obvious way would be to
>> implement services such as
>> EntityBuilderFactory/QueryBuilderFactory/EntityFinder - and then more
>> services to support new persistence/composites.
>> The UOW can NOT depend directly on those, since it cannot know which
>> sort of composites are persisted. So the services will have to manage
>> that themselves, and need to implement a common interface for receiving
>> notifications before/during/after completion of a UOW. The services
>> could have a concern ensuring the existence of an active UOW.
>>
>> I believe it CAN be done - but would require a lot of collaboration
>> between the UOW and those services - and a lot of the services would
>> have to hold maps Map<UnitOfWork,"Stuff"> for stores and pending changes.
>>
>> Can it be improved?
>>
>> Currently all services are singletons (at least within a module) - that
>> is, they have module scope. The services may implement interfaces
>> allowing them to get notified when the module is activated/passivated.
>>
>> What if we had "UnitOfWork-scoped" services?
>>
>> A service with UOW scope is having a single instance per UOW (if used) -
>> which is ensured by the runtime itself. No need for having a concern
>> ensuring an active UOW - if there is no UOW there is simply no instance!
>> References to a UOW-scoped service is simply a proxy delegating to the
>> correct implementation at runtime - (the same pattern has been used in
>> other frameworks, normally to delegate to a ThreadLocal instance). The
>> service may implement interfaces allowing to get notified when the UOW
>> is activated/passivated - maybe the usual Activation/Activator
>> interfaces can be used?
>>
>> The point is that these services can be stateful - and could be used to
>> hold eg. pending changes (entities/messages/events/aggregates) inside a
>> UOW. No Maps necessary. No need for the UOW to do that book keeping. So
>> EntityBuilderFactory/EntityFinder/QueryBuilderFactory could be
>> UOW-scoped services. The services would not need to register themselves
>> with the UOW (that is done automatically). And the UOW would not need to
>> tell a lot of details on completion, since those details are remembered
>> by the service itself.
>>
>> This can be used for persistance stuff but other ideas come to mind
>>  - a service could implement a logging API, but only flush all debug
>> messages regarding a UOW to disk in case of that UOW failing.
>> - another service could track usage patterns of the application sorted
>> by use cases.
>>
>> We could have a SpatialQueryBuilderFactory as a UOW-scoped service,
>> using an underlying indexer
>> We could have a SOLRQueryBuilderFactory as a UOW-scoped service using an
>> underlying indexer.
>> We could have a MessageDispatcher as a UOW-scoped service using a JMS
>> implementation below
>> .......
>>
>> Maybe we could even have services with other kind of scopes - the
>> obvious case is HttpSession - but i guess we are now far beyond 3.0 scope.
>>
>> Am I making sense? Would it be worthwhile to attempt a prototype based
>> on some of these ideas?
>>
>> /Kent
>>
>>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>



-- 
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java

Re: A few zest 3.0 thought?

[Niclas Hedhman <ni...@hedhman.org>]

Yes, you are making sense, and aligns with my thoughts.

The challenge lies in the details, and why QI4j pre-1.0 got seriously
burned, by something we don't even have a name for (call it Visibility
migration). The UnitOfWork adjusts the visibility as the calls traverses
the modules and layers. So, although the UoW is created high up, it needs
to "adapt" to visibility for each composite/object asking for something.
The mechanism for this is both very delicate, not obvious, and hard to
reason around. It is also likely to be filled with bugs.

Most of the Persistence does the right thing, but in Indexing it works for
rdf-indexer, but IIRC it doesn't work in the sql-indexer. A lot of that is
probably related to broken separation of concerns, in that UoW should
"somehow" ensure that the SPI level stuff doesn't need to worry about it.

And it could be that there is missing a "ViewPoint" concept, i.e. the
location from where the Visibility is derived, and that the ViewPoint might
be even more central to the invocation stack than belonging inside the UoW,
since I assume UoW isn't necessarily required (OR IS IT??).

Hard to discuss this on mailing list, and a whiteboard and couple of days
of time would sure be more efficient.

I more than welcome you to drive this with some stylized examples, that
tries to take into consideration what happens during persistence, querying
and other future "services" in such a world.


Cheers
Niclas

On Tue, Aug 25, 2015 at 7:29 AM, Kent Sølvsten <ke...@gmail.com>
wrote:

> Gang,
>
> A lot of stuff has been discussed during the summer - suggestions for an
> upcoming 3.0 release.
> A lot of it consists of new composite types and new ways of persistence
> - thus challenging the existing way of persisting stuff.
>
> A Zest application consists of a single application with a CONFIGURABLE
> number of layers
> Each layer has a CONFIGURABLE number of modules.
> Each module has a FIXED set of structural parts: The UOW and
> implementations of some factories, each factory handling the lifecycle
> of a single composite type.
> The UOW stuff implements a FIXED set of lifecycle interfaces, adding
> ways to query, fetch and manipulate entities (a single composite type).
>
> Below the fixed API a few service provider interfaces allow for several
> implementations, especially for storing and querying stuff.
>
> We have already stuff challenging that model, since the native queries
> (SOLR/SERQL if i recall correctly) are not a great fit.
> We have the spatial query stuff, which is great, but the existing model
> is not really prepared to add that.
>
> In the pipeline we have more stuff ....
>
> Message composites with a message store?
> Aggregates with an event store?
> And maybe some ORM stuff too?
>
> The "natural" way to add that would be by adding more factories to the
> UOW - and maybe some more query interfaces to the modules.
> Probably not a good idea to overload those already large interfaces -
> and a suggestion that has already (implicitly?) been rejected for the
> native query stuff.
>
> I think we simply *need* a way to extend the core with new stuff in a
> natural way.
>
> Could we use services for that?
>
> What would happen if we removed
> ObjectFactory/ValueBuilderFactory/TransientBuilderFactory from the
> module itself, and added those as services instead?
> Application code could no longer use the functionality from the module
> directly, but would have to use the ServiceFinder or @Structure injections.
> A bonus would be adding visibility - allowing assemblers to enforce that
> some of the factories can only be used from inside the module.
>
> Not trivial to implement, but not that bad either, I think.
>
> Somewhere down the road, that might even suggest a way to add new
> composite types. Bootstrapping TransientComposites could in some way be
> regarded as syntactical sugar around configuring the
> TransientBuilderFactory structural service - and maybe the model
> definitions could even live inside that service instead of inside the
> module?
>
> But .....
>
> The hairy stuff is the persistent stuff living inside the UnitOfWork!
>
> Going down the road suggested above, the obvious way would be to
> implement services such as
> EntityBuilderFactory/QueryBuilderFactory/EntityFinder - and then more
> services to support new persistence/composites.
> The UOW can NOT depend directly on those, since it cannot know which
> sort of composites are persisted. So the services will have to manage
> that themselves, and need to implement a common interface for receiving
> notifications before/during/after completion of a UOW. The services
> could have a concern ensuring the existence of an active UOW.
>
> I believe it CAN be done - but would require a lot of collaboration
> between the UOW and those services - and a lot of the services would
> have to hold maps Map<UnitOfWork,"Stuff"> for stores and pending changes.
>
> Can it be improved?
>
> Currently all services are singletons (at least within a module) - that
> is, they have module scope. The services may implement interfaces
> allowing them to get notified when the module is activated/passivated.
>
> What if we had "UnitOfWork-scoped" services?
>
> A service with UOW scope is having a single instance per UOW (if used) -
> which is ensured by the runtime itself. No need for having a concern
> ensuring an active UOW - if there is no UOW there is simply no instance!
> References to a UOW-scoped service is simply a proxy delegating to the
> correct implementation at runtime - (the same pattern has been used in
> other frameworks, normally to delegate to a ThreadLocal instance). The
> service may implement interfaces allowing to get notified when the UOW
> is activated/passivated - maybe the usual Activation/Activator
> interfaces can be used?
>
> The point is that these services can be stateful - and could be used to
> hold eg. pending changes (entities/messages/events/aggregates) inside a
> UOW. No Maps necessary. No need for the UOW to do that book keeping. So
> EntityBuilderFactory/EntityFinder/QueryBuilderFactory could be
> UOW-scoped services. The services would not need to register themselves
> with the UOW (that is done automatically). And the UOW would not need to
> tell a lot of details on completion, since those details are remembered
> by the service itself.
>
> This can be used for persistance stuff but other ideas come to mind
>  - a service could implement a logging API, but only flush all debug
> messages regarding a UOW to disk in case of that UOW failing.
> - another service could track usage patterns of the application sorted
> by use cases.
>
> We could have a SpatialQueryBuilderFactory as a UOW-scoped service,
> using an underlying indexer
> We could have a SOLRQueryBuilderFactory as a UOW-scoped service using an
> underlying indexer.
> We could have a MessageDispatcher as a UOW-scoped service using a JMS
> implementation below
> .......
>
> Maybe we could even have services with other kind of scopes - the
> obvious case is HttpSession - but i guess we are now far beyond 3.0 scope.
>
> Am I making sense? Would it be worthwhile to attempt a prototype based
> on some of these ideas?
>
> /Kent
>
>


-- 
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java