You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polygene.apache.org by Kent Sølvsten <ke...@gmail.com> on 2016/08/10 20:21:41 UTC
Re: JPA?
This is a tough, but interesting question ......
I have seen a few other attempts at bridging the NoSql with the JPA world -
you could consider taking a look at DataNucleus or Hibernate OGM.
Both seem to generate native queries based on the JPA API (similar to
the Zest QueryBuilders) - and then attempt to integrate all changes with
the global (XA) transaction of the Application Server
(constrained be the presence/absence of XA support in the stores).
I am not quite convinced it would be the way to do this - the fit
between the Zest entities and the JPA API is not quite obvious to me -
but a similar strategy *should* be possible.
But is it really what is asked for?
- would it be better to allow more seamless integration where parts of
an application is persisted in a traditional relational database (using
JPA), and other parts are "DDD/Zest" entities?
That could present a nice migration path, I think. And the integration
would be, not so much on the query/persistence part, but more in the
UnitOfWork/transaction part.
A few ideas:
(1)
If the persistence engines in the entitystores are XA capable (such as
for JDBC/NEO4J entitystores) it should not be too difficult.
The basic idea would be to inject a DataSource provided by the appserver
into the EntityStore and let the EntityStore avoid committing - leaning
on commit of the global XA transaction
- and postpone notifying indexers until the global transaction is
committed (we should be able to listen to "commit events").
(2)
Abother possible strategy could be to let the UnitOfWork be a resource
managed by the appservers transaction manager - A lot more complex!
But maybe we could be able to use a "last resource gambit" - so the UOW
will need to be a resource managed by the TransactionManager, but at
least it should not be necessary to be XA compliant.
(3)
Another idea could be to separate the UOW completion from the "flush" to
the EntityStore(s).
Right now the "definition" of UOW#complete is flushing changes to the
EntityStore(s) - with the (eventual?) sideeffect of notifying indexers.
What if this was changed, so that the "definition" of UOW#complete is
persisting some record to a UnitOfWorkLog
- with the (eventual) sideeffect of flushing changes to EntityStore(s)
- with the (eventual) sideeffect of notifying indexers.
(actually this is not too far from eventsourcing - so maybe the
UnitOfWorkLog is simply an "EventStore"?).
The point is that the unitOfWorkLog could have a fixed schema, making it
suitable for an (XA-compliant) relational database.
A call flow could be:
- interceptor1 starts global transaction
- interceptor2 start UOW
- some changes are made to JPA entities
- some changes are made to Zest entities
- interceptor2 completes UOW (updates UOWLog record in a relational
database)
- interceptor1 commits global transaction
- we listen for commit events and calculates/flushes changes to
EntityStores based on the UOWLog (this begin to look more and more like
an eventstore)
- with the sideeffect of notifying indexers....
Not simple, but maybe not too hopeless. Note that this could potentially
be useful in a Zest-only app with a single UOW leading to changes to
multiple EntityStores.
WDYT?
/Kent
Den 31-07-2016 kl. 01:42 skrev Niclas Hedhman:
> I just came across something that has been discussed several times in the
> past. It is a anser on [1]
>
> <quote>
> When I read through the Qi4j in 2 minutes, 10 minutes etc. tutorials (the
> last one is incomplete), the one obvious question that came to mind was how
> do I integrate this with JPA/Hibernate managed entities? I would like to
> see a solution that seamlessly integrates with JPA. No JPA implies no Qi4j
> adoption in my opinion. I'd love to see an article by the author(s) that
> shows integration with JPA and Spring, two things that have deep
> penetration in the Java Enterprise world. If the integration is
> straightforward, rapid adoption will follow.
> </quote>
>
> As I said, this has surfaced several times in the past, and each time it
> has been discarded as "impossible" due to our models are significantly
> different from JPA in general and Hibernate in particular.
>
> Now, since we are breaking compatibility in a major way, the question that
> must be asked is;
>
> Is there anything we can do to make this a possibility?
>
> Because, even though Hibernate has its serious issues, I think the post
> highlights the gist of the problem; Spring and Hibernate is popular. We
> should consider that fact seriously.
>
>
> [1]
> http://stackoverflow.com/questions/2196612/anybody-using-qi4j/9442920#9442920
>
> Cheers
Re: JPA?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
On 21.9.2016 16:01, Niclas Hedhman wrote:
> Thanks a lot Stan for the overview picture, which is sometimes hard to
> acquire by looking at the code.
No problems at all!
Especially it can be hard to acquire because the code is really old and
poorly documented. :)
> 3. All state that is not exposed by an EntityType is still expected to be
> preserved if an Entity of that EntityType is updated. The trick is that
> some EntityTypes overlap and many don't, and currently there is no way to
> know when EntityTypes overlap and need to share a table for
> Property/Association. IF that were to be introduced somehow, then one table
> per over-arching entity typo could be created with a column per
> Property/Association, and possibly even manage a separate table for each of
> ManyAssoc/NamedAssoc dynamically. I think that in reality that is always
> the case, both for myself and what I remember from StreamFlow. The problem
> is that some small subtypes will cross tables, and it is this feature that
> we could decide to prevent for RDBMS used as 1c above.
Actually, this is the reason why SQL indexing has one table per
property/asso.
In order for data to be properly normalized, it should reside in *one*
place.
To elaborate, suppose you have:
interface MyBaseEntity {
Property<String> someProperty();
}
interface MySubEntity extends MyBaseEntity {
Property<String> someOtherProperty();
}
interface MyAnotherSubEntity extends MyBaseEntity {
Property<String> yetAnotherProperty();
}
If you have added "MySubEntity" and "MyAnotherSubEntity" types as entity
composites in your Zest application setup, then current SQL indexing
will generate exactly *one* table for MyBaseEntity.someProperty property.
That is, even when it is seen as "double" - being present in both
MySubEntity and MyAnotherSubEntity, in database it still exists in
exactly one place - the dedicated table.
I had to do this in quite early stage of the code, otherwise the
indexing tests would not have passed for SQL indexing. :)
Niclas, did I talk about the same thing you were talking in #3? :)
>
> 4. For 1b above, the existing EntityStore might be the fastest option and
> for some people that is reasonable choice. However, data lives for long,
> and we should in that case make sure that all the metadata that is in the
> Zest type system, is also stored alongside this compact, serialized format,
> and that we have a generic tool that can read 1b storage type and create a
> 1c output, in case people change their mind, throw away Zest or other
> unforeseen circumstances.
>
> I hope you all get a feel for my thinking here... The default Zest type
> system is way too "cool" for RDBMS, but we could introduce some limitations
> to better support RDBMS in an enterprise setting.
I think the current Zest data model is very nice.
Any limitations to be introduced should be optional - enabled by user of
Zest, if (s)he so requires.
In the end, it's all about trade-offs: You can always have the 1b/c
options, if you compromise disk space/performance/sync time.
Related to metadata: SQL indexing stores that as well, in auxiliary
tables of the schema.
That had to be done in order to detect the changes in entity types: e.g.
adding a new property to entity type, and SQL indexing automatically
generating a new table for that.
Re: JPA?
Posted by Niclas Hedhman <ni...@hedhman.org>.
Thanks a lot Stan for the overview picture, which is sometimes hard to
acquire by looking at the code.
On this topic, there are a few "constraints" that perhaps should be
considered as Core Zest features somehow, that could make life easier when
insisting on RDBMS backing an Zest application. After all, most of my
Entities end up having a static data structure, and perhaps this fact can
aid implementation of both storage and indexing.
1. There is a big difference between;
a. SQL Schema is completely foreign, and Zest developer wants to read
that into Entities.
b. Zest owning the schema, optimized for performance
c. Zest owning the schema, optimized for interoperability over SQL data.
2. For 1a. above, I think it is reasonable to have some restrictions on
what is possible. For instance, we could create an interactive tool that
queries the schema from a database, allows people mark what are Entities
(and their friendlier name) and to tie together Association fields, and
many-to-many tables as ManyAssocations. Once such a tool generates the
EntityTypes they are set in stone, and one can't become fancy and introduce
supertypes that crosses tables and such advanced features.
3. All state that is not exposed by an EntityType is still expected to be
preserved if an Entity of that EntityType is updated. The trick is that
some EntityTypes overlap and many don't, and currently there is no way to
know when EntityTypes overlap and need to share a table for
Property/Association. IF that were to be introduced somehow, then one table
per over-arching entity typo could be created with a column per
Property/Association, and possibly even manage a separate table for each of
ManyAssoc/NamedAssoc dynamically. I think that in reality that is always
the case, both for myself and what I remember from StreamFlow. The problem
is that some small subtypes will cross tables, and it is this feature that
we could decide to prevent for RDBMS used as 1c above.
4. For 1b above, the existing EntityStore might be the fastest option and
for some people that is reasonable choice. However, data lives for long,
and we should in that case make sure that all the metadata that is in the
Zest type system, is also stored alongside this compact, serialized format,
and that we have a generic tool that can read 1b storage type and create a
1c output, in case people change their mind, throw away Zest or other
unforeseen circumstances.
I hope you all get a feel for my thinking here... The default Zest type
system is way too "cool" for RDBMS, but we could introduce some limitations
to better support RDBMS in an enterprise setting.
Niclas
On Wed, Sep 21, 2016 at 5:59 PM, Stanislav Muhametsin <
stanislav.muhametsin@zest.mail.kapsi.fi> wrote:
> On 21.9.2016 12:00, Stanislav Muhametsin wrote:
>
>> On 21.9.2016 0:08, Jiri Jetmar wrote:
>>
>>> Independently of that that, things starts to be complicated in the SQL
>>> world with large data when you are submitting e.g. a inner JOIN statement
>>> in a transactional INSERT expression, where the tables are located (the
>>> data) on different nodes, simply because of a single node limit.
>>>
>>
>> Well... there are ways to circumvent this, too. :)
>> Unfortunately, can't tell much more about that, as it is key concept of
>> the company where I work right now.
>> But very large and globally distributed RDBMS nodes, which don't lag
>> under high stress, are very much so possible!
>>
>> I would like to work on this task. Therefore smart ideas are highly
>>> welcome
>>> ! :-)
>>>
>>
>> I am responsible for current SQL indexing code, and Paul is responsible
>> for current SQL entitystore code.
>> The indexing code is, unfortunately, quite spaghettified, and I would do
>> lots of things probably very differently now, than how I did them several
>> years ago.
>>
>> I think one strategy to approach would be to re-write whole thing from
>> scratch, but keeping the main principles:
>> 1. Each property and association should be stored in separate table. This
>> is required for the very dynamic nature of Zest queries.
>> This implicates that the whole DB schema will be owned by this
>> indexing service - it is not compatible with some external tool generating
>> tables (unless tables are, of course, in this exact required format).
>>
>> Example: if you have interface MyEntity { Property<String> myProp();
>> Association<SomeOtherEntity> myAsso(); },
>> That would end up with 3 tables:
>> 1.1. The table for implicit 'identity' property. Current indexing
>> code has separate DB-specific entity primary key which is of type 'long'
>> (64bit integer), since that makes foreign key references much faster and
>> compacter. Not sure if that is good idea anymore.
>> 1.2. The table for 'myProp' property with 2 columns: foreign key
>> reference to identity table and actual column of type 'text' containing the
>> contents of 'myProp' property.
>> 1.3. The table for 'myAsso' association with 2 columns: foreign key
>> reference to identity table of 'MyEntity' and foreign key reference to
>> identity table of 'SomeOtherEntity'. Obviously this covers the
>> 'ManyAssociation's as well, if they are still present in Zest.
>>
>
> I forgot one thing to mention: the collectionized properties were cause of
> major headache in SQL indexing.
> IIRC I made separate tables and/or columns to model collections properly,
> in order to properly support queries which had conditions like "property X
> is list containing entity Y with property Z at value Ö".
>
> However, with the JSON datatype, it might be better (?) idea to make
> collectionized properties as single column with JSON datatype, and just
> JSON-ize collections.
>
>
>
>> 2. The SQL query will then be created by translating Zest query objects
>> into SQL query objects. You could also just build SQL strings, but it would
>> be quite nasty and most likely very unmaintaineable task - current code
>> uses java-sql-generator, which is my library on github, to create SQL
>> statements using DSL, and not by concatenating strings.
>> IIRC Zest AND-conditions are SQL INTERSECTIONs of single
>> property/asso queries, Zest OR-conditions are SQL UNIONs of single
>> property/asso queries, and Zest NOT-conditions are SQL NOTs of single
>> property/asso queries.
>>
>> 3. Keeping DB schema in sync with Zest application schema.
>> This is very tough one - if someone adds/removes
>> properties/associations, the database schema should be kept up-to-date.
>> I remember having problems with this one - it also implies that you
>> have to walk through the *whole* application structure in order to generate
>> tables for all the properties/assos of all the entities visible to this
>> indexing service.
>>
>>
>>
>
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: JPA?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
On 21.9.2016 12:00, Stanislav Muhametsin wrote:
> On 21.9.2016 0:08, Jiri Jetmar wrote:
>> Independently of that that, things starts to be complicated in the SQL
>> world with large data when you are submitting e.g. a inner JOIN
>> statement
>> in a transactional INSERT expression, where the tables are located (the
>> data) on different nodes, simply because of a single node limit.
>
> Well... there are ways to circumvent this, too. :)
> Unfortunately, can't tell much more about that, as it is key concept
> of the company where I work right now.
> But very large and globally distributed RDBMS nodes, which don't lag
> under high stress, are very much so possible!
>
>> I would like to work on this task. Therefore smart ideas are highly
>> welcome
>> ! :-)
>
> I am responsible for current SQL indexing code, and Paul is
> responsible for current SQL entitystore code.
> The indexing code is, unfortunately, quite spaghettified, and I would
> do lots of things probably very differently now, than how I did them
> several years ago.
>
> I think one strategy to approach would be to re-write whole thing from
> scratch, but keeping the main principles:
> 1. Each property and association should be stored in separate table.
> This is required for the very dynamic nature of Zest queries.
> This implicates that the whole DB schema will be owned by this
> indexing service - it is not compatible with some external tool
> generating tables (unless tables are, of course, in this exact
> required format).
>
> Example: if you have interface MyEntity { Property<String>
> myProp(); Association<SomeOtherEntity> myAsso(); },
> That would end up with 3 tables:
> 1.1. The table for implicit 'identity' property. Current indexing
> code has separate DB-specific entity primary key which is of type
> 'long' (64bit integer), since that makes foreign key references much
> faster and compacter. Not sure if that is good idea anymore.
> 1.2. The table for 'myProp' property with 2 columns: foreign key
> reference to identity table and actual column of type 'text'
> containing the contents of 'myProp' property.
> 1.3. The table for 'myAsso' association with 2 columns: foreign
> key reference to identity table of 'MyEntity' and foreign key
> reference to identity table of 'SomeOtherEntity'. Obviously this
> covers the 'ManyAssociation's as well, if they are still present in Zest.
I forgot one thing to mention: the collectionized properties were cause
of major headache in SQL indexing.
IIRC I made separate tables and/or columns to model collections
properly, in order to properly support queries which had conditions like
"property X is list containing entity Y with property Z at value �".
However, with the JSON datatype, it might be better (?) idea to make
collectionized properties as single column with JSON datatype, and just
JSON-ize collections.
>
> 2. The SQL query will then be created by translating Zest query
> objects into SQL query objects. You could also just build SQL strings,
> but it would be quite nasty and most likely very unmaintaineable task
> - current code uses java-sql-generator, which is my library on github,
> to create SQL statements using DSL, and not by concatenating strings.
> IIRC Zest AND-conditions are SQL INTERSECTIONs of single
> property/asso queries, Zest OR-conditions are SQL UNIONs of single
> property/asso queries, and Zest NOT-conditions are SQL NOTs of single
> property/asso queries.
>
> 3. Keeping DB schema in sync with Zest application schema.
> This is very tough one - if someone adds/removes
> properties/associations, the database schema should be kept up-to-date.
> I remember having problems with this one - it also implies that
> you have to walk through the *whole* application structure in order to
> generate tables for all the properties/assos of all the entities
> visible to this indexing service.
>
>
Re: JPA?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
On 21.9.2016 0:08, Jiri Jetmar wrote:
> Independently of that that, things starts to be complicated in the SQL
> world with large data when you are submitting e.g. a inner JOIN statement
> in a transactional INSERT expression, where the tables are located (the
> data) on different nodes, simply because of a single node limit.
Well... there are ways to circumvent this, too. :)
Unfortunately, can't tell much more about that, as it is key concept of
the company where I work right now.
But very large and globally distributed RDBMS nodes, which don't lag
under high stress, are very much so possible!
> I would like to work on this task. Therefore smart ideas are highly welcome
> ! :-)
I am responsible for current SQL indexing code, and Paul is responsible
for current SQL entitystore code.
The indexing code is, unfortunately, quite spaghettified, and I would do
lots of things probably very differently now, than how I did them
several years ago.
I think one strategy to approach would be to re-write whole thing from
scratch, but keeping the main principles:
1. Each property and association should be stored in separate table.
This is required for the very dynamic nature of Zest queries.
This implicates that the whole DB schema will be owned by this
indexing service - it is not compatible with some external tool
generating tables (unless tables are, of course, in this exact required
format).
Example: if you have interface MyEntity { Property<String>
myProp(); Association<SomeOtherEntity> myAsso(); },
That would end up with 3 tables:
1.1. The table for implicit 'identity' property. Current indexing
code has separate DB-specific entity primary key which is of type 'long'
(64bit integer), since that makes foreign key references much faster and
compacter. Not sure if that is good idea anymore.
1.2. The table for 'myProp' property with 2 columns: foreign key
reference to identity table and actual column of type 'text' containing
the contents of 'myProp' property.
1.3. The table for 'myAsso' association with 2 columns: foreign key
reference to identity table of 'MyEntity' and foreign key reference to
identity table of 'SomeOtherEntity'. Obviously this covers the
'ManyAssociation's as well, if they are still present in Zest.
2. The SQL query will then be created by translating Zest query objects
into SQL query objects. You could also just build SQL strings, but it
would be quite nasty and most likely very unmaintaineable task - current
code uses java-sql-generator, which is my library on github, to create
SQL statements using DSL, and not by concatenating strings.
IIRC Zest AND-conditions are SQL INTERSECTIONs of single
property/asso queries, Zest OR-conditions are SQL UNIONs of single
property/asso queries, and Zest NOT-conditions are SQL NOTs of single
property/asso queries.
3. Keeping DB schema in sync with Zest application schema.
This is very tough one - if someone adds/removes
properties/associations, the database schema should be kept up-to-date.
I remember having problems with this one - it also implies that you
have to walk through the *whole* application structure in order to
generate tables for all the properties/assos of all the entities visible
to this indexing service.
Re: JPA?
Posted by Jiri Jetmar <ju...@gmail.com>.
JSON Indexing @Postgresql :
https://www.postgresql.org/docs/9.4/static/datatype-json.html
should work, looks promising so far.
The point with NoSQL is from my perspective i) the sheer amount of
accumulated data and ii) the need to "interact" with those data, like index
them, modify them, etc. Further I think we have to distinguish between
NoSQL - the concept, SQL - the CRUD(Q) expression and the relational
repository design. This are simply different things.
I mean the relational world using SQL expression for interactions with the
repository was just fine as long as one could build a "SQL-cluster" with
just few nodes. This is my personal experience based on some Oracle DB
setups where I was involved in. At the other side they are relational
setups with hundreds of nodes, e.g. the Skype Backoffice is (was) build on
top of Postgresql.
Independently of that that, things starts to be complicated in the SQL
world with large data when you are submitting e.g. a inner JOIN statement
in a transactional INSERT expression, where the tables are located (the
data) on different nodes, simply because of a single node limit. Together
with other relational features like the visibility of data (level of
isolation) things are that complicated that the whole thing does not scale
further. Therefore the NoSQL concept was born, where the applications are
(partially) responsible for the data management - a bit like in the pre-SQL
era. Due to this responsibility for data handling, NoSQL application are
more complex then purely SQL based and require longer time to market. So
from my perspective, whenever possible, one should use SQL and
transactional (ACID) repositories.
Now on Zest, we have the JSON serialization of Entities and we "degrade"
relational repositories to KeyValue storage engines. We should change it
and also allow external modeled schemas.
I would like to work on this task. Therefore smart ideas are highly welcome
! :-)
Cheers,
Jiri
2016-09-20 1:02 GMT+02:00 Niclas Hedhman <ni...@hedhman.org>:
> Right, but with a single JSON column you have reduced RDBMS to a KeyValue
> store. Can the JSON document be indexed in some intelligent way on
> Postgres?
>
> Isn't the SQL EntityStore already doing this in Zest?
>
> Love your; "YesSQL"
>
> Although I saw another funny meme;l
> NoSQL 2005 = No freaking SQL
> NoSQL 2010 = Not Only SQL
> NoSQL 2015 = Not Yet SQL
>
> Niclas
>
> On Mon, Sep 19, 2016 at 5:15 PM, Stanislav Muhametsin <
> stanislav.muhametsin@zest.mail.kapsi.fi> wrote:
>
> > On 19.9.2016 4:40, Niclas Hedhman wrote:
> >
> >> I agree that there is always a schema, and I don't think that anyone
> >> really
> >> disagrees with that. It is more a matter of "rigid" or "flexible"
> schema.
> >> The "rigid" world requires more process overhead to create and evolve,
> and
> >> over time end up with 500 columns that are mostly empty.
> >>
> >
> > Depends on how lazy the developer is in regard of keeping SQL schema up
> to
> > date with application schema.
> > That can be done with ALTER statements, which pretty much any relational
> > DB engine has these days.
> > The boundary between 'rigid' and 'flexible' schema is becoming blurry,
> > since e.g. PostgreSQL has been supporting 'json' column type now for a
> > while.
> > I think storing data in RDB using JSON should be investigated in Zest SQL
> > entitystore/indexing.
> >
> > From my own personal experience, I would never ever use NoSQL solutions
> > for any application I would develop - I think it is one of those useless
> > things spread by uneducated people.
> > Everything that NoSQL solution can do, the YesSQL solution can do better
> -
> > with proper tooling and modeling support, and with "think first, do then"
> > kind of approach.
> > Obviously, if one's working process lacks design/modeling/specsing of the
> > domain of one's application, NoSQL approach is more suitable.
> >
> >
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
Re: JPA?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
On 20.9.2016 2:02, Niclas Hedhman wrote:
>
> Right, but with a single JSON column you have reduced RDBMS to a
> KeyValue store. Can the JSON document be indexed in some intelligent
> way on Postgres?
Jiri managed to reply to this before me. :)
>
> Isn't the SQL EntityStore already doing this in Zest?
IIRC the SQL ES in Zest stringifies the Zest entity state (it will be
JSON string, but it doesn't utilize 'json' datatype, as such didn't
exist back then in any RDBMS).
>
> Love your; "YesSQL"
>
> Although I saw another funny meme;l
> NoSQL 2005 = No freaking SQL
> NoSQL 2010 = Not Only SQL
> NoSQL 2015 = Not Yet SQL
>
Hah, good one! :D
Re: JPA?
Posted by Niclas Hedhman <ni...@hedhman.org>.
Right, but with a single JSON column you have reduced RDBMS to a KeyValue
store. Can the JSON document be indexed in some intelligent way on Postgres?
Isn't the SQL EntityStore already doing this in Zest?
Love your; "YesSQL"
Although I saw another funny meme;l
NoSQL 2005 = No freaking SQL
NoSQL 2010 = Not Only SQL
NoSQL 2015 = Not Yet SQL
Niclas
On Mon, Sep 19, 2016 at 5:15 PM, Stanislav Muhametsin <
stanislav.muhametsin@zest.mail.kapsi.fi> wrote:
> On 19.9.2016 4:40, Niclas Hedhman wrote:
>
>> I agree that there is always a schema, and I don't think that anyone
>> really
>> disagrees with that. It is more a matter of "rigid" or "flexible" schema.
>> The "rigid" world requires more process overhead to create and evolve, and
>> over time end up with 500 columns that are mostly empty.
>>
>
> Depends on how lazy the developer is in regard of keeping SQL schema up to
> date with application schema.
> That can be done with ALTER statements, which pretty much any relational
> DB engine has these days.
> The boundary between 'rigid' and 'flexible' schema is becoming blurry,
> since e.g. PostgreSQL has been supporting 'json' column type now for a
> while.
> I think storing data in RDB using JSON should be investigated in Zest SQL
> entitystore/indexing.
>
> From my own personal experience, I would never ever use NoSQL solutions
> for any application I would develop - I think it is one of those useless
> things spread by uneducated people.
> Everything that NoSQL solution can do, the YesSQL solution can do better -
> with proper tooling and modeling support, and with "think first, do then"
> kind of approach.
> Obviously, if one's working process lacks design/modeling/specsing of the
> domain of one's application, NoSQL approach is more suitable.
>
>
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: JPA?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
On 19.9.2016 4:40, Niclas Hedhman wrote:
> I agree that there is always a schema, and I don't think that anyone really
> disagrees with that. It is more a matter of "rigid" or "flexible" schema.
> The "rigid" world requires more process overhead to create and evolve, and
> over time end up with 500 columns that are mostly empty.
Depends on how lazy the developer is in regard of keeping SQL schema up
to date with application schema.
That can be done with ALTER statements, which pretty much any relational
DB engine has these days.
The boundary between 'rigid' and 'flexible' schema is becoming blurry,
since e.g. PostgreSQL has been supporting 'json' column type now for a
while.
I think storing data in RDB using JSON should be investigated in Zest
SQL entitystore/indexing.
From my own personal experience, I would never ever use NoSQL solutions
for any application I would develop - I think it is one of those useless
things spread by uneducated people.
Everything that NoSQL solution can do, the YesSQL solution can do better
- with proper tooling and modeling support, and with "think first, do
then" kind of approach.
Obviously, if one's working process lacks design/modeling/specsing of
the domain of one's application, NoSQL approach is more suitable.
Re: JPA?
Posted by Niclas Hedhman <he...@gmail.com>.
Good thoughts.... I think it is a long path forward, and looking at JPA
"fragility" and I now doubt it is doable. With fragility, I mean how
sensitive JPA applications are to the execution model of JPA itself. Since
we don't follow that, I doubt we can make it work reliably.
In regards to SQL Schema to Zest generation; that is actually a really
persuasive argument. Are there any existing tools that let you work with an
SQL Schema and tell the tool what object model you want out of it?
I agree that there is always a schema, and I don't think that anyone really
disagrees with that. It is more a matter of "rigid" or "flexible" schema.
The "rigid" world requires more process overhead to create and evolve, and
over time end up with 500 columns that are mostly empty.
So, I am somewhat keen to explore the JDBC track further.
Niclas
On Sep 18, 2016 20:04, "Jiri Jetmar" <ju...@gmail.com> wrote:
> Hi guys,
>
> some lines on this JPA / Zest from me.
>
> The requirement to be able to use JPA on Zest has from my perspective three
> reasons:
>
> 1) Standardized, aka "JPA-way" to persist and query Entities (persistent
> Objects with Identities)
> 2) Be able to use a (relational-) schema that has been externally modeled
> and "materialized" in the repository, e.g. kind of ER-Model
> 3) Accessibility by other applications
>
> The reason for 1) is pretty straightforward - there are potentially
> millions of developers that knows what JPA is and how to use it. When you
> have a project X and the target platform is for what ever reasons Java and
> you have to store & query application related states - then you are looking
> for JPA support. That simple. On a long term, you are adding lot of
> complexity and issues using the nowadays popular Titanic-Class ORM mappers,
> but this "secrets" are mostly unknown by the decision-makers.
>
> From my own experience I know how hard it is to convince developers to use
> the "Zest" way of Domain Modeling and it takes months of hard work before
> the brain switches over to the Zest way. This is for large projects a
> serious issue and a huge risk that just few managers will take !
>
> The second reason is a bit more versatile. The last 20-30 years was the
> "relational era". The relational model was derived from the relational
> algebra and in the beginning people believed that it is impossible to
> develop it in software - but then the first relational databases was
> introduced..
>
> Now some people believe that we are living in the post relational era,
> where things are "schema-less" and therefore the NoSQL repositories are
> buzzing.. Then we have the CAP Theorem that indicates the limits of the
> "classical" repositores.. (but that has nothing to do directly with
> SQL/NoSQL repositories..)
>
> I personally believe that this schema-less argumentation is a just
> non-sense. There is always a schema. Either there is a single and unique
> place, a kind of "horizon of truth", like that what can be found in the
> typical ER-Models or the model is distributed across a number of places,
> like the application code or even in the developers brain (where is
> gradually evaporates). There is always a schema.
>
> The third point is that one wants sometimes to access (and in some case
> mutate) the models (and also the states) using 3th party tools. The reason
> can be e.g. to evolve the Domain Model using some Modelling Tools or even
> doing things like fixing issues directly in the repository - means changing
> data. The current approach in Zest where simply the serialized Objects as
> JSONs are stored in the repository is simply hard to be used by 3th party
> tools. I played with the JSON build-in type in Postgresql where then it is
> possible to CRUD using SQL the Zest Entities, but this is everything else
> then straightforward.
>
> If I had some free wishes on Zest, I would argue in this directions :
>
> i) Support external schema development, at least for the relational (SQL)
> repositories
> ii) Keep the ES / EI segregation as this is the coolest thing since sliced
> bread; optional support SQL based indexes.
> iii) Find a nice way to map Zest entities to the external schema; keep it
> simple, do not build a ORM Titanic
> iv) JPA is at least for me not required, plain old (and direct) SQL is fine
>
> With i) it would be possible to use Zest even on existing applications
> where the schema already exists and can not be changed. ii) is clear - the
> external index is just crazy cool. Optionally it would be nice to use a SQL
> Index, when one does not want to use ElasticSearch for Indexing iii)
> requires some nice/simply way how to model entities in Zest and map them to
> the external schema. iv) well, JPA is from my perceptive not a must. Plain
> old SQL is just fine. e.g. a tool like http://www.jooq.org/ can be used to
> generated the required SQL statements ?
>
> Cheers,
> Jiri
>
>
>
>
>
> 2016-09-05 3:49 GMT+02:00 Niclas Hedhman <he...@gmail.com>:
>
> > I am doing the Pet Clinic migration to Zest, partly to show how horrid
> the
> > HOA stuff is in comparison. In real world project, the situation is often
> > much worse as entities and data transfer classes are separate (not the
> case
> > in Pet Clinic)
> >
> > So, perhaps the answer is simply a migration strategy and a great vision
> > that reduces "clutter".
> >
> > Cheers
> > Niclas
> >
> > On Sep 5, 2016 08:02, "Paul Merlin" <pa...@apache.org> wrote:
> >
> > > Kent Sølvsten wrote:
> > >
> > >> This is a tough, but interesting question ......
> > >>
> > >> [SNIP]
> > >>
> > >> A few ideas:
> > >>
> > >> (1)
> > >> If the persistence engines in the entitystores are XA capable (such as
> > >> for JDBC/NEO4J entitystores) it should not be too difficult.
> > >> The basic idea would be to inject a DataSource provided by the
> appserver
> > >> into the EntityStore and let the EntityStore avoid committing -
> leaning
> > >> on commit of the global XA transaction
> > >> - and postpone notifying indexers until the global transaction is
> > >> committed (we should be able to listen to "commit events").
> > >>
> > >
> > > I did just that with the SQL ES a few years ago.
> > > I provided the SQL ES with a Connection that would delegate to a XA
> > > Connection and avoid committing, letting the responsibility to commit
> to
> > > the application server.
> > > It's working in production since then. But it's nasty and I've been
> > lucky,
> > > especially because of the presence of an Indexer in that application
> ....
> > >
> > > One simple way to look at JPA integration would be to simply use Zest
> > > without using its persistence related features.
> > > i.e. "if you use JPA, you don't need (cough!) Zest persistance"
> > >
> > > For this we could do various things to ease users pain:
> > > - show (sample, doc, whatever) how one can use composites for e.g. only
> > > their service layer while running in some horrid application server and
> > > persisting with JPA
> > > - have a Value <-> POJO converter, to move state from one world to the
> > > other if you really want to
> > >
> > > IOW, I'm note sure about the value of some JPA/UoW integration.
> > >
> > > My 0.2 cents
> > >
> > >
> >
>
Re: JPA?
Posted by Jiri Jetmar <ju...@gmail.com>.
Hi guys,
some lines on this JPA / Zest from me.
The requirement to be able to use JPA on Zest has from my perspective three
reasons:
1) Standardized, aka "JPA-way" to persist and query Entities (persistent
Objects with Identities)
2) Be able to use a (relational-) schema that has been externally modeled
and "materialized" in the repository, e.g. kind of ER-Model
3) Accessibility by other applications
The reason for 1) is pretty straightforward - there are potentially
millions of developers that knows what JPA is and how to use it. When you
have a project X and the target platform is for what ever reasons Java and
you have to store & query application related states - then you are looking
for JPA support. That simple. On a long term, you are adding lot of
complexity and issues using the nowadays popular Titanic-Class ORM mappers,
but this "secrets" are mostly unknown by the decision-makers.
From my own experience I know how hard it is to convince developers to use
the "Zest" way of Domain Modeling and it takes months of hard work before
the brain switches over to the Zest way. This is for large projects a
serious issue and a huge risk that just few managers will take !
The second reason is a bit more versatile. The last 20-30 years was the
"relational era". The relational model was derived from the relational
algebra and in the beginning people believed that it is impossible to
develop it in software - but then the first relational databases was
introduced..
Now some people believe that we are living in the post relational era,
where things are "schema-less" and therefore the NoSQL repositories are
buzzing.. Then we have the CAP Theorem that indicates the limits of the
"classical" repositores.. (but that has nothing to do directly with
SQL/NoSQL repositories..)
I personally believe that this schema-less argumentation is a just
non-sense. There is always a schema. Either there is a single and unique
place, a kind of "horizon of truth", like that what can be found in the
typical ER-Models or the model is distributed across a number of places,
like the application code or even in the developers brain (where is
gradually evaporates). There is always a schema.
The third point is that one wants sometimes to access (and in some case
mutate) the models (and also the states) using 3th party tools. The reason
can be e.g. to evolve the Domain Model using some Modelling Tools or even
doing things like fixing issues directly in the repository - means changing
data. The current approach in Zest where simply the serialized Objects as
JSONs are stored in the repository is simply hard to be used by 3th party
tools. I played with the JSON build-in type in Postgresql where then it is
possible to CRUD using SQL the Zest Entities, but this is everything else
then straightforward.
If I had some free wishes on Zest, I would argue in this directions :
i) Support external schema development, at least for the relational (SQL)
repositories
ii) Keep the ES / EI segregation as this is the coolest thing since sliced
bread; optional support SQL based indexes.
iii) Find a nice way to map Zest entities to the external schema; keep it
simple, do not build a ORM Titanic
iv) JPA is at least for me not required, plain old (and direct) SQL is fine
With i) it would be possible to use Zest even on existing applications
where the schema already exists and can not be changed. ii) is clear - the
external index is just crazy cool. Optionally it would be nice to use a SQL
Index, when one does not want to use ElasticSearch for Indexing iii)
requires some nice/simply way how to model entities in Zest and map them to
the external schema. iv) well, JPA is from my perceptive not a must. Plain
old SQL is just fine. e.g. a tool like http://www.jooq.org/ can be used to
generated the required SQL statements ?
Cheers,
Jiri
2016-09-05 3:49 GMT+02:00 Niclas Hedhman <he...@gmail.com>:
> I am doing the Pet Clinic migration to Zest, partly to show how horrid the
> HOA stuff is in comparison. In real world project, the situation is often
> much worse as entities and data transfer classes are separate (not the case
> in Pet Clinic)
>
> So, perhaps the answer is simply a migration strategy and a great vision
> that reduces "clutter".
>
> Cheers
> Niclas
>
> On Sep 5, 2016 08:02, "Paul Merlin" <pa...@apache.org> wrote:
>
> > Kent Sølvsten wrote:
> >
> >> This is a tough, but interesting question ......
> >>
> >> [SNIP]
> >>
> >> A few ideas:
> >>
> >> (1)
> >> If the persistence engines in the entitystores are XA capable (such as
> >> for JDBC/NEO4J entitystores) it should not be too difficult.
> >> The basic idea would be to inject a DataSource provided by the appserver
> >> into the EntityStore and let the EntityStore avoid committing - leaning
> >> on commit of the global XA transaction
> >> - and postpone notifying indexers until the global transaction is
> >> committed (we should be able to listen to "commit events").
> >>
> >
> > I did just that with the SQL ES a few years ago.
> > I provided the SQL ES with a Connection that would delegate to a XA
> > Connection and avoid committing, letting the responsibility to commit to
> > the application server.
> > It's working in production since then. But it's nasty and I've been
> lucky,
> > especially because of the presence of an Indexer in that application ....
> >
> > One simple way to look at JPA integration would be to simply use Zest
> > without using its persistence related features.
> > i.e. "if you use JPA, you don't need (cough!) Zest persistance"
> >
> > For this we could do various things to ease users pain:
> > - show (sample, doc, whatever) how one can use composites for e.g. only
> > their service layer while running in some horrid application server and
> > persisting with JPA
> > - have a Value <-> POJO converter, to move state from one world to the
> > other if you really want to
> >
> > IOW, I'm note sure about the value of some JPA/UoW integration.
> >
> > My 0.2 cents
> >
> >
>
Re: JPA?
Posted by Niclas Hedhman <he...@gmail.com>.
I am doing the Pet Clinic migration to Zest, partly to show how horrid the
HOA stuff is in comparison. In real world project, the situation is often
much worse as entities and data transfer classes are separate (not the case
in Pet Clinic)
So, perhaps the answer is simply a migration strategy and a great vision
that reduces "clutter".
Cheers
Niclas
On Sep 5, 2016 08:02, "Paul Merlin" <pa...@apache.org> wrote:
> Kent Sølvsten wrote:
>
>> This is a tough, but interesting question ......
>>
>> [SNIP]
>>
>> A few ideas:
>>
>> (1)
>> If the persistence engines in the entitystores are XA capable (such as
>> for JDBC/NEO4J entitystores) it should not be too difficult.
>> The basic idea would be to inject a DataSource provided by the appserver
>> into the EntityStore and let the EntityStore avoid committing - leaning
>> on commit of the global XA transaction
>> - and postpone notifying indexers until the global transaction is
>> committed (we should be able to listen to "commit events").
>>
>
> I did just that with the SQL ES a few years ago.
> I provided the SQL ES with a Connection that would delegate to a XA
> Connection and avoid committing, letting the responsibility to commit to
> the application server.
> It's working in production since then. But it's nasty and I've been lucky,
> especially because of the presence of an Indexer in that application ....
>
> One simple way to look at JPA integration would be to simply use Zest
> without using its persistence related features.
> i.e. "if you use JPA, you don't need (cough!) Zest persistance"
>
> For this we could do various things to ease users pain:
> - show (sample, doc, whatever) how one can use composites for e.g. only
> their service layer while running in some horrid application server and
> persisting with JPA
> - have a Value <-> POJO converter, to move state from one world to the
> other if you really want to
>
> IOW, I'm note sure about the value of some JPA/UoW integration.
>
> My 0.2 cents
>
>
Re: JPA?
Posted by Paul Merlin <pa...@apache.org>.
Kent S�lvsten wrote:
> This is a tough, but interesting question ......
>
> [SNIP]
>
> A few ideas:
>
> (1)
> If the persistence engines in the entitystores are XA capable (such as
> for JDBC/NEO4J entitystores) it should not be too difficult.
> The basic idea would be to inject a DataSource provided by the
> appserver
> into the EntityStore and let the EntityStore avoid committing - leaning
> on commit of the global XA transaction
> - and postpone notifying indexers until the global transaction is
> committed (we should be able to listen to "commit events").
I did just that with the SQL ES a few years ago.
I provided the SQL ES with a Connection that would delegate to a XA
Connection and avoid committing, letting the responsibility to commit to
the application server.
It's working in production since then. But it's nasty and I've been
lucky, especially because of the presence of an Indexer in that
application ....
One simple way to look at JPA integration would be to simply use Zest
without using its persistence related features.
i.e. "if you use JPA, you don't need (cough!) Zest persistance"
For this we could do various things to ease users pain:
- show (sample, doc, whatever) how one can use composites for e.g. only
their service layer while running in some horrid application server and
persisting with JPA
- have a Value <-> POJO converter, to move state from one world to the
other if you really want to
IOW, I'm note sure about the value of some JPA/UoW integration.
My 0.2 cents