You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cayenne.apache.org by Andrus Adamchik <an...@objectstyle.org> on 2015/12/05 09:33:30 UTC

Re: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP

Yeah, Hessian looks dead. 

Protobuf is widely used as a wire protocol for NoSQL databases, etc. From that perspective it is a very good candidate. Though IIRC it requires some form of a "schema", while ROP relies on a generic serialization approach. 

My earlier ideas of using LinkRest for ROP are probably not (yet?) practical. LinkRest stays away from generic data operations (in other words, each LR operation is rooted in some entity). Though we can create an extension that changes this assumption, and still use the underlying machinery.

I think the cheapest option for us is to use Java built-in serialization. It kind of works already with a few quirks. The downside is that the client must be Java, but this is a reality of ROP anyways. 

Andrus


> On Nov 24, 2015, at 4:22 AM, Ari Maniatis (JIRA) <ji...@apache.org> wrote:
> 
> 
>    [ https://issues.apache.org/jira/browse/CAY-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023542#comment-15023542 ] 
> 
> Ari Maniatis commented on CAY-2038:
> -----------------------------------
> 
> I found an old post on the Hessian list [1], but with no reply. Looks like we should consider Hessian effectively abandoned. To fix this and also review the library for serialisation security issues (like was found for the Java serialisation) we should decide whether we:
> 
> 1. Fix Hessian and fork it
> 2. Move to something else
> 
> If we move, some options are here: http://docs.spring.io/spring/docs/current/spring-framework-reference/html/remoting.html  Also, there is a Google library https://developers.google.com/protocol-buffers/ which is interesting and licensed with something that looks like a simple BSD license.
> 
> I think our criteria should be:
> 
> * not XML (to keep the size of the data to a minimum)
> * over HTTP (that's a significant benefit of the current approach since it makes things like SSL easy)
> 
> Against some tests https://github.com/eishay/jvm-serializers/wiki protocol-buffers look to be smaller/faster than Hessian. And the community appears to be alive and well: https://groups.google.com/forum/#!forum/protobuf  However, it appears that protocol-buffers require a .proto file to define the object serialisation so either that would need to be generated dynamically at runtime or by the cgen velocity scripts. I think.
> 
> If we stay, one of the bigger problems is that Caucho have historically rarely accepted patches or responded to community discussions. There is no bug tracker, no official separate source control. Just a module buried inside Resin which doesn't get touched much and only sporadically is released to maven. Having said that, perhaps the patches are simple.
> 
> Someone talks about using protocol-buffers with Spring HTTP invoker: http://www.eishay.com/2008/11/using-spring-rpc-for-protobuf-transport.html  but I'm not deep enough into this yet to understand the benefits.
> 
> 
> Thoughts?
> 
> [1] http://maillist.caucho.com/pipermail/hessian-interest/2014-June/thread.html#1150
> 
>> Hessian serialization error when using JSR-310 Date types with ROP
>> ------------------------------------------------------------------
>> 
>>                Key: CAY-2038
>>                URL: https://issues.apache.org/jira/browse/CAY-2038
>>            Project: Cayenne
>>         Issue Type: Bug
>>         Components: ROP
>>   Affects Versions: 4.0.M3
>>           Reporter: Dzmitry Kazimirchyk
>>           Assignee: Savva Kolbachev
>>        Attachments: jsr310-dates-rop.patch
>> 
>> 
>> Getting StackOverflowError during hessian serialization when querying for entities which have LocalDate, LocalDateTime or LocaTime type properties.
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)

Re: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP

Posted by Michael Gentry <mg...@masslight.net>.

HTTP/2 would also apply to a JS implementation, too.

On Tue, Dec 8, 2015 at 7:40 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:

> I've just been talking to Dima about this and I think we are in agreement
> that the first most important issue for us is not the serialisation, but
> rather introducing clean separation between three things:
>
> * cayenne
> * hessian
> * transport (http)
>
> Currently, those things are deeply mixed up, and that makes it somewhat
> tricky to add gzip compression to the connection or switch from HTTP to
> HTTP2. We believe that HTTP2 will give us significant performance
> improvements for the type of workloads we see. Latency is very important
> because of the large number of small requests Cayenne will make (for
> example, every time a bit of UI needs to follow a join).
>
> So even keeping Hessian in place, we need to make the parts more
> pluggable. I'm interested in any opinions people might have on this work. I
> am assuming we'll look at the Cayenne DI library to provide simple ways to
> replace the serialisation library or the transport choice.
>
> Ari
>
>
> On 7/12/2015 10:01pm, Andrus Adamchik wrote:
> >
> >> On Dec 7, 2015, at 12:53 PM, Aristedes Maniatis <ar...@maniatis.org>
> wrote:
> >>
> >> Also, I just saw this: http://www.grpc.io/ which is licensed under a
> BSD-style three clause thing. That looks like it overlaps linkrest a bit in
> features, but perhaps we can learn a bit about their approaches to RPC
> service calls which need to wrap around the actual Cayenne objects.
> >
> >
> > LinkRest objective is building RESTful APIs, which is the opposite of
> RPC. ROP though (at least currently) is RPC. So yeah, there may be
> something there.
> >
> > Andrus
> >
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

Re: ROP

Posted by Andrus Adamchik <an...@objectstyle.org>.

Proxy was a raw idea. Physically separating non-object parts of Cayenne from "business" objects and associated business logic is still very appealing. This is about encapsulating ~60-70% of Cayenne stack (mapping model, SQL query translation, caching, etc.) into a web service with clear simple interface. The public API of the proxy will be (serialized) objects though (JSON, protobuf). So that's your generic ORM in a box, serving your object model, but free from your application java dependencies. So the consumers will need to do no ORM translation. 

Anyways, I need to sit on this for some time. I feel there's something there in this approach. This is a solution for something, but not for the problem at hand :) Protocol on the other hand (in a Hessian-free future) will likely be the same for both "proxy" and "ROP" designs. So I am more interested in the protocol right now.

Andrus


> On Dec 14, 2015, at 1:05 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
> 
> On 14/12/2015 6:56pm, Andrus Adamchik wrote:
>> For technologies like Swift that don't have native drivers yet it provides JDBC driver.
> 
> Even Swift already has native database drivers, according to my quick search of github.
> 
> 
> On 14/12/2015 7:00pm, Andrus Adamchik wrote:
>> Though I guess the main motivation for Java ROP now is the ability to run code both on the client and on the server, right?
> 
> 
> Absolutely. Lifecycle events are important so that the server doesn't have to trust the client to perform validation and authorisation, and also trigger other processes like audit trails.
> 
> Also important are clever query concepts like pagination and prefetches.
> 
> What could be better is integrated support for distributed caches and cache invalidation. Or at least the hooks to make it easier to leverage ActiveMQ/EHCache/etc as a way to do these things.
> 
> 
> 
> On 14/12/2015 7:18pm, Andrus Adamchik wrote:
>> Another thing that I hoped to achieve in Cayenne is seamless reverse-engineering flow. The biggest hurdle in ORM is the constant need to sync up multiple layers: DB -> ORM -> code. Annotation-based frameworks combine ORM with code, somewhat easing the pain (though making things dirty in the process). Our full separation of all 3 layers always seemed right. But now I think of it as an advantage not being utilized. 
> 
> 
> I guess I'm not seeing that pain. I really should get around to cleaning up and contributing my cgen plugin for gradle, but really this should all be automated by the build tool.
> 
> Perhaps we should work on some liquibase integration to help with the database update process. We do this by hand, but it would be nice for cgen to detect the changes and append a liquibase XML snippet.
> 
> 
>> So the hope is that we can do better, and perhaps a zero-manitenance proxy is the answer to the sync problem as well. But maybe I am on the wrong track completely by mixing modeling and query execution aspects here. 
>> 
>> Anyways, just thinking out loud :)
> 
> 
> If a proxy layer doesn't let you run code, then what does it do? You have Hessian serialisation instead of SQL serialisation. You have HTTP instead of JDBC. But then what?
> 
> Certainly the limitation of ROP is that the client is currently limited to Java, which means roughly Swing/JavaFX. Opening it up to other UI choices means porting a great part of Cayenne (all the client bits are still quite a lot) to Swift or Android or .NET or something. But (to bring this back on topic) protobuf does help there since the client and server can be different languages.
> 
> 
> Ari
> 
> 
> -- 
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: ROP

Posted by Aristedes Maniatis <ar...@maniatis.org>.

On 14/12/2015 6:56pm, Andrus Adamchik wrote:
> For technologies like Swift that don't have native drivers yet it provides JDBC driver.

Even Swift already has native database drivers, according to my quick search of github.

On 14/12/2015 7:00pm, Andrus Adamchik wrote:
> Though I guess the main motivation for Java ROP now is the ability to run code both on the client and on the server, right?

Absolutely. Lifecycle events are important so that the server doesn't have to trust the client to perform validation and authorisation, and also trigger other processes like audit trails.

Also important are clever query concepts like pagination and prefetches.

What could be better is integrated support for distributed caches and cache invalidation. Or at least the hooks to make it easier to leverage ActiveMQ/EHCache/etc as a way to do these things.

On 14/12/2015 7:18pm, Andrus Adamchik wrote:
> Another thing that I hoped to achieve in Cayenne is seamless reverse-engineering flow. The biggest hurdle in ORM is the constant need to sync up multiple layers: DB -> ORM -> code. Annotation-based frameworks combine ORM with code, somewhat easing the pain (though making things dirty in the process). Our full separation of all 3 layers always seemed right. But now I think of it as an advantage not being utilized. 

I guess I'm not seeing that pain. I really should get around to cleaning up and contributing my cgen plugin for gradle, but really this should all be automated by the build tool.

Perhaps we should work on some liquibase integration to help with the database update process. We do this by hand, but it would be nice for cgen to detect the changes and append a liquibase XML snippet.

> So the hope is that we can do better, and perhaps a zero-manitenance proxy is the answer to the sync problem as well. But maybe I am on the wrong track completely by mixing modeling and query execution aspects here. 
> 
> Anyways, just thinking out loud :)

If a proxy layer doesn't let you run code, then what does it do? You have Hessian serialisation instead of SQL serialisation. You have HTTP instead of JDBC. But then what?

Certainly the limitation of ROP is that the client is currently limited to Java, which means roughly Swing/JavaFX. Opening it up to other UI choices means porting a great part of Cayenne (all the client bits are still quite a lot) to Swift or Android or .NET or something. But (to bring this back on topic) protobuf does help there since the client and server can be different languages.

Ari

-- 
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: ROP [Was: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP]

Posted by Andrus Adamchik <an...@objectstyle.org>.

Sorry for the series of messages. Trying to brainstorm this.

Another thing that I hoped to achieve in Cayenne is seamless reverse-engineering flow. The biggest hurdle in ORM is the constant need to sync up multiple layers: DB -> ORM -> code. Annotation-based frameworks combine ORM with code, somewhat easing the pain (though making things dirty in the process). Our full separation of all 3 layers always seemed right. But now I think of it as an advantage not being utilized. 

So the hope is that we can do better, and perhaps a zero-manitenance proxy is the answer to the sync problem as well. But maybe I am on the wrong track completely by mixing modeling and query execution aspects here. 

Anyways, just thinking out loud :)

Andrus




> On Dec 14, 2015, at 11:00 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
> 
> Though I guess the main motivation for Java ROP now is the ability to run code both on the client and on the server, right? 
> 
> If so, let's focus on the data protocol, which can then be used either with the current ROP or a stripped down proxy.
> 
> Andrus
> 
> 
>> On Dec 14, 2015, at 10:56 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
>> 
>> For technologies like Swift that don't have native drivers yet it provides JDBC driver.
>> 
>> And it provides ORM mapping and object queries and updates for everyone.
>> 
>> Andrus
>> 
>>> On Dec 14, 2015, at 4:41 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>>> 
>>> On 13/12/2015 7:00am, Andrus Adamchik wrote:
>>>> Another possible direction with ROP is to strip it down to a very simple and performant "ORM proxy":
>>>> 
>>>> * define a protocol for protobuf and JSON serialization (still need to look at Kryo). Limit it to the smallest usable subset of queries (EJBQL is a good candidate ... any other object select can be translated to it) and generic update operations.
>>>> * make the server generic - use generic entities instead of precompiled Java classes.
>>>> * remove stateful ObjectContext layer from the server. Perhaps serialize DataRows directly (can be tricky with prefetching, but doable). Will still need request-scoped ObjectContext for commits I guess.
>>> 
>>> 
>>> I'm not really understanding the point of an ORM proxy. What does it give you that JDBC doesn't?
>>> 
>>> Ari
>>> 
>>> 
>>> -- 
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>> 
>

Re: ROP [Was: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP]

Posted by Andrus Adamchik <an...@objectstyle.org>.

Though I guess the main motivation for Java ROP now is the ability to run code both on the client and on the server, right? 

If so, let's focus on the data protocol, which can then be used either with the current ROP or a stripped down proxy.

Andrus


> On Dec 14, 2015, at 10:56 AM, Andrus Adamchik <an...@objectstyle.org> wrote:
> 
> For technologies like Swift that don't have native drivers yet it provides JDBC driver.
> 
> And it provides ORM mapping and object queries and updates for everyone.
> 
> Andrus
> 
>> On Dec 14, 2015, at 4:41 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>> 
>> On 13/12/2015 7:00am, Andrus Adamchik wrote:
>>> Another possible direction with ROP is to strip it down to a very simple and performant "ORM proxy":
>>> 
>>> * define a protocol for protobuf and JSON serialization (still need to look at Kryo). Limit it to the smallest usable subset of queries (EJBQL is a good candidate ... any other object select can be translated to it) and generic update operations.
>>> * make the server generic - use generic entities instead of precompiled Java classes.
>>> * remove stateful ObjectContext layer from the server. Perhaps serialize DataRows directly (can be tricky with prefetching, but doable). Will still need request-scoped ObjectContext for commits I guess.
>> 
>> 
>> I'm not really understanding the point of an ORM proxy. What does it give you that JDBC doesn't?
>> 
>> Ari
>> 
>> 
>> -- 
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

Re: ROP [Was: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP]

Posted by Andrus Adamchik <an...@objectstyle.org>.

For technologies like Swift that don't have native drivers yet it provides JDBC driver.

And it provides ORM mapping and object queries and updates for everyone.

Andrus

> On Dec 14, 2015, at 4:41 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:
> 
> On 13/12/2015 7:00am, Andrus Adamchik wrote:
>> Another possible direction with ROP is to strip it down to a very simple and performant "ORM proxy":
>> 
>> * define a protocol for protobuf and JSON serialization (still need to look at Kryo). Limit it to the smallest usable subset of queries (EJBQL is a good candidate ... any other object select can be translated to it) and generic update operations.
>> * make the server generic - use generic entities instead of precompiled Java classes.
>> * remove stateful ObjectContext layer from the server. Perhaps serialize DataRows directly (can be tricky with prefetching, but doable). Will still need request-scoped ObjectContext for commits I guess.
> 
> 
> I'm not really understanding the point of an ORM proxy. What does it give you that JDBC doesn't?
> 
> Ari
> 
> 
> -- 
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: ROP [Was: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP]

Posted by Aristedes Maniatis <ar...@maniatis.org>.

On 13/12/2015 7:00am, Andrus Adamchik wrote:
> Another possible direction with ROP is to strip it down to a very simple and performant "ORM proxy":
> 
> * define a protocol for protobuf and JSON serialization (still need to look at Kryo). Limit it to the smallest usable subset of queries (EJBQL is a good candidate ... any other object select can be translated to it) and generic update operations.
> * make the server generic - use generic entities instead of precompiled Java classes.
> * remove stateful ObjectContext layer from the server. Perhaps serialize DataRows directly (can be tricky with prefetching, but doable). Will still need request-scoped ObjectContext for commits I guess.


I'm not really understanding the point of an ORM proxy. What does it give you that JDBC doesn't?

Ari


-- 
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

ROP [Was: [jira] [Commented] (CAY-2038) Hessian serialization error when using JSR-310 Date types with ROP]

Posted by Andrus Adamchik <an...@objectstyle.org>.

Another possible direction with ROP is to strip it down to a very simple and performant "ORM proxy":

* define a protocol for protobuf and JSON serialization (still need to look at Kryo). Limit it to the smallest usable subset of queries (EJBQL is a good candidate ... any other object select can be translated to it) and generic update operations.
* make the server generic - use generic entities instead of precompiled Java classes.
* remove stateful ObjectContext layer from the server. Perhaps serialize DataRows directly (can be tricky with prefetching, but doable). Will still need request-scoped ObjectContext for commits I guess.

While this will take away the ability to add business logic to the server, it has a number of advantages too:

* no more modeling the client and server separately and messing with 2 sets of classes
* possible to package server as a reusable Docker image that will automatically run against any DB (perhaps with auto reverse engineering of the model [1]).
* simplicity of writing a client. E.g. Apple just open-sourced Swift. To the best of my knowledge Swift has no decent DB drivers or ORM. A reusable Java/Cayenne proxy with clear protocol would allow writing a Swift ORM with a fairly low effort. Same goes for other languages.
* even LinkRest can probably use such an ORM proxy via a Java ROP client

Thoughts?
Andrus

[1] Imagine you do this:

# Docker used as an example; proxy can be packaged in many different ways
$ docker pull cayenne/proxy

# Start the proxy. On first run it reverse-engineers
# the DB and drops the XML somewhere
$ docker -p 1234:1234 run cayenne/proxy <db_url>

# you open XML with the Modeler,
# tweak it to your liking, save and put in Git
# the server auto-reloads
# On subsequnt runs, DB changes are detected and merged into the Model

# "client" is the client to the proxy;
# it can actually be a server-side app
cd client/
mvn cgen-from-proxy

# now you have your app...