You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cayenne.apache.org by Savva Kolbachev <s....@gmail.com> on 2016/05/05 09:35:55 UTC
Cayenne ROP Improvements
Hi All,
As you already may know, Dima did a lot of refactoring and made
connectivity layer/serialization pluggable after 4.0.M3 release. So now
ROPConnector and ROPSerializationService implementations responsible for
connectivity layer and serialization accordingly.
Also we moved from Hessian as a default implementation for connectivity
layer to the standard Java implementation. Although, Hessian is still used
for serialization by default.
I've been working on some ROP improvements recently and I want to hear your
opinion on these:
1. Jetty HTTP/2 Client for connectivity layer.
related source:
http://www.eclipse.org/jetty/documentation/current/http2.html
related JIRA: https://issues.apache.org/jira/browse/CAY-2076
related pull-request: https://github.com/apache/cayenne/pull/96
So if you want to use HTTP/2 in your ROP application all you need is just
add Http2ClientModule to Client Runtime and Jetty will take care of all
things related to HTTP/2. This is a deafult solution and it uses prior
knowledge about HTTP/2 on server-side so you could use it without providing
ALPN. If you want to handle some things related to HTTP/2, there are some
other options. More information you could find in related JIRA ticket.
2. Protostuff for serialization.
related source: http://www.protostuff.io/
related JIRA: https://issues.apache.org/jira/browse/CAY-2083
related pull-request: https://github.com/apache/cayenne/pull/102
Protostuff (licensed under Apache 2.0 licence) is based on Google's
Protocol-Buffers (Protobuf) but has some optimizations and some cool things
like runtime serialization graph of objects (like Hessian). It also could
generate schema on runtime so we shouldn't define .proto files although it
might increase efficiency. It works faster than Hessian and could handle
Java8 Date and Time types. Here is some benchmarks. Take a look at Full
Object Graph Serializers section.
http://hperadin.github.io/jvm-serializers-report/report.html
https://github.com/eishay/jvm-serializers/wiki
To use it you should add ProtostuffModule for both Server and Client
runtime configurations. And also add
-Dprotostuff.runtime.collection_schema_on_repeated_fields=true as VM option
because some of Cayenne objects have cyclic collection fields (more info:
http://www.protostuff.io/documentation/object-graphs/)
If anybody has some suggestions or questions, please, feel free to comment.
If there are no objections, I'm going to merge it to master branch soon.
Also I want to suggest to use Protostuff serialization instead of Hessian
by default. What do you think about it?
--
Thanks and Regards
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
> On May 5, 2016, at 5:35 AM, Savva Kolbachev <s....@gmail.com> wrote:
> Also we moved from Hessian as a default implementation for connectivity
> layer to the standard Java implementation. Although, Hessian is still used
> for serialization by default.
So which one is the default, Hessian or Java?
> Also I want to suggest to use Protostuff serialization instead of Hessian
> by default. What do you think about it?
I don't have a problem with Protostuff being a recommended default, but for dependency management purposes I'd rather we split all third-party integrations in separate modules, and use whatever provider is hooked up in runtime. Kind of what we do with Joda/Java8 extensions.
Going forward we may simplify dependency integration to be "drag and drop" by using Java ServiceLoader API. But even without it, it is very simple to switch between "providers" of various things.
Andrus
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
> Please, let me know what do you think about all this things?
Sounds reasonable to me.
> Finally, as Cayenne is becoming more modular, it will be helpful to have
> server and client test jars. It is sometimes frustrating to write
> integration tests for new modules.
No objections here. Let's make sure we set deploy plugin "skip" property to "true". We don't want them to be part of releases.
Andrus
> On Jun 13, 2016, at 11:40 AM, Savva Kolbachev <s....@gmail.com> wrote:
>
> After some time I think that my current HTTP/2 ROP implementation for
> Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> it as clear as possible, so it will be simple to use out of box.
> So, I suggest to rename module to something like "cayenne-jetty-client" and
> implement only two ROPConnectors: JettyHttpConnector and
> JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> high-level Jetty Client API.
> If someone wants something specific, they could provide their own similar
> implementation. I think the most common case for this is to share one
> HttpClient between ROP and other parts of a project.
> In addition, we will be open to make more modules for other clients, like
> Apache HTTP Client, Netty, OkHttp and etc.
>
> BTW, I got rid of Hessian ServiceContext in ROPServlet [2] by implementing
> something very similar in Cayenne
> https://github.com/apache/cayenne/commit/85852f92ee05f5e1ae70dae7315beda32f2b4077
> So now Hessian is used only for serialization purposes. As we want to move
> all ROP functionality to the separate module and Hessian serialization has
> problems with Java8, we should decide which serialization service should be
> used by default. For example Protostuff or Java serialization. Hessian one
> we could remain as pluggable.
>
> Finally, as Cayenne is becoming more modular, it will be helpful to have
> server and client test jars. It is sometimes frustrating to write
> integration tests for new modules.
>
> Please, let me know what do you think about all this things?
>
> [1] https://github.com/apache/cayenne/pull/96
> [2] https://issues.apache.org/jira/browse/CAY-2090
>
>
> 2016-05-06 3:29 GMT+03:00 Andrus Adamchik <an...@objectstyle.org>:
>
>> Good idea. The server part should probably be split from cayenne-server.
>>
>> Andrus
>>
>>> On May 5, 2016, at 8:26 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>>>
>>> Perhaps the whole of ROP becomes an optional module?
>>>
>>> Ari
>>>
>>>
>>> On 6/05/2016 10:18am, Andrus Adamchik wrote:
>>>> It's been a while since I touched the ROP code. Back in the day Java
>> serialization "kind of worked", but not completely. So you are probably
>> right that it is not a real option. I am just trying to avoid new
>> dependencies (even optional) on third-party libs in the Cayenne core. So
>> perhaps we can simply leave out any "default" serialization and always
>> require an explicit serialization provider.
>>>>
>>>> Andrus
>>>>
>>>>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis <ar...@maniatis.org>
>> wrote:
>>>>>
>>>>> Maybe I'm not understanding correctly, but I don't think Java
>> serialisation has been implemented in ROP. The work Dima did was to move
>> away from the Hessian servlet stuff for making the HTTP connection, to
>> plain Java with the option for plugging in Jetty libraries for HTTP/2.
>>>>>
>>>>> The work Savva did just now was to use protostuff for serialisation,
>> but I'm not sure what's now needed if we wanted plain Java serialisation or
>> whether that's even possible without some sort of library to handle an
>> object graph with cycles.
>>>>>
>>>>> Or at least that's my understanding.
>>>>>
>>>>> Ari
>>>>>
>>>>>
>>>>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>>>>>> Thanks for clarification. I would say use Java serialization as a
>> default, and make it easy to plugin Hessian and Protostuff as separate
>> modules.
>>>>>>
>>>>>> A.
>>>>>>
>>>>>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com>
>> wrote:
>>>>>>>
>>>>>>> Hi Andrus,
>>>>>>>
>>>>>>>> So which one is the default, Hessian or Java?
>>>>>>> We still use Hessian for serialization by default
>>>>>>>
>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>>>>>> But we use java.net.URLConnection for establish connection and
>> sending
>>>>>>> messages from client to server
>>>>>>>
>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>>>>>> So we have escaped from Hessian only in connectivity layer.
>>>>>>>
>>>>>>>> I don't have a problem with Protostuff being a recommended default,
>> but
>>>>>>> for dependency management purposes I'd rather we split all
>> third-party
>>>>>>> integrations in separate modules, and use whatever provider is
>> hooked up in
>>>>>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>>>>>> I already did it in this way. I created separate module for
>> Protostuff
>>>>>>> serialization.
>>>>>>>
>>>>>>> As Hessian serialization has some troubles with Java8 types and
>> provide
>>>>>>> less efficient serialization than Protostuff, I suggest to use
>> Protostuff
>>>>>>> as default serialization service or to use Java serialization. So I
>> just
>>>>>>> suggest to escape from Hessian :)
>>>>>>>
>>>>>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>>>>>>>
>>>>>>>> Hi Ari,
>>>>>>>>
>>>>>>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>>>>>>> example Serializers (no shared refs) and Cross Lang Binary
>> Serializers
>>>>>>>> sections here
>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>>>>
>>>>>>>> In our case we need to serialize graph of objects (Full Object Graph
>>>>>>>> Serializers section in link above). Protobuf can't do it out of the
>> box
>>>>>>>> but Protostuff can. In my implementation I use
>> protostuff-graph-runtime
>>>>>>>> which generates a schema from objects at runtime and caches it.
>>>>>>>>
>>>>>>>> Protostuff schema is something like .proto files but in Java:
>>>>>>>> http://www.protostuff.io/documentation/schema/
>>>>>>>> Runtime schema:
>> http://www.protostuff.io/documentation/runtime-schema/
>>>>>>>>
>>>>>>>> As you could see in benchmarks there is a small difference in
>> efficiency
>>>>>>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>>>>>>> overhead is related to runtime schema generation. The size penalty
>> is that
>>>>>>>> Protostuff adds class name for objects and than uses those for find
>>>>>>>> appropriate classes via reflection.
>>>>>>>> Hessian also adds fields names so the size of Hessian serialization
>> is
>>>>>>>> much bigger. In my small example with selection of 6 objects Hessian
>>>>>>>> serialization size is more than 2400 bytes while Protostuff runtime
>> is
>>>>>>>> about 800 bytes.
>>>>>>>>
>>>>>>>> If we don't want to have ser/deser and size overhead we could find
>> a way
>>>>>>>> to generate schemas via Velocity. And we should provide schemas for
>> some
>>>>>>>> Cayenne classes. But it will require a lot of efforts.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>>>>>>>
>>>>>>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>>>>>>>> Protostuff (licensed under Apache 2.0 licence) is based on
>> Google's
>>>>>>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some
>> cool
>>>>>>>>> things
>>>>>>>>>> like runtime serialization graph of objects (like Hessian). It
>> also
>>>>>>>>> could
>>>>>>>>>> generate schema on runtime so we shouldn't define .proto files
>> although
>>>>>>>>> it
>>>>>>>>>> might increase efficiency. It works faster than Hessian and could
>> handle
>>>>>>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look
>> at Full
>>>>>>>>>> Object Graph Serializers section.
>>>>>>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>>>>>> https://github.com/eishay/jvm-serializers/wiki
>>>>>>>>>
>>>>>>>>> According to those benchmarks there appears to be no performance
>> or size
>>>>>>>>> penalty to using protostuff over protobuffers. Am I reading that
>> right?
>>>>>>>>>
>>>>>>>>> I don't really understand... doesn't the serialiser have to
>> construct a
>>>>>>>>> .proto definition and then include it in the message? So shouldn't
>> it be
>>>>>>>>> faster/smaller to predefine these?
>>>>>>>>>
>>>>>>>>> If we did, we could create them with velocity in the same way we
>> create
>>>>>>>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>>>>>>>
>>>>>>>>> Ari
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -------------------------->
>>>>>>>>> Aristedes Maniatis
>>>>>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards
>>>>>>>> Savva Kolbachev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards
>>>>>>> Savva Kolbachev
>>>>>>
>>>>>
>>>>> --
>>>>> -------------------------->
>>>>> Aristedes Maniatis
>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>
>>>
>>> --
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
>>
>
>
> --
> Thanks and Regards
> Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
Nice!
With Thrift and Protobuf gathering more acceptance as JSON alternatives for web services in general, perhaps ROP has a brighter future then we thought :)
Andrus
> On Jun 30, 2016, at 6:36 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
> On 30/06/2016 7:21pm, Savva Kolbachev wrote:
>> BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
>> how it's going. If it's OK, I'll notify user-list about this features.
>
> We've actually just deployed the upgrade into our staging environment are will likely roll out the first customers early next week. If our preliminary (very non-scientific) results are anything to go by, almost everything is faster across a slow link.
>
> Some parts of the UI (opening a certain window) went from 14 seconds to 5 seconds. Another from 87 seconds to 10 seconds. And one from 8 to 5. The difference is likely how effectively we used prefetches already. In some places where we follow a lot of relation joins, we could easily generate 50 separate fetches across the network. And on a slow link (say over 40ms latency and bandwidth under 2Mb/s) with SSL handshaking that adds up very quickly.
>
> I'll report more after a few weeks once we have real customers using this in production.
>
> Because we deployed both protocol-buffers and HTTP/2 at the same time, I can't really tell which one had more effect, but the combination looks great.
>
> Ari
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
Sure, but let's not lose sight of what makes Cayenne ROP interesting and different. It exposes the whole stack of Cayenne contexts, nested contexts, rollback, prefetch, pagination, etc and not just the data objects themselves.
Otherwise it is just another restful, json, routing stack of which there are quite a few. Not that those simple rest stacks aren't useful, but what we have in Cayenne ROP is really quite different and interesting.
By coincidence I'm just today thinking about the lifecycle of data in the backbonejs library. That a javascript library for creating a data model and dealing with routing. It already has connectors for LocalStorage (storing data in the browser db). And of course for json/ajax. The question is, should we overlay a concept of Context to group the data in the model, track changes and move it from LocalStorage to ajax onto the server?
Ari
On 8/07/2016 10:16pm, Andrus Adamchik wrote:
> I suspect we'll need to step away from "blind" serialization of our server-side data structures and devise a JSON object protocol for any JS integrations to become practical. This may actually be simpler then it sounds (LinkRest does a good deal of it already: http://linkrest.io/docs/protocol.html ) and if there's a right client-side framework to work with such protocol, it would certainly be a big motivation.
>
> Andrus
>
>> On Jul 8, 2016, at 3:11 PM, Michael Gentry <bl...@gmail.com> wrote:
>>
>> Another item on my ever-growing to-do list is to explore if BreezeJS could
>> be used nicely with Cayenne. From my initial glance at the project, it
>> seems to have the spirit of Cayenne in a JS stack.
>>
>>
>> http://breeze.github.io/doc-js/
>>
>> http://www.getbreezenow.com/
>>
>> http://learn.breezejs.com/
>>
>>
>> mrg
>>
>>
>>
>> On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>>
>>> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
>>>> I'll report more after a few weeks once we have real customers using
>>> this in production.
>>>
>>> We now have this version of Cayenne (pretty close to what is now trunk) in
>>> production at around 20 sites with more to come over the next weeks.
>>>
>>> Not a single reported problem and significant performance improvements
>>> across the board. Any site with more than 40ms latency to the data centre
>>> (that is, between the Cayenne ROP client and server) is seeing noticeable
>>> improvements.
>>>
>>> Savva, very nice work indeed.
>>>
>>>
>>>
>>> Andrus, if you have json workflows protostuff also supports json encoding.
>>> [1] Of course that means a whole bunch of json encoded Cayenne objects
>>> like ObjectContext and I have no idea what you'd do with them in
>>> javascript, but if someone wanted to code a js version of Cayenne client, I
>>> guess they could.
>>>
>>> That would be an interesting idea to layer on top of backbonejs or similar.
>>>
>>>
>>> Ari
>>>
>>>
>>>
>>>
>>> [1]
>>> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>>>
>>> --
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>
>
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
I suspect we'll need to step away from "blind" serialization of our server-side data structures and devise a JSON object protocol for any JS integrations to become practical. This may actually be simpler then it sounds (LinkRest does a good deal of it already: http://linkrest.io/docs/protocol.html ) and if there's a right client-side framework to work with such protocol, it would certainly be a big motivation.
Andrus
> On Jul 8, 2016, at 3:11 PM, Michael Gentry <bl...@gmail.com> wrote:
>
> Another item on my ever-growing to-do list is to explore if BreezeJS could
> be used nicely with Cayenne. From my initial glance at the project, it
> seems to have the spirit of Cayenne in a JS stack.
>
>
> http://breeze.github.io/doc-js/
>
> http://www.getbreezenow.com/
>
> http://learn.breezejs.com/
>
>
> mrg
>
>
>
> On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
>> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
>>> I'll report more after a few weeks once we have real customers using
>> this in production.
>>
>> We now have this version of Cayenne (pretty close to what is now trunk) in
>> production at around 20 sites with more to come over the next weeks.
>>
>> Not a single reported problem and significant performance improvements
>> across the board. Any site with more than 40ms latency to the data centre
>> (that is, between the Cayenne ROP client and server) is seeing noticeable
>> improvements.
>>
>> Savva, very nice work indeed.
>>
>>
>>
>> Andrus, if you have json workflows protostuff also supports json encoding.
>> [1] Of course that means a whole bunch of json encoded Cayenne objects
>> like ObjectContext and I have no idea what you'd do with them in
>> javascript, but if someone wanted to code a js version of Cayenne client, I
>> guess they could.
>>
>> That would be an interesting idea to layer on top of backbonejs or similar.
>>
>>
>> Ari
>>
>>
>>
>>
>> [1]
>> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>>
>> --
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
Re: Cayenne ROP Improvements
Posted by Savva Kolbachev <s....@gmail.com>.
> Not a single reported problem and significant performance improvements
across the board. Any site with more than 40ms latency to the data > centre
(that is, between the Cayenne ROP client and server) is seeing noticeable
improvements.
>
> Savva, very nice work indeed.
Thanks! I'm very happy that it works well for you. I hope it will be
helpful for other Cayenne users as well.
2016-07-08 10:55 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
> On 8/07/2016 4:24pm, Andrus Adamchik wrote:
> > One possible usage is ROP debugging. It was noticed before that due to
> the blind serialization nature of ROP, occasionally unintended server-side
> objects are sent to the client and vice versa (like someone patches a query
> to "precompile" its state, adding a reference to EntityResolver or
> something). Wonder if switching things to JSON in development/testing would
> help in visual tracking of those extraneous serialized objects?
>
>
> That's a nice idea, especially if it is possible to dump the json stream
> to disk while still using the protostuff binary stream to the client.
>
> Ari
>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
--
Best Regards,
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 8/07/2016 4:24pm, Andrus Adamchik wrote:
> One possible usage is ROP debugging. It was noticed before that due to the blind serialization nature of ROP, occasionally unintended server-side objects are sent to the client and vice versa (like someone patches a query to "precompile" its state, adding a reference to EntityResolver or something). Wonder if switching things to JSON in development/testing would help in visual tracking of those extraneous serialized objects?
That's a nice idea, especially if it is possible to dump the json stream to disk while still using the protostuff binary stream to the client.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
> Not a single reported problem and significant performance improvements across the board. Any site with more than 40ms latency to the data centre (that is, between the Cayenne ROP client and server) is seeing noticeable improvements.
>
> Savva, very nice work indeed.
Nice!
> Andrus, if you have json workflows protostuff also supports json encoding. [1] Of course that means a whole bunch of json encoded Cayenne objects like ObjectContext
One possible usage is ROP debugging. It was noticed before that due to the blind serialization nature of ROP, occasionally unintended server-side objects are sent to the client and vice versa (like someone patches a query to "precompile" its state, adding a reference to EntityResolver or something). Wonder if switching things to JSON in development/testing would help in visual tracking of those extraneous serialized objects?
Andrus
> On Jul 8, 2016, at 4:01 AM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
>> I'll report more after a few weeks once we have real customers using this in production.
>
> We now have this version of Cayenne (pretty close to what is now trunk) in production at around 20 sites with more to come over the next weeks.
>
> Not a single reported problem and significant performance improvements across the board. Any site with more than 40ms latency to the data centre (that is, between the Cayenne ROP client and server) is seeing noticeable improvements.
>
> Savva, very nice work indeed.
>
>
>
> Andrus, if you have json workflows protostuff also supports json encoding. [1] Of course that means a whole bunch of json encoded Cayenne objects like ObjectContext and I have no idea what you'd do with them in javascript, but if someone wanted to code a js version of Cayenne client, I guess they could.
>
> That would be an interesting idea to layer on top of backbonejs or similar.
>
>
> Ari
>
>
>
>
> [1] https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Michael Gentry <bl...@gmail.com>.
Another item on my ever-growing to-do list is to explore if BreezeJS could
be used nicely with Cayenne. From my initial glance at the project, it
seems to have the spirit of Cayenne in a JS stack.
http://breeze.github.io/doc-js/
http://www.getbreezenow.com/
http://learn.breezejs.com/
mrg
On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
> > I'll report more after a few weeks once we have real customers using
> this in production.
>
> We now have this version of Cayenne (pretty close to what is now trunk) in
> production at around 20 sites with more to come over the next weeks.
>
> Not a single reported problem and significant performance improvements
> across the board. Any site with more than 40ms latency to the data centre
> (that is, between the Cayenne ROP client and server) is seeing noticeable
> improvements.
>
> Savva, very nice work indeed.
>
>
>
> Andrus, if you have json workflows protostuff also supports json encoding.
> [1] Of course that means a whole bunch of json encoded Cayenne objects
> like ObjectContext and I have no idea what you'd do with them in
> javascript, but if someone wanted to code a js version of Cayenne client, I
> guess they could.
>
> That would be an interesting idea to layer on top of backbonejs or similar.
>
>
> Ari
>
>
>
>
> [1]
> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
> I'll report more after a few weeks once we have real customers using this in production.
We now have this version of Cayenne (pretty close to what is now trunk) in production at around 20 sites with more to come over the next weeks.
Not a single reported problem and significant performance improvements across the board. Any site with more than 40ms latency to the data centre (that is, between the Cayenne ROP client and server) is seeing noticeable improvements.
Savva, very nice work indeed.
Andrus, if you have json workflows protostuff also supports json encoding. [1] Of course that means a whole bunch of json encoded Cayenne objects like ObjectContext and I have no idea what you'd do with them in javascript, but if someone wanted to code a js version of Cayenne client, I guess they could.
That would be an interesting idea to layer on top of backbonejs or similar.
Ari
[1] https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 30/06/2016 7:21pm, Savva Kolbachev wrote:
> BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
> how it's going. If it's OK, I'll notify user-list about this features.
We've actually just deployed the upgrade into our staging environment are will likely roll out the first customers early next week. If our preliminary (very non-scientific) results are anything to go by, almost everything is faster across a slow link.
Some parts of the UI (opening a certain window) went from 14 seconds to 5 seconds. Another from 87 seconds to 10 seconds. And one from 8 to 5. The difference is likely how effectively we used prefetches already. In some places where we follow a lot of relation joins, we could easily generate 50 separate fetches across the network. And on a slow link (say over 40ms latency and bandwidth under 2Mb/s) with SSL handshaking that adds up very quickly.
I'll report more after a few weeks once we have real customers using this in production.
Because we deployed both protocol-buffers and HTTP/2 at the same time, I can't really tell which one had more effect, but the combination looks great.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Savva Kolbachev <s....@gmail.com>.
I've finished work on Jetty Client implementation for Cayenne ROP by
https://github.com/apache/cayenne/commit/34646fd5baf3640208399b2f21afc7afff6def51
So now we have cayenne-client-jetty module which works as follows:
There is a one JettyHttpROPConnector
<https://github.com/apache/cayenne/blob/master/cayenne-client-jetty/src/main/java/org/apache/cayenne/rop/http/JettyHttpROPConnector.java>
which
takes and uses Jetty HttpClient (high-level API). It could be configured by
JettyHttpClientConnectionProvider
<https://github.com/apache/cayenne/blob/master/cayenne-client-jetty/src/main/java/org/apache/cayenne/rop/JettyHttpClientConnectionProvider.java>
for
HTTP/1.1 and by JettyHttp2ClientConnectionProvider
<https://github.com/apache/cayenne/blob/master/cayenne-client-jetty/src/main/java/org/apache/cayenne/rop/JettyHttp2ClientConnectionProvider.java>
for
HTTP/2. The last one initializes Jetty HttpClient over HTTP2Client
through HttpClientTransportOverHTTP2 and works without ALPN by default.
ALPN could be configured
via Constants.ROP_SERVICE_USE_ALPN_PROPERTY. However, I'm not sure it's the
best way to do it.
Those ConnectionProviders are bound by ClientJettyHttpModule
<https://github.com/apache/cayenne/blob/master/cayenne-client-jetty/src/main/java/org/apache/cayenne/configuration/rop/client/ClientJettyHttpModule.java>
and ClientJettyHttp2Module
<https://github.com/apache/cayenne/blob/master/cayenne-client-jetty/src/main/java/org/apache/cayenne/configuration/rop/client/ClientJettyHttp2Module.java>
accordingly.
So, users just need to add the appropriate Module to their ClientRuntime.
I think all those things could be used by themselves or as examples to
providing custom implementations. Also, I've added the simple ROP tutorial
with both ClientJettyHttp2Module and ProtostuffModule:
Http2Client
<https://github.com/apache/cayenne/blob/master/tutorials/tutorial-rop-client-http2/src/main/java/org/apache/cayenne/tutorial/Http2Client.java>
Http2Server
<https://github.com/apache/cayenne/blob/master/tutorials/tutorial-rop-server-http2/src/main/java/org/apache/cayenne/tutorial/Http2Server.java>
BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
how it's going. If it's OK, I'll notify user-list about this features.
2016-06-21 3:54 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
> On 13/06/2016 6:40pm, Savva Kolbachev wrote:
> > After some time I think that my current HTTP/2 ROP implementation for
> > Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> > it as clear as possible, so it will be simple to use out of box.
> > So, I suggest to rename module to something like "cayenne-jetty-client"
> and
> > implement only two ROPConnectors: JettyHttpConnector and
> > JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> > high-level Jetty Client API.
>
> When I look at https://github.com/apache/cayenne/pull/96/files I think
> I'm seeing two separate things.
>
> 1. Adjustments and improvements to the Cayenne ROP implementation. For
> example changes to the exception handling in cayenne/rop/ROPConnector.java
>
> 2. An implementation of HTTP2/jetty.
>
>
> It would be interesting to split the two. First to commit the ROP
> improvements, but then to see what is left over. If it is only a couple of
> classes HighHttp2... LowHttp2... etc then another option is not to
> implement this at all as a compiled jar which users add to their own
> project. Rather, they could be some code examples that users copy and paste.
>
>
> I'm not convinced this is the best approach, but I wanted to raise it as
> an option. I guess the question is whether users would find the
> org.apache.cayenne.configuration.Constants approach to configuration
> easier, or would prefer just editing the relevant files to suit their own
> config system.
>
>
>
> At any rate, now that the Jetty project have fixed the bug you found
> there, it looks like HTTP2-Cayenne-ROP is ready for production! Nice work.
>
>
> Ari
>
>
>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
--
Best Regards,
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 13/06/2016 6:40pm, Savva Kolbachev wrote:
> After some time I think that my current HTTP/2 ROP implementation for
> Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> it as clear as possible, so it will be simple to use out of box.
> So, I suggest to rename module to something like "cayenne-jetty-client" and
> implement only two ROPConnectors: JettyHttpConnector and
> JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> high-level Jetty Client API.
When I look at https://github.com/apache/cayenne/pull/96/files I think I'm seeing two separate things.
1. Adjustments and improvements to the Cayenne ROP implementation. For example changes to the exception handling in cayenne/rop/ROPConnector.java
2. An implementation of HTTP2/jetty.
It would be interesting to split the two. First to commit the ROP improvements, but then to see what is left over. If it is only a couple of classes HighHttp2... LowHttp2... etc then another option is not to implement this at all as a compiled jar which users add to their own project. Rather, they could be some code examples that users copy and paste.
I'm not convinced this is the best approach, but I wanted to raise it as an option. I guess the question is whether users would find the org.apache.cayenne.configuration.Constants approach to configuration easier, or would prefer just editing the relevant files to suit their own config system.
At any rate, now that the Jetty project have fixed the bug you found there, it looks like HTTP2-Cayenne-ROP is ready for production! Nice work.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Savva Kolbachev <s....@gmail.com>.
After some time I think that my current HTTP/2 ROP implementation for
Cayenne [1] is cumbersome and complicated. I want to refactor it and make
it as clear as possible, so it will be simple to use out of box.
So, I suggest to rename module to something like "cayenne-jetty-client" and
implement only two ROPConnectors: JettyHttpConnector and
JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
high-level Jetty Client API.
If someone wants something specific, they could provide their own similar
implementation. I think the most common case for this is to share one
HttpClient between ROP and other parts of a project.
In addition, we will be open to make more modules for other clients, like
Apache HTTP Client, Netty, OkHttp and etc.
BTW, I got rid of Hessian ServiceContext in ROPServlet [2] by implementing
something very similar in Cayenne
https://github.com/apache/cayenne/commit/85852f92ee05f5e1ae70dae7315beda32f2b4077
So now Hessian is used only for serialization purposes. As we want to move
all ROP functionality to the separate module and Hessian serialization has
problems with Java8, we should decide which serialization service should be
used by default. For example Protostuff or Java serialization. Hessian one
we could remain as pluggable.
Finally, as Cayenne is becoming more modular, it will be helpful to have
server and client test jars. It is sometimes frustrating to write
integration tests for new modules.
Please, let me know what do you think about all this things?
[1] https://github.com/apache/cayenne/pull/96
[2] https://issues.apache.org/jira/browse/CAY-2090
2016-05-06 3:29 GMT+03:00 Andrus Adamchik <an...@objectstyle.org>:
> Good idea. The server part should probably be split from cayenne-server.
>
> Andrus
>
> > On May 5, 2016, at 8:26 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
> >
> > Perhaps the whole of ROP becomes an optional module?
> >
> > Ari
> >
> >
> > On 6/05/2016 10:18am, Andrus Adamchik wrote:
> >> It's been a while since I touched the ROP code. Back in the day Java
> serialization "kind of worked", but not completely. So you are probably
> right that it is not a real option. I am just trying to avoid new
> dependencies (even optional) on third-party libs in the Cayenne core. So
> perhaps we can simply leave out any "default" serialization and always
> require an explicit serialization provider.
> >>
> >> Andrus
> >>
> >>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis <ar...@maniatis.org>
> wrote:
> >>>
> >>> Maybe I'm not understanding correctly, but I don't think Java
> serialisation has been implemented in ROP. The work Dima did was to move
> away from the Hessian servlet stuff for making the HTTP connection, to
> plain Java with the option for plugging in Jetty libraries for HTTP/2.
> >>>
> >>> The work Savva did just now was to use protostuff for serialisation,
> but I'm not sure what's now needed if we wanted plain Java serialisation or
> whether that's even possible without some sort of library to handle an
> object graph with cycles.
> >>>
> >>> Or at least that's my understanding.
> >>>
> >>> Ari
> >>>
> >>>
> >>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
> >>>> Thanks for clarification. I would say use Java serialization as a
> default, and make it easy to plugin Hessian and Protostuff as separate
> modules.
> >>>>
> >>>> A.
> >>>>
> >>>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com>
> wrote:
> >>>>>
> >>>>> Hi Andrus,
> >>>>>
> >>>>>> So which one is the default, Hessian or Java?
> >>>>> We still use Hessian for serialization by default
> >>>>>
> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
> >>>>> But we use java.net.URLConnection for establish connection and
> sending
> >>>>> messages from client to server
> >>>>>
> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
> >>>>> So we have escaped from Hessian only in connectivity layer.
> >>>>>
> >>>>>> I don't have a problem with Protostuff being a recommended default,
> but
> >>>>> for dependency management purposes I'd rather we split all
> third-party
> >>>>> integrations in separate modules, and use whatever provider is
> hooked up in
> >>>>> runtime. Kind of what we do with Joda/Java8 extensions.
> >>>>> I already did it in this way. I created separate module for
> Protostuff
> >>>>> serialization.
> >>>>>
> >>>>> As Hessian serialization has some troubles with Java8 types and
> provide
> >>>>> less efficient serialization than Protostuff, I suggest to use
> Protostuff
> >>>>> as default serialization service or to use Java serialization. So I
> just
> >>>>> suggest to escape from Hessian :)
> >>>>>
> >>>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
> >>>>>
> >>>>>> Hi Ari,
> >>>>>>
> >>>>>> Looks like Protostuff works faster than Protobuf in some cases. For
> >>>>>> example Serializers (no shared refs) and Cross Lang Binary
> Serializers
> >>>>>> sections here
> http://hperadin.github.io/jvm-serializers-report/report.html
> >>>>>>
> >>>>>> In our case we need to serialize graph of objects (Full Object Graph
> >>>>>> Serializers section in link above). Protobuf can't do it out of the
> box
> >>>>>> but Protostuff can. In my implementation I use
> protostuff-graph-runtime
> >>>>>> which generates a schema from objects at runtime and caches it.
> >>>>>>
> >>>>>> Protostuff schema is something like .proto files but in Java:
> >>>>>> http://www.protostuff.io/documentation/schema/
> >>>>>> Runtime schema:
> http://www.protostuff.io/documentation/runtime-schema/
> >>>>>>
> >>>>>> As you could see in benchmarks there is a small difference in
> efficiency
> >>>>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
> >>>>>> overhead is related to runtime schema generation. The size penalty
> is that
> >>>>>> Protostuff adds class name for objects and than uses those for find
> >>>>>> appropriate classes via reflection.
> >>>>>> Hessian also adds fields names so the size of Hessian serialization
> is
> >>>>>> much bigger. In my small example with selection of 6 objects Hessian
> >>>>>> serialization size is more than 2400 bytes while Protostuff runtime
> is
> >>>>>> about 800 bytes.
> >>>>>>
> >>>>>> If we don't want to have ser/deser and size overhead we could find
> a way
> >>>>>> to generate schemas via Velocity. And we should provide schemas for
> some
> >>>>>> Cayenne classes. But it will require a lot of efforts.
> >>>>>>
> >>>>>>
> >>>>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
> >>>>>>
> >>>>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> >>>>>>>> Protostuff (licensed under Apache 2.0 licence) is based on
> Google's
> >>>>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some
> cool
> >>>>>>> things
> >>>>>>>> like runtime serialization graph of objects (like Hessian). It
> also
> >>>>>>> could
> >>>>>>>> generate schema on runtime so we shouldn't define .proto files
> although
> >>>>>>> it
> >>>>>>>> might increase efficiency. It works faster than Hessian and could
> handle
> >>>>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look
> at Full
> >>>>>>>> Object Graph Serializers section.
> >>>>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
> >>>>>>>> https://github.com/eishay/jvm-serializers/wiki
> >>>>>>>
> >>>>>>> According to those benchmarks there appears to be no performance
> or size
> >>>>>>> penalty to using protostuff over protobuffers. Am I reading that
> right?
> >>>>>>>
> >>>>>>> I don't really understand... doesn't the serialiser have to
> construct a
> >>>>>>> .proto definition and then include it in the message? So shouldn't
> it be
> >>>>>>> faster/smaller to predefine these?
> >>>>>>>
> >>>>>>> If we did, we could create them with velocity in the same way we
> create
> >>>>>>> Java _superclasses today. Fairly trivial I'm guessing.
> >>>>>>>
> >>>>>>> Ari
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> -------------------------->
> >>>>>>> Aristedes Maniatis
> >>>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Thanks and Regards
> >>>>>> Savva Kolbachev
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Thanks and Regards
> >>>>> Savva Kolbachev
> >>>>
> >>>
> >>> --
> >>> -------------------------->
> >>> Aristedes Maniatis
> >>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
> >>
> >
> > --
> > -------------------------->
> > Aristedes Maniatis
> > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
>
--
Thanks and Regards
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
Good idea. The server part should probably be split from cayenne-server.
Andrus
> On May 5, 2016, at 8:26 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
> Perhaps the whole of ROP becomes an optional module?
>
> Ari
>
>
> On 6/05/2016 10:18am, Andrus Adamchik wrote:
>> It's been a while since I touched the ROP code. Back in the day Java serialization "kind of worked", but not completely. So you are probably right that it is not a real option. I am just trying to avoid new dependencies (even optional) on third-party libs in the Cayenne core. So perhaps we can simply leave out any "default" serialization and always require an explicit serialization provider.
>>
>> Andrus
>>
>>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>>>
>>> Maybe I'm not understanding correctly, but I don't think Java serialisation has been implemented in ROP. The work Dima did was to move away from the Hessian servlet stuff for making the HTTP connection, to plain Java with the option for plugging in Jetty libraries for HTTP/2.
>>>
>>> The work Savva did just now was to use protostuff for serialisation, but I'm not sure what's now needed if we wanted plain Java serialisation or whether that's even possible without some sort of library to handle an object graph with cycles.
>>>
>>> Or at least that's my understanding.
>>>
>>> Ari
>>>
>>>
>>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>>>> Thanks for clarification. I would say use Java serialization as a default, and make it easy to plugin Hessian and Protostuff as separate modules.
>>>>
>>>> A.
>>>>
>>>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com> wrote:
>>>>>
>>>>> Hi Andrus,
>>>>>
>>>>>> So which one is the default, Hessian or Java?
>>>>> We still use Hessian for serialization by default
>>>>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>>>> But we use java.net.URLConnection for establish connection and sending
>>>>> messages from client to server
>>>>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>>>> So we have escaped from Hessian only in connectivity layer.
>>>>>
>>>>>> I don't have a problem with Protostuff being a recommended default, but
>>>>> for dependency management purposes I'd rather we split all third-party
>>>>> integrations in separate modules, and use whatever provider is hooked up in
>>>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>>>> I already did it in this way. I created separate module for Protostuff
>>>>> serialization.
>>>>>
>>>>> As Hessian serialization has some troubles with Java8 types and provide
>>>>> less efficient serialization than Protostuff, I suggest to use Protostuff
>>>>> as default serialization service or to use Java serialization. So I just
>>>>> suggest to escape from Hessian :)
>>>>>
>>>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>>>>>
>>>>>> Hi Ari,
>>>>>>
>>>>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>>>>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>>>>>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>>
>>>>>> In our case we need to serialize graph of objects (Full Object Graph
>>>>>> Serializers section in link above). Protobuf can't do it out of the box
>>>>>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>>>>>> which generates a schema from objects at runtime and caches it.
>>>>>>
>>>>>> Protostuff schema is something like .proto files but in Java:
>>>>>> http://www.protostuff.io/documentation/schema/
>>>>>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>>>>>
>>>>>> As you could see in benchmarks there is a small difference in efficiency
>>>>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>>>>> overhead is related to runtime schema generation. The size penalty is that
>>>>>> Protostuff adds class name for objects and than uses those for find
>>>>>> appropriate classes via reflection.
>>>>>> Hessian also adds fields names so the size of Hessian serialization is
>>>>>> much bigger. In my small example with selection of 6 objects Hessian
>>>>>> serialization size is more than 2400 bytes while Protostuff runtime is
>>>>>> about 800 bytes.
>>>>>>
>>>>>> If we don't want to have ser/deser and size overhead we could find a way
>>>>>> to generate schemas via Velocity. And we should provide schemas for some
>>>>>> Cayenne classes. But it will require a lot of efforts.
>>>>>>
>>>>>>
>>>>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>>>>>
>>>>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>>>>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>>>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>>>>>> things
>>>>>>>> like runtime serialization graph of objects (like Hessian). It also
>>>>>>> could
>>>>>>>> generate schema on runtime so we shouldn't define .proto files although
>>>>>>> it
>>>>>>>> might increase efficiency. It works faster than Hessian and could handle
>>>>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>>>>>>>> Object Graph Serializers section.
>>>>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>>>> https://github.com/eishay/jvm-serializers/wiki
>>>>>>>
>>>>>>> According to those benchmarks there appears to be no performance or size
>>>>>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>>>>>>
>>>>>>> I don't really understand... doesn't the serialiser have to construct a
>>>>>>> .proto definition and then include it in the message? So shouldn't it be
>>>>>>> faster/smaller to predefine these?
>>>>>>>
>>>>>>> If we did, we could create them with velocity in the same way we create
>>>>>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>>>>>
>>>>>>> Ari
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -------------------------->
>>>>>>> Aristedes Maniatis
>>>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards
>>>>>> Savva Kolbachev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards
>>>>> Savva Kolbachev
>>>>
>>>
>>> --
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
Perhaps the whole of ROP becomes an optional module?
Ari
On 6/05/2016 10:18am, Andrus Adamchik wrote:
> It's been a while since I touched the ROP code. Back in the day Java serialization "kind of worked", but not completely. So you are probably right that it is not a real option. I am just trying to avoid new dependencies (even optional) on third-party libs in the Cayenne core. So perhaps we can simply leave out any "default" serialization and always require an explicit serialization provider.
>
> Andrus
>
>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>>
>> Maybe I'm not understanding correctly, but I don't think Java serialisation has been implemented in ROP. The work Dima did was to move away from the Hessian servlet stuff for making the HTTP connection, to plain Java with the option for plugging in Jetty libraries for HTTP/2.
>>
>> The work Savva did just now was to use protostuff for serialisation, but I'm not sure what's now needed if we wanted plain Java serialisation or whether that's even possible without some sort of library to handle an object graph with cycles.
>>
>> Or at least that's my understanding.
>>
>> Ari
>>
>>
>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>>> Thanks for clarification. I would say use Java serialization as a default, and make it easy to plugin Hessian and Protostuff as separate modules.
>>>
>>> A.
>>>
>>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com> wrote:
>>>>
>>>> Hi Andrus,
>>>>
>>>>> So which one is the default, Hessian or Java?
>>>> We still use Hessian for serialization by default
>>>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>>> But we use java.net.URLConnection for establish connection and sending
>>>> messages from client to server
>>>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>>> So we have escaped from Hessian only in connectivity layer.
>>>>
>>>>> I don't have a problem with Protostuff being a recommended default, but
>>>> for dependency management purposes I'd rather we split all third-party
>>>> integrations in separate modules, and use whatever provider is hooked up in
>>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>>> I already did it in this way. I created separate module for Protostuff
>>>> serialization.
>>>>
>>>> As Hessian serialization has some troubles with Java8 types and provide
>>>> less efficient serialization than Protostuff, I suggest to use Protostuff
>>>> as default serialization service or to use Java serialization. So I just
>>>> suggest to escape from Hessian :)
>>>>
>>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>>>>
>>>>> Hi Ari,
>>>>>
>>>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>>>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>>>>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>
>>>>> In our case we need to serialize graph of objects (Full Object Graph
>>>>> Serializers section in link above). Protobuf can't do it out of the box
>>>>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>>>>> which generates a schema from objects at runtime and caches it.
>>>>>
>>>>> Protostuff schema is something like .proto files but in Java:
>>>>> http://www.protostuff.io/documentation/schema/
>>>>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>>>>
>>>>> As you could see in benchmarks there is a small difference in efficiency
>>>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>>>> overhead is related to runtime schema generation. The size penalty is that
>>>>> Protostuff adds class name for objects and than uses those for find
>>>>> appropriate classes via reflection.
>>>>> Hessian also adds fields names so the size of Hessian serialization is
>>>>> much bigger. In my small example with selection of 6 objects Hessian
>>>>> serialization size is more than 2400 bytes while Protostuff runtime is
>>>>> about 800 bytes.
>>>>>
>>>>> If we don't want to have ser/deser and size overhead we could find a way
>>>>> to generate schemas via Velocity. And we should provide schemas for some
>>>>> Cayenne classes. But it will require a lot of efforts.
>>>>>
>>>>>
>>>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>>>>
>>>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>>>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>>>>> things
>>>>>>> like runtime serialization graph of objects (like Hessian). It also
>>>>>> could
>>>>>>> generate schema on runtime so we shouldn't define .proto files although
>>>>>> it
>>>>>>> might increase efficiency. It works faster than Hessian and could handle
>>>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>>>>>>> Object Graph Serializers section.
>>>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>>> https://github.com/eishay/jvm-serializers/wiki
>>>>>>
>>>>>> According to those benchmarks there appears to be no performance or size
>>>>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>>>>>
>>>>>> I don't really understand... doesn't the serialiser have to construct a
>>>>>> .proto definition and then include it in the message? So shouldn't it be
>>>>>> faster/smaller to predefine these?
>>>>>>
>>>>>> If we did, we could create them with velocity in the same way we create
>>>>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>>>>
>>>>>> Ari
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -------------------------->
>>>>>> Aristedes Maniatis
>>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards
>>>>> Savva Kolbachev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards
>>>> Savva Kolbachev
>>>
>>
>> --
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
It's been a while since I touched the ROP code. Back in the day Java serialization "kind of worked", but not completely. So you are probably right that it is not a real option. I am just trying to avoid new dependencies (even optional) on third-party libs in the Cayenne core. So perhaps we can simply leave out any "default" serialization and always require an explicit serialization provider.
Andrus
> On May 5, 2016, at 8:12 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
> Maybe I'm not understanding correctly, but I don't think Java serialisation has been implemented in ROP. The work Dima did was to move away from the Hessian servlet stuff for making the HTTP connection, to plain Java with the option for plugging in Jetty libraries for HTTP/2.
>
> The work Savva did just now was to use protostuff for serialisation, but I'm not sure what's now needed if we wanted plain Java serialisation or whether that's even possible without some sort of library to handle an object graph with cycles.
>
> Or at least that's my understanding.
>
> Ari
>
>
> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>> Thanks for clarification. I would say use Java serialization as a default, and make it easy to plugin Hessian and Protostuff as separate modules.
>>
>> A.
>>
>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com> wrote:
>>>
>>> Hi Andrus,
>>>
>>>> So which one is the default, Hessian or Java?
>>> We still use Hessian for serialization by default
>>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>> But we use java.net.URLConnection for establish connection and sending
>>> messages from client to server
>>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>> So we have escaped from Hessian only in connectivity layer.
>>>
>>>> I don't have a problem with Protostuff being a recommended default, but
>>> for dependency management purposes I'd rather we split all third-party
>>> integrations in separate modules, and use whatever provider is hooked up in
>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>> I already did it in this way. I created separate module for Protostuff
>>> serialization.
>>>
>>> As Hessian serialization has some troubles with Java8 types and provide
>>> less efficient serialization than Protostuff, I suggest to use Protostuff
>>> as default serialization service or to use Java serialization. So I just
>>> suggest to escape from Hessian :)
>>>
>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>>>
>>>> Hi Ari,
>>>>
>>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>>>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>>>
>>>> In our case we need to serialize graph of objects (Full Object Graph
>>>> Serializers section in link above). Protobuf can't do it out of the box
>>>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>>>> which generates a schema from objects at runtime and caches it.
>>>>
>>>> Protostuff schema is something like .proto files but in Java:
>>>> http://www.protostuff.io/documentation/schema/
>>>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>>>
>>>> As you could see in benchmarks there is a small difference in efficiency
>>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>>> overhead is related to runtime schema generation. The size penalty is that
>>>> Protostuff adds class name for objects and than uses those for find
>>>> appropriate classes via reflection.
>>>> Hessian also adds fields names so the size of Hessian serialization is
>>>> much bigger. In my small example with selection of 6 objects Hessian
>>>> serialization size is more than 2400 bytes while Protostuff runtime is
>>>> about 800 bytes.
>>>>
>>>> If we don't want to have ser/deser and size overhead we could find a way
>>>> to generate schemas via Velocity. And we should provide schemas for some
>>>> Cayenne classes. But it will require a lot of efforts.
>>>>
>>>>
>>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>>>
>>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>>>> things
>>>>>> like runtime serialization graph of objects (like Hessian). It also
>>>>> could
>>>>>> generate schema on runtime so we shouldn't define .proto files although
>>>>> it
>>>>>> might increase efficiency. It works faster than Hessian and could handle
>>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>>>>>> Object Graph Serializers section.
>>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>>> https://github.com/eishay/jvm-serializers/wiki
>>>>>
>>>>> According to those benchmarks there appears to be no performance or size
>>>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>>>>
>>>>> I don't really understand... doesn't the serialiser have to construct a
>>>>> .proto definition and then include it in the message? So shouldn't it be
>>>>> faster/smaller to predefine these?
>>>>>
>>>>> If we did, we could create them with velocity in the same way we create
>>>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>>>
>>>>> Ari
>>>>>
>>>>>
>>>>> --
>>>>> -------------------------->
>>>>> Aristedes Maniatis
>>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards
>>>> Savva Kolbachev
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Savva Kolbachev
>>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
Maybe I'm not understanding correctly, but I don't think Java serialisation has been implemented in ROP. The work Dima did was to move away from the Hessian servlet stuff for making the HTTP connection, to plain Java with the option for plugging in Jetty libraries for HTTP/2.
The work Savva did just now was to use protostuff for serialisation, but I'm not sure what's now needed if we wanted plain Java serialisation or whether that's even possible without some sort of library to handle an object graph with cycles.
Or at least that's my understanding.
Ari
On 6/05/2016 9:55am, Andrus Adamchik wrote:
> Thanks for clarification. I would say use Java serialization as a default, and make it easy to plugin Hessian and Protostuff as separate modules.
>
> A.
>
>> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com> wrote:
>>
>> Hi Andrus,
>>
>>> So which one is the default, Hessian or Java?
>> We still use Hessian for serialization by default
>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>> But we use java.net.URLConnection for establish connection and sending
>> messages from client to server
>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>> So we have escaped from Hessian only in connectivity layer.
>>
>>> I don't have a problem with Protostuff being a recommended default, but
>> for dependency management purposes I'd rather we split all third-party
>> integrations in separate modules, and use whatever provider is hooked up in
>> runtime. Kind of what we do with Joda/Java8 extensions.
>> I already did it in this way. I created separate module for Protostuff
>> serialization.
>>
>> As Hessian serialization has some troubles with Java8 types and provide
>> less efficient serialization than Protostuff, I suggest to use Protostuff
>> as default serialization service or to use Java serialization. So I just
>> suggest to escape from Hessian :)
>>
>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>>
>>> Hi Ari,
>>>
>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>>
>>> In our case we need to serialize graph of objects (Full Object Graph
>>> Serializers section in link above). Protobuf can't do it out of the box
>>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>>> which generates a schema from objects at runtime and caches it.
>>>
>>> Protostuff schema is something like .proto files but in Java:
>>> http://www.protostuff.io/documentation/schema/
>>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>>
>>> As you could see in benchmarks there is a small difference in efficiency
>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>> overhead is related to runtime schema generation. The size penalty is that
>>> Protostuff adds class name for objects and than uses those for find
>>> appropriate classes via reflection.
>>> Hessian also adds fields names so the size of Hessian serialization is
>>> much bigger. In my small example with selection of 6 objects Hessian
>>> serialization size is more than 2400 bytes while Protostuff runtime is
>>> about 800 bytes.
>>>
>>> If we don't want to have ser/deser and size overhead we could find a way
>>> to generate schemas via Velocity. And we should provide schemas for some
>>> Cayenne classes. But it will require a lot of efforts.
>>>
>>>
>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>>
>>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>>> things
>>>>> like runtime serialization graph of objects (like Hessian). It also
>>>> could
>>>>> generate schema on runtime so we shouldn't define .proto files although
>>>> it
>>>>> might increase efficiency. It works faster than Hessian and could handle
>>>>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>>>>> Object Graph Serializers section.
>>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>>> https://github.com/eishay/jvm-serializers/wiki
>>>>
>>>> According to those benchmarks there appears to be no performance or size
>>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>>>
>>>> I don't really understand... doesn't the serialiser have to construct a
>>>> .proto definition and then include it in the message? So shouldn't it be
>>>> faster/smaller to predefine these?
>>>>
>>>> If we did, we could create them with velocity in the same way we create
>>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>>
>>>> Ari
>>>>
>>>>
>>>> --
>>>> -------------------------->
>>>> Aristedes Maniatis
>>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Savva Kolbachev
>>>
>>
>>
>>
>> --
>> Thanks and Regards
>> Savva Kolbachev
>
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Andrus Adamchik <an...@objectstyle.org>.
Thanks for clarification. I would say use Java serialization as a default, and make it easy to plugin Hessian and Protostuff as separate modules.
A.
> On May 5, 2016, at 5:39 PM, Savva Kolbachev <s....@gmail.com> wrote:
>
> Hi Andrus,
>
>> So which one is the default, Hessian or Java?
> We still use Hessian for serialization by default
> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
> But we use java.net.URLConnection for establish connection and sending
> messages from client to server
> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
> So we have escaped from Hessian only in connectivity layer.
>
>> I don't have a problem with Protostuff being a recommended default, but
> for dependency management purposes I'd rather we split all third-party
> integrations in separate modules, and use whatever provider is hooked up in
> runtime. Kind of what we do with Joda/Java8 extensions.
> I already did it in this way. I created separate module for Protostuff
> serialization.
>
> As Hessian serialization has some troubles with Java8 types and provide
> less efficient serialization than Protostuff, I suggest to use Protostuff
> as default serialization service or to use Java serialization. So I just
> suggest to escape from Hessian :)
>
> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
>
>> Hi Ari,
>>
>> Looks like Protostuff works faster than Protobuf in some cases. For
>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>
>> In our case we need to serialize graph of objects (Full Object Graph
>> Serializers section in link above). Protobuf can't do it out of the box
>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>> which generates a schema from objects at runtime and caches it.
>>
>> Protostuff schema is something like .proto files but in Java:
>> http://www.protostuff.io/documentation/schema/
>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>
>> As you could see in benchmarks there is a small difference in efficiency
>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>> overhead is related to runtime schema generation. The size penalty is that
>> Protostuff adds class name for objects and than uses those for find
>> appropriate classes via reflection.
>> Hessian also adds fields names so the size of Hessian serialization is
>> much bigger. In my small example with selection of 6 objects Hessian
>> serialization size is more than 2400 bytes while Protostuff runtime is
>> about 800 bytes.
>>
>> If we don't want to have ser/deser and size overhead we could find a way
>> to generate schemas via Velocity. And we should provide schemas for some
>> Cayenne classes. But it will require a lot of efforts.
>>
>>
>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>>
>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>> things
>>>> like runtime serialization graph of objects (like Hessian). It also
>>> could
>>>> generate schema on runtime so we shouldn't define .proto files although
>>> it
>>>> might increase efficiency. It works faster than Hessian and could handle
>>>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>>>> Object Graph Serializers section.
>>>> http://hperadin.github.io/jvm-serializers-report/report.html
>>>> https://github.com/eishay/jvm-serializers/wiki
>>>
>>> According to those benchmarks there appears to be no performance or size
>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>>
>>> I don't really understand... doesn't the serialiser have to construct a
>>> .proto definition and then include it in the message? So shouldn't it be
>>> faster/smaller to predefine these?
>>>
>>> If we did, we could create them with velocity in the same way we create
>>> Java _superclasses today. Fairly trivial I'm guessing.
>>>
>>> Ari
>>>
>>>
>>> --
>>> -------------------------->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>>
>>
>>
>>
>> --
>> Thanks and Regards
>> Savva Kolbachev
>>
>
>
>
> --
> Thanks and Regards
> Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Savva Kolbachev <s....@gmail.com>.
Hi Andrus,
> So which one is the default, Hessian or Java?
We still use Hessian for serialization by default
https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
But we use java.net.URLConnection for establish connection and sending
messages from client to server
https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
So we have escaped from Hessian only in connectivity layer.
> I don't have a problem with Protostuff being a recommended default, but
for dependency management purposes I'd rather we split all third-party
integrations in separate modules, and use whatever provider is hooked up in
runtime. Kind of what we do with Joda/Java8 extensions.
I already did it in this way. I created separate module for Protostuff
serialization.
As Hessian serialization has some troubles with Java8 types and provide
less efficient serialization than Protostuff, I suggest to use Protostuff
as default serialization service or to use Java serialization. So I just
suggest to escape from Hessian :)
2016-05-05 19:41 GMT+03:00 Savva Kolbachev <s....@gmail.com>:
> Hi Ari,
>
> Looks like Protostuff works faster than Protobuf in some cases. For
> example Serializers (no shared refs) and Cross Lang Binary Serializers
> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>
> In our case we need to serialize graph of objects (Full Object Graph
> Serializers section in link above). Protobuf can't do it out of the box
> but Protostuff can. In my implementation I use protostuff-graph-runtime
> which generates a schema from objects at runtime and caches it.
>
> Protostuff schema is something like .proto files but in Java:
> http://www.protostuff.io/documentation/schema/
> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>
> As you could see in benchmarks there is a small difference in efficiency
> between protostuff-graph and protostuff-graph-runtime. The ser/deser
> overhead is related to runtime schema generation. The size penalty is that
> Protostuff adds class name for objects and than uses those for find
> appropriate classes via reflection.
> Hessian also adds fields names so the size of Hessian serialization is
> much bigger. In my small example with selection of 6 objects Hessian
> serialization size is more than 2400 bytes while Protostuff runtime is
> about 800 bytes.
>
> If we don't want to have ser/deser and size overhead we could find a way
> to generate schemas via Velocity. And we should provide schemas for some
> Cayenne classes. But it will require a lot of efforts.
>
>
> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
>
>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>> > Protostuff (licensed under Apache 2.0 licence) is based on Google's
>> > Protocol-Buffers (Protobuf) but has some optimizations and some cool
>> things
>> > like runtime serialization graph of objects (like Hessian). It also
>> could
>> > generate schema on runtime so we shouldn't define .proto files although
>> it
>> > might increase efficiency. It works faster than Hessian and could handle
>> > Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>> > Object Graph Serializers section.
>> > http://hperadin.github.io/jvm-serializers-report/report.html
>> > https://github.com/eishay/jvm-serializers/wiki
>>
>> According to those benchmarks there appears to be no performance or size
>> penalty to using protostuff over protobuffers. Am I reading that right?
>>
>> I don't really understand... doesn't the serialiser have to construct a
>> .proto definition and then include it in the message? So shouldn't it be
>> faster/smaller to predefine these?
>>
>> If we did, we could create them with velocity in the same way we create
>> Java _superclasses today. Fairly trivial I'm guessing.
>>
>> Ari
>>
>>
>> --
>> -------------------------->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
>
>
>
> --
> Thanks and Regards
> Savva Kolbachev
>
--
Thanks and Regards
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Savva Kolbachev <s....@gmail.com>.
Hi Ari,
Looks like Protostuff works faster than Protobuf in some cases. For
example Serializers
(no shared refs) and Cross Lang Binary Serializers sections here
http://hperadin.github.io/jvm-serializers-report/report.html
In our case we need to serialize graph of objects (Full Object Graph
Serializers section in link above). Protobuf can't do it out of the box but
Protostuff can. In my implementation I use protostuff-graph-runtime which
generates a schema from objects at runtime and caches it.
Protostuff schema is something like .proto files but in Java:
http://www.protostuff.io/documentation/schema/
Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
As you could see in benchmarks there is a small difference in efficiency
between protostuff-graph and protostuff-graph-runtime. The ser/deser
overhead is related to runtime schema generation. The size penalty is that
Protostuff adds class name for objects and than uses those for find
appropriate classes via reflection.
Hessian also adds fields names so the size of Hessian serialization is much
bigger. In my small example with selection of 6 objects Hessian
serialization size is more than 2400 bytes while Protostuff runtime is
about 800 bytes.
If we don't want to have ser/deser and size overhead we could find a way to
generate schemas via Velocity. And we should provide schemas for some
Cayenne classes. But it will require a lot of efforts.
2016-05-05 13:44 GMT+03:00 Aristedes Maniatis <ar...@maniatis.org>:
> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> > Protostuff (licensed under Apache 2.0 licence) is based on Google's
> > Protocol-Buffers (Protobuf) but has some optimizations and some cool
> things
> > like runtime serialization graph of objects (like Hessian). It also could
> > generate schema on runtime so we shouldn't define .proto files although
> it
> > might increase efficiency. It works faster than Hessian and could handle
> > Java8 Date and Time types. Here is some benchmarks. Take a look at Full
> > Object Graph Serializers section.
> > http://hperadin.github.io/jvm-serializers-report/report.html
> > https://github.com/eishay/jvm-serializers/wiki
>
> According to those benchmarks there appears to be no performance or size
> penalty to using protostuff over protobuffers. Am I reading that right?
>
> I don't really understand... doesn't the serialiser have to construct a
> .proto definition and then include it in the message? So shouldn't it be
> faster/smaller to predefine these?
>
> If we did, we could create them with velocity in the same way we create
> Java _superclasses today. Fairly trivial I'm guessing.
>
> Ari
>
>
> --
> -------------------------->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>
--
Thanks and Regards
Savva Kolbachev
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> Protostuff (licensed under Apache 2.0 licence) is based on Google's
> Protocol-Buffers (Protobuf) but has some optimizations and some cool things
> like runtime serialization graph of objects (like Hessian). It also could
> generate schema on runtime so we shouldn't define .proto files although it
> might increase efficiency. It works faster than Hessian and could handle
> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
> Object Graph Serializers section.
> http://hperadin.github.io/jvm-serializers-report/report.html
> https://github.com/eishay/jvm-serializers/wiki
According to those benchmarks there appears to be no performance or size penalty to using protostuff over protobuffers. Am I reading that right?
I don't really understand... doesn't the serialiser have to construct a .proto definition and then include it in the message? So shouldn't it be faster/smaller to predefine these?
If we did, we could create them with velocity in the same way we create Java _superclasses today. Fairly trivial I'm guessing.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: Cayenne ROP Improvements
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> Also I want to suggest to use Protostuff serialization instead of Hessian
> by default. What do you think about it?
Nicely done Savva. I know that our application will be the first off the rank to adopt this code, so once we run it through our testing and QA, I think that Cayenne should be switched to protostuff by default.
There are no downsides and plenty of upsides: better performance and fewer known bugs.
We'll know once we get the entire stack together inside a real application, but I've got high hopes that the combination of the new serialiser and HTTP/2 will have great benefits for applications running over high latency links ( > 50ms).
For others thinking about ROP, we've got it working really well once you keep in mind a combination of Cayenne pagination (which allows you to efficiently send a 100,000 hollow object list over the internet in a second or two) and prefetching (which dramatically improves performance by combining many high latency 'chatty' connections into a single request-response).
With HTTP/2 we can reduce the latency of establishing SSL connections (and add compression to the headers). And with protocol-buffers we'll be able to send LocalDate and other JDK8 objects without problem.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A