You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directmemory.apache.org by Roman Levenstein <ro...@gmail.com> on 2012/04/19 09:50:27 UTC
New version of protostuff with more efficient serialization is available
Hi,
I just wanted to mention on this mailing list, that there is a new
version of the Protostuff that supports a more efficient serialization
(see https://groups.google.com/forum/?fromgroups#!topic/protostuff/MeTR9J_ClSM)
If you make use of IdStrategies (see more info at
http://code.google.com/p/protostuff/wiki/ThingsYouNeedToKnow), you can
get a much smaller serialized representation and it is also much
faster to produce.
-Dprotostuff.runtime.id_strategy_factory=com.dyuproject.protostuff.runtime.IncrementalIdStrategy$Factory
By default (if property is not present), the DefaultIdStrategy
is used, which means a polymorphic pojo is identified by serializing
its type as a string (FQCN). If you set the above property, int ids
are generated on the fly (thread-safe/atomic) and are mapped to your
polymorphic pojos. The end result is faster ser/deser and the
serialized size is smaller (around 1/3-1/4 the size of the default
strategy)
You can also reserve the first few ids (via
IncrementalIdStrategy.Registry) for your core pojos (remember that ids
1-15 are serialized as 1-byte together with the protobuf tag), as well
as set the max size for the ArrayList which holds the ids.
I think DirectMemory can greatly benefit from this feature, because
now you can keep more objects in the same amount of memory and you get
much faster serialization/deserialization.
Best Regards,
Roman
P.S. If it is not against Apache policies, would it be possible to
have a Google Groups for the direct-memory mailing lists, so that they
can be used as a different web-based front-end? It could be much more
convenient for many people I guess.
Re: New version of protostuff with more efficient serialization is available
Posted by Olivier Lamy <ol...@apache.org>.
Hello,
Version upgraded in poms.
2012/4/19 Roman Levenstein <ro...@gmail.com>:
> Hi,
>
> I'm afraid that at the moment I'm not able to do it, because I'm
> rather busy with some other projects at work. And I do not use
> DirectMemory yet ;-(, but just plan to do so in the future. But I
> follow the development of this project with a big interest.
>
> As for the required changes, I think they can be rather trivial. There
> are some unit tests that show how to use the new IdStrategy feature:
>
> http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/IncrementalRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=6
>
> http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/ExplicitRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=7
>
> So, basically one needs to register a mapping from a class to its
> numeric id. Depending on the strategy, you can assign the numeric id
> yourself (ExplicitIdStrategy) or let protostuff do it for you
> (IncrementalIdStrategy). Once this is done, no further changes are
> required.
>
> See also this page for some info at
> http://code.google.com/p/protostuff/wiki/ProtostuffRuntime (in the
> Performance Guidelines section)
>
> -Roman
>
>
> On Thu, Apr 19, 2012 at 11:00 AM, Olivier Lamy <ol...@apache.org> wrote:
>> Hello,
>> Nice and thanks for the tip.
>> Would you like contribute a patch ?
>>
>> 2012/4/19 Roman Levenstein:
>>> Hi,
>>>
>>> I just wanted to mention on this mailing list, that there is a new
>>> version of the Protostuff that supports a more efficient serialization
>>> (see https://groups.google.com/forum/?fromgroups#!topic/protostuff/MeTR9J_ClSM)
>>>
>>> If you make use of IdStrategies (see more info at
>>> http://code.google.com/p/protostuff/wiki/ThingsYouNeedToKnow), you can
>>> get a much smaller serialized representation and it is also much
>>> faster to produce.
>>>
>>> -Dprotostuff.runtime.id_strategy_factory=com.dyuproject.protostuff.runtime.IncrementalIdStrategy$Factory
>>> By default (if property is not present), the DefaultIdStrategy
>>> is used, which means a polymorphic pojo is identified by serializing
>>> its type as a string (FQCN). If you set the above property, int ids
>>> are generated on the fly (thread-safe/atomic) and are mapped to your
>>> polymorphic pojos. The end result is faster ser/deser and the
>>> serialized size is smaller (around 1/3-1/4 the size of the default
>>> strategy)
>>> You can also reserve the first few ids (via
>>> IncrementalIdStrategy.Registry) for your core pojos (remember that ids
>>> 1-15 are serialized as 1-byte together with the protobuf tag), as well
>>> as set the max size for the ArrayList which holds the ids.
>>>
>>> I think DirectMemory can greatly benefit from this feature, because
>>> now you can keep more objects in the same amount of memory and you get
>>> much faster serialization/deserialization.
>>>
>>> Best Regards,
>>> Roman
>>>
>>> P.S. If it is not against Apache policies, would it be possible to
>>> have a Google Groups for the direct-memory mailing lists, so that they
>>> can be used as a different web-based front-end? It could be much more
>>> convenient for many people I guess.
>>
>> Not possible. That's against ASF policies.
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: New version of protostuff with more efficient serialization is available
Posted by Simone Tripodi <si...@apache.org>.
Thanks a lot Roman, much more than appreciated!
all the best,
-Simo
http://people.apache.org/~simonetripodi/
http://simonetripodi.livejournal.com/
http://twitter.com/simonetripodi
http://www.99soft.org/
On Thu, Apr 19, 2012 at 11:34 AM, Roman Levenstein <ro...@gmail.com> wrote:
> Hi,
>
> I'm afraid that at the moment I'm not able to do it, because I'm
> rather busy with some other projects at work. And I do not use
> DirectMemory yet ;-(, but just plan to do so in the future. But I
> follow the development of this project with a big interest.
>
> As for the required changes, I think they can be rather trivial. There
> are some unit tests that show how to use the new IdStrategy feature:
>
> http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/IncrementalRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=6
>
> http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/ExplicitRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=7
>
> So, basically one needs to register a mapping from a class to its
> numeric id. Depending on the strategy, you can assign the numeric id
> yourself (ExplicitIdStrategy) or let protostuff do it for you
> (IncrementalIdStrategy). Once this is done, no further changes are
> required.
>
> See also this page for some info at
> http://code.google.com/p/protostuff/wiki/ProtostuffRuntime (in the
> Performance Guidelines section)
>
> -Roman
>
>
> On Thu, Apr 19, 2012 at 11:00 AM, Olivier Lamy <ol...@apache.org> wrote:
>> Hello,
>> Nice and thanks for the tip.
>> Would you like contribute a patch ?
>>
>> 2012/4/19 Roman Levenstein:
>>> Hi,
>>>
>>> I just wanted to mention on this mailing list, that there is a new
>>> version of the Protostuff that supports a more efficient serialization
>>> (see https://groups.google.com/forum/?fromgroups#!topic/protostuff/MeTR9J_ClSM)
>>>
>>> If you make use of IdStrategies (see more info at
>>> http://code.google.com/p/protostuff/wiki/ThingsYouNeedToKnow), you can
>>> get a much smaller serialized representation and it is also much
>>> faster to produce.
>>>
>>> -Dprotostuff.runtime.id_strategy_factory=com.dyuproject.protostuff.runtime.IncrementalIdStrategy$Factory
>>> By default (if property is not present), the DefaultIdStrategy
>>> is used, which means a polymorphic pojo is identified by serializing
>>> its type as a string (FQCN). If you set the above property, int ids
>>> are generated on the fly (thread-safe/atomic) and are mapped to your
>>> polymorphic pojos. The end result is faster ser/deser and the
>>> serialized size is smaller (around 1/3-1/4 the size of the default
>>> strategy)
>>> You can also reserve the first few ids (via
>>> IncrementalIdStrategy.Registry) for your core pojos (remember that ids
>>> 1-15 are serialized as 1-byte together with the protobuf tag), as well
>>> as set the max size for the ArrayList which holds the ids.
>>>
>>> I think DirectMemory can greatly benefit from this feature, because
>>> now you can keep more objects in the same amount of memory and you get
>>> much faster serialization/deserialization.
>>>
>>> Best Regards,
>>> Roman
>>>
>>> P.S. If it is not against Apache policies, would it be possible to
>>> have a Google Groups for the direct-memory mailing lists, so that they
>>> can be used as a different web-based front-end? It could be much more
>>> convenient for many people I guess.
>>
>> Not possible. That's against ASF policies.
Re: New version of protostuff with more efficient serialization is available
Posted by Roman Levenstein <ro...@gmail.com>.
Hi,
I'm afraid that at the moment I'm not able to do it, because I'm
rather busy with some other projects at work. And I do not use
DirectMemory yet ;-(, but just plan to do so in the future. But I
follow the development of this project with a big interest.
As for the required changes, I think they can be rather trivial. There
are some unit tests that show how to use the new IdStrategy feature:
http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/IncrementalRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=6
http://code.google.com/searchframe#0-WbZJYkJ-4/trunk/protostuff-runtime-registry/src/test/java/com/dyuproject/protostuff/runtime/ExplicitRuntimeObjectSchemaTest.java&q=IdStrategy%20package:http://protostuff\.googlecode\.com&sq=&ct=rc&cd=7
So, basically one needs to register a mapping from a class to its
numeric id. Depending on the strategy, you can assign the numeric id
yourself (ExplicitIdStrategy) or let protostuff do it for you
(IncrementalIdStrategy). Once this is done, no further changes are
required.
See also this page for some info at
http://code.google.com/p/protostuff/wiki/ProtostuffRuntime (in the
Performance Guidelines section)
-Roman
On Thu, Apr 19, 2012 at 11:00 AM, Olivier Lamy <ol...@apache.org> wrote:
> Hello,
> Nice and thanks for the tip.
> Would you like contribute a patch ?
>
> 2012/4/19 Roman Levenstein:
>> Hi,
>>
>> I just wanted to mention on this mailing list, that there is a new
>> version of the Protostuff that supports a more efficient serialization
>> (see https://groups.google.com/forum/?fromgroups#!topic/protostuff/MeTR9J_ClSM)
>>
>> If you make use of IdStrategies (see more info at
>> http://code.google.com/p/protostuff/wiki/ThingsYouNeedToKnow), you can
>> get a much smaller serialized representation and it is also much
>> faster to produce.
>>
>> -Dprotostuff.runtime.id_strategy_factory=com.dyuproject.protostuff.runtime.IncrementalIdStrategy$Factory
>> By default (if property is not present), the DefaultIdStrategy
>> is used, which means a polymorphic pojo is identified by serializing
>> its type as a string (FQCN). If you set the above property, int ids
>> are generated on the fly (thread-safe/atomic) and are mapped to your
>> polymorphic pojos. The end result is faster ser/deser and the
>> serialized size is smaller (around 1/3-1/4 the size of the default
>> strategy)
>> You can also reserve the first few ids (via
>> IncrementalIdStrategy.Registry) for your core pojos (remember that ids
>> 1-15 are serialized as 1-byte together with the protobuf tag), as well
>> as set the max size for the ArrayList which holds the ids.
>>
>> I think DirectMemory can greatly benefit from this feature, because
>> now you can keep more objects in the same amount of memory and you get
>> much faster serialization/deserialization.
>>
>> Best Regards,
>> Roman
>>
>> P.S. If it is not against Apache policies, would it be possible to
>> have a Google Groups for the direct-memory mailing lists, so that they
>> can be used as a different web-based front-end? It could be much more
>> convenient for many people I guess.
>
> Not possible. That's against ASF policies.
Re: New version of protostuff with more efficient serialization is available
Posted by Olivier Lamy <ol...@apache.org>.
Hello,
Nice and thanks for the tip.
Would you like contribute a patch ?
2012/4/19 Roman Levenstein <ro...@gmail.com>:
> Hi,
>
> I just wanted to mention on this mailing list, that there is a new
> version of the Protostuff that supports a more efficient serialization
> (see https://groups.google.com/forum/?fromgroups#!topic/protostuff/MeTR9J_ClSM)
>
> If you make use of IdStrategies (see more info at
> http://code.google.com/p/protostuff/wiki/ThingsYouNeedToKnow), you can
> get a much smaller serialized representation and it is also much
> faster to produce.
>
> -Dprotostuff.runtime.id_strategy_factory=com.dyuproject.protostuff.runtime.IncrementalIdStrategy$Factory
> By default (if property is not present), the DefaultIdStrategy
> is used, which means a polymorphic pojo is identified by serializing
> its type as a string (FQCN). If you set the above property, int ids
> are generated on the fly (thread-safe/atomic) and are mapped to your
> polymorphic pojos. The end result is faster ser/deser and the
> serialized size is smaller (around 1/3-1/4 the size of the default
> strategy)
> You can also reserve the first few ids (via
> IncrementalIdStrategy.Registry) for your core pojos (remember that ids
> 1-15 are serialized as 1-byte together with the protobuf tag), as well
> as set the max size for the ArrayList which holds the ids.
>
> I think DirectMemory can greatly benefit from this feature, because
> now you can keep more objects in the same amount of memory and you get
> much faster serialization/deserialization.
>
> Best Regards,
> Roman
>
> P.S. If it is not against Apache policies, would it be possible to
> have a Google Groups for the direct-memory mailing lists, so that they
> can be used as a different web-based front-end? It could be much more
> convenient for many people I guess.
Not possible. That's against ASF policies.
--
Olivier Lamy
Talend: http://coders.talend.com
http://twitter.com/olamy | http://linkedin.com/in/olamy