You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Leo Romanoff (JIRA)" <ji...@apache.org> on 2013/03/22 18:09:15 UTC

[jira] [Commented] (AVRO-1282) Make use of the sun.misc.Unsafe class during serialization if a JDK supports it

    [ https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13610942#comment-13610942 ] 

Leo Romanoff commented on AVRO-1282:
------------------------------------

One thing I'd like to ask is the following:

I've implemented this kind of optimization for Kryo and the results are impressive.
While doing this, I almost have rewritten most of the Input/Output classes from Kryo. They now support Unsafe-based streams and contribute the most to the speed ups. Since these IO classes are pretty self-contained and do not really depend on Kryo, it could be interesting to reuse them for Avro (and may be for other serialization frameworks). 

What do you think about this idea?

Note:
One of the biggest wins for Kryo was Unsafe-based serialization of arrays. In this mode, Kryo would use native byte order and native size of primitive types (because this is what Unsafe supports), but it can serialize whole arrays of primitive types extremely quickly. Overall, most of speed ups coming from Unsafe-based streams are due to the usage of very efficient native byte order operations by Unsafe. E.g. it can serialize primitive types as a whole and not byte-by-byte.

Since Avro uses a platform-independent format, this feature cannot be used with Avro then I guess. Or one could enable this mode optionally with a disclaimer that is makes serialized representation non-portable between platforms with different byte-orders.
                
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports it
> -------------------------------------------------------------------------------
>
>                 Key: AVRO-1282
>                 URL: https://issues.apache.org/jira/browse/AVRO-1282
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.7.4
>            Reporter: Leo Romanoff
>            Priority: Minor
>
> Unsafe can be used to significantly speed up serialization process, if a JDK implementation supports java.misc.Unsafe properly. Most JDKs running on PCs support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick input/output.
>  
> 3) More over, Unsafe makes it possible to serialize to/deserialize from off-heap memory directly and very quickly, without any intermediate buffers allocated on heap. There is virtually no overhead compared to the usual byte arrays.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira