Dynamic serializer for java

[Martin Traverso <mt...@gmail.com>]

Hi all,
I'm working on thrift-based serializer/deserializer for Java that does not
require generating code. It works by creating bytecode on the fly (using
ASM) and can read from TProtocol into Maps or plain javabeans, and
viceversa.

There's lots still missing, but the basic uses cases are working. I thought
you guys might find it interesting. The code is up at
http://github.com/martint/swift

Cheers,
Martin

Re: Dynamic serializer for java

[Pete Wyckoff <pw...@facebook.com>]

There's also something to be said for compile time checks in Java and C++.

> Mainly, the ability to deserialize data in consumers without knowing the schema at development time

So, this is using thrift for storing structured data and later deserializing it?


On 11/11/08 10:41 AM, "Chad Walters" <Ch...@microsoft.com> wrote:

I think there is a place for both static generation and dynamic generation -- I'd personally love to see Thrift embrace both.

Getting high performance in C++ is one of the key drivers of static generation, I think. I realize that not everyone is interested in C++, but Thrift's heritage includes a healthy dose of C++.

Chad

On 11/11/08 9:38 AM, "Martin Traverso" <mt...@gmail.com> wrote:

> Interesting. What is your use case?


Mainly, the ability to deserialize data in consumers without knowing the
schema at development time. But there's a couple of other things that
prompted me to try this approach:
* Serializable objects are not tied to Thrift's hierarchy. They don't need
to extend from TBase. The only requirement is for them to have appropriate
setters and getters. Or if you don't even have javabeans, you can get the
data placed in a Map<String, Object>
* I think static code generation is not the right approach in general. It
makes development more cumbersome. I know the analogy is not perfect, but
compare to old day RMI where you had to create the stub and skeleton files
from your class file and today's approach, where it uses dynamic proxies
instead.

Martin

Re: Dynamic serializer for java

[Chad Walters <Ch...@microsoft.com>]

I think there is a place for both static generation and dynamic generation -- I'd personally love to see Thrift embrace both.

Getting high performance in C++ is one of the key drivers of static generation, I think. I realize that not everyone is interested in C++, but Thrift's heritage includes a healthy dose of C++.

Chad

On 11/11/08 9:38 AM, "Martin Traverso" <mt...@gmail.com> wrote:

> Interesting. What is your use case?


Mainly, the ability to deserialize data in consumers without knowing the
schema at development time. But there's a couple of other things that
prompted me to try this approach:
* Serializable objects are not tied to Thrift's hierarchy. They don't need
to extend from TBase. The only requirement is for them to have appropriate
setters and getters. Or if you don't even have javabeans, you can get the
data placed in a Map<String, Object>
* I think static code generation is not the right approach in general. It
makes development more cumbersome. I know the analogy is not perfect, but
compare to old day RMI where you had to create the stub and skeleton files
from your class file and today's approach, where it uses dynamic proxies
instead.

Martin

Re: Dynamic serializer for java

[Martin Traverso <mt...@gmail.com>]

> Interesting. What is your use case?


Mainly, the ability to deserialize data in consumers without knowing the
schema at development time. But there's a couple of other things that
prompted me to try this approach:
* Serializable objects are not tied to Thrift's hierarchy. They don't need
to extend from TBase. The only requirement is for them to have appropriate
setters and getters. Or if you don't even have javabeans, you can get the
data placed in a Map<String, Object>
* I think static code generation is not the right approach in general. It
makes development more cumbersome. I know the analogy is not perfect, but
compare to old day RMI where you had to create the stub and skeleton files
from your class file and today's approach, where it uses dynamic proxies
instead.

Martin

Re: Dynamic serializer for java

[Torsten Curdt <tc...@apache.org>]

...and there is another code base.

 http://github.com/tcurdt/drift/tree/master

I've implemented the things I brought up on the mailing list before as
a proof of concept (very simple extension mechanism and hash based
slot numbers) and refactored the java code base. If you separate the
serialization from the rpc focus things like migrations become much
more important. At least linear migrations are supported by drift
already.

 http://github.com/tcurdt/drift/tree/master/src/test/resources/grammars/Complex.dg

I met Martin at ApacheCon in New Orleans and found it quite
interesting we have similar gripes about Thrift.

cheers
--
Torsten

Re: Dynamic serializer for java

[Pete Wyckoff <pw...@facebook.com>]

Interesting. What is your use case? The Hadoop Hive project has something very similar although the implementation is different. The use case there is for serializing/deserializing data in a data warehouse; it can serialize/deserialize thrift data but also has some optimizations when all the fields are required and also for skipping deserializing fields when they won't be used.

-- pete


On 11/10/08 9:13 AM, "Martin Traverso" <mt...@gmail.com> wrote:

Hi all,
I'm working on thrift-based serializer/deserializer for Java that does not
require generating code. It works by creating bytecode on the fly (using
ASM) and can read from TProtocol into Maps or plain javabeans, and
viceversa.

There's lots still missing, but the basic uses cases are working. I thought
you guys might find it interesting. The code is up at
http://github.com/martint/swift

Cheers,
Martin