double serialization/deserialization issue in Java/Python

[Andrew Watts <an...@gmail.com>]

I'm writing a python client that needs to deserialize a thrift blob
that was serialized in a java server. Both sides use serializers and
the CompactProtocol from their respective libraries.  Java writes the
blob to a log file as a base64 encoded string, then python reads the
base64 encoded string from the log.  During development I discovered
double values were not being deserialized correctly.  However, when
using the Binary Protocol, the value is deserialized correctly.  Am I
doing something incorrectly or this there a possible bug in either the
python or java implementation of the compact protocol.

Below is a simple thrift definition, java program and python program
that demonstrate the problem.

Thanks.

- Andrew


» java -cp .:../lib/libthrift-0.8.0.jar:../lib/slf4j-api-1.6.4.jar ThriftTest
fooObj.bar: 3.456
base64String: F9nO91PjpQtAAA==

» python ../python/tdouble_test.py
base64 string:  F9nO91PjpQtAAA==
foo_obj.bar:  -4.09406819342e+124    # expect 3.456


Thrift Definition
================

struct FooObj {
    1: double bar
}


Java Code
=========

import org.apache.thrift.TException;
import org.apache.thrift.TSerializer;
import org.apache.thrift.protocol.TCompactProtocol;

import javax.xml.bind.DatatypeConverter;


public class ThriftTest {

    public static void main(String[] args) {

        final TSerializer serializer = new TSerializer(new
TCompactProtocol.Factory());

        // create a FooObj with double
        final FooObj fooObj = new FooObj(3.456);
        System.out.println("fooObj.bar: " + fooObj.bar);

        // serialize to bytes
        byte[] fooObjBlob = null;
        try {
            fooObjBlob = serializer.serialize(fooObj);
        } catch (TException e) {
            e.printStackTrace();
        }

        // encode to base64 string
        final String base64String =
DatatypeConverter.printBase64Binary(fooObjBlob);
        System.out.println("base64String: " + base64String);
    }

}


Python Code
===========

#!/bin/env python

import base64

from thrift.protocol import TCompactProtocol
from thrift.TSerialization import deserialize

from foo.ttypes import FooObj


def main():

    protocol_factory = TCompactProtocol.TCompactProtocolFactory
    base64_string = 'F9nO91PjpQtAAA=='

    print 'base64 string: ', base64_string

    # deserialize the string back into an object
    foo_blob = base64.urlsafe_b64decode(base64_string)
    foo_obj = FooObj()
    deserialize(foo_obj, foo_blob, protocol_factory=protocol_factory())
    print 'foo_obj.bar: ', foo_obj.bar

if __name__ == '__main__':
    main()

Re: double serialization/deserialization issue in Java/Python

[Bryan Duxbury <br...@gmail.com>]

I suspect there's a mismatch in the serialization/deserialization between
the languages. Can you open a ticket?

On Wed, Jun 27, 2012 at 8:11 PM, Andrew Watts <an...@gmail.com> wrote:

> I'm writing a python client that needs to deserialize a thrift blob
> that was serialized in a java server. Both sides use serializers and
> the CompactProtocol from their respective libraries.  Java writes the
> blob to a log file as a base64 encoded string, then python reads the
> base64 encoded string from the log.  During development I discovered
> double values were not being deserialized correctly.  However, when
> using the Binary Protocol, the value is deserialized correctly.  Am I
> doing something incorrectly or this there a possible bug in either the
> python or java implementation of the compact protocol.
>
> Below is a simple thrift definition, java program and python program
> that demonstrate the problem.
>
> Thanks.
>
> - Andrew
>
>
> » java -cp .:../lib/libthrift-0.8.0.jar:../lib/slf4j-api-1.6.4.jar
> ThriftTest
> fooObj.bar: 3.456
> base64String: F9nO91PjpQtAAA==
>
> » python ../python/tdouble_test.py
> base64 string:  F9nO91PjpQtAAA==
> foo_obj.bar:  -4.09406819342e+124    # expect 3.456
>
>
> Thrift Definition
> ================
>
> struct FooObj {
>     1: double bar
> }
>
>
> Java Code
> =========
>
> import org.apache.thrift.TException;
> import org.apache.thrift.TSerializer;
> import org.apache.thrift.protocol.TCompactProtocol;
>
> import javax.xml.bind.DatatypeConverter;
>
>
> public class ThriftTest {
>
>    public static void main(String[] args) {
>
>        final TSerializer serializer = new TSerializer(new
> TCompactProtocol.Factory());
>
>        // create a FooObj with double
>        final FooObj fooObj = new FooObj(3.456);
>        System.out.println("fooObj.bar: " + fooObj.bar);
>
>        // serialize to bytes
>        byte[] fooObjBlob = null;
>        try {
>            fooObjBlob = serializer.serialize(fooObj);
>        } catch (TException e) {
>            e.printStackTrace();
>        }
>
>        // encode to base64 string
>        final String base64String =
> DatatypeConverter.printBase64Binary(fooObjBlob);
>        System.out.println("base64String: " + base64String);
>    }
>
> }
>
>
> Python Code
> ===========
>
> #!/bin/env python
>
> import base64
>
> from thrift.protocol import TCompactProtocol
> from thrift.TSerialization import deserialize
>
> from foo.ttypes import FooObj
>
>
> def main():
>
>    protocol_factory = TCompactProtocol.TCompactProtocolFactory
>    base64_string = 'F9nO91PjpQtAAA=='
>
>    print 'base64 string: ', base64_string
>
>    # deserialize the string back into an object
>    foo_blob = base64.urlsafe_b64decode(base64_string)
>    foo_obj = FooObj()
>    deserialize(foo_obj, foo_blob, protocol_factory=protocol_factory())
>    print 'foo_obj.bar: ', foo_obj.bar
>
> if __name__ == '__main__':
>    main()
>