You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Chad Walters <Ch...@microsoft.com> on 2009/05/12 11:57:38 UTC
Report on thrift-protobuf-compare
I did some digging into the benchmarking code at:
http://code.google.com/p/thrift-protobuf-compare/
If folks could look this over and give me comments, I'll make any edits
suggested and pass this information along to the owner of
thrift-protobuf-compare.
Here is the short version:
1. The performance figures that thrift-protobuf-compare provides for dynamic
serialization systems like JSON are not currently valid since the tests do
not really test them as a fully general serialization/deserialization
framework.
2. Using TCompactProtocol, Thrift serialization speed and serialized size
are basically equivalent to protocol buffers
3. FWIW object creation is quite a bit faster in Thrift than it is in
protocol buffers, at least in this test. I am not sure how important this is
for real world workloads.
4. Deserialization comes out slower in Thrift than in protocol buffers. I
think this may be because the test harness' API matches up directly with
APIs that protocol buffers provides that Thrift does not, causing Thrift to
do more work. I am still investigating this.
Here is the long version:
First, the test is not comparing apples-to-apples when it compares any of
the dynamic serialization systems (JSON, yaml, etc.) with the static
serialization systems (Thrift, protocol buffers).
For example, the JSON serialization and deserialization looks like this:
public byte[] serialize(MediaContent content) throws Exception
{
ByteArrayOutputStream baos = new ByteArrayOutputStream(expectedSize);
JsonGenerator generator = _factory.createJsonGenerator(baos,
JsonEncoding.UTF8);
generator.writeStartObject();
writeMedia(generator, content.getMedia());
writeImage(generator, content.getImage(0));
writeImage(generator, content.getImage(1));
generator.writeEndObject();
generator.close();
byte[] array = baos.toByteArray();
expectedSize = array.length;
return array;
}
public MediaContent deserialize(byte[] array) throws Exception
{
JsonParser parser = _factory.createJsonParser(array);
parser.nextToken(); // start object
MediaContent mc = new MediaContent(readMedia(parser));
mc.addImage(readImage(parser));
mc.addImage(readImage(parser));
parser.nextToken(); // end object
parser.close();
return mc;
}
Notice that the serializer somehow knows that there are two images to be
written without looping over the object being serialized and that the
deserializer somehow knows that there are two images to be read without any
metadata to carry that information in the serialized form.
The upshot is that, as written, none of the dynamic mechanisms are really
being used in a manner that results in a valid serialization/deserialization
framework and any performance numbers derived from them are invalid.
Furthermore, the data being serialized is not the same. Compare this method
from StdMediaSerializer.java:
public final MediaContent create() throws Exception
{
Media media = new Media(null, "video/mpg4", Media.Player.JAVA,
"Javaone Keynote", "http:
//javaone.com/keynote.mpg", 1234567, 123, 0, 0, 0);
media.addToPerson("Bill Gates");
media.addToPerson("Steve Jobs");
Image image1 = new Image(0, "Javaone Keynote", "A", 0,
Image.Size.LARGE);
Image image2 = new Image(0, "Javaone Keynote", "B", 0,
Image.Size.SMALL);
MediaContent content = new MediaContent(media);
content.addImage(image1);
content.addImage(image2);
return content;
}
With the same method in ThriftSerializer.java:
public MediaContent create()
{
Media media = new Media();
media.setUri("http://javaone.com/keynote.mpg");
media.setFormat("video/mpg4");
media.setTitle("Javaone Keynote");
media.setDuration(1234567);
media.setBitrate(123);
media.addToPerson("Bill Gates");
media.addToPerson("Steve Jobs");
media.setPlayer(Player.JAVA);
Image image1 = new Image();
image1.setUri("http://javaone.com/keynote_large.jpg");
image1.setSize(Size.LARGE);
image1.setTitle("Javaone Keynote");
Image image2 = new Image("http://javaone.com/keynote_thumbnail.jpg",
"Javaone Keynote", -1, -1, Size.SMALL);
MediaContent content = new MediaContent();
content.setMedia(media);
content.addToImage(image1);
content.addToImage(image2);
return content;
}
And in ProtobufSerializer.java:
public MediaContent create()
{
MediaContent content = MediaContent.newBuilder().
setMedia(
Media.newBuilder().setUri("http://javaone.com/keynote.mpg").setFormat("video
/mpg4").setTit
le("Javaone Keynote").setDuration(1234567).
setBitrate(123).addPerson("Bill Gates").addPerson("Steve
Jobs").setPlayer(Player.JAVA).b
uild()).
addImage(
Image.newBuilder().setUri("http://javaone.com/keynote_large.jpg").setSize(Si
ze.LARGE).setT
itle("Javaone Keynote").build()).
addImage(
Image.newBuilder().setUri("http://javaone.com/keynote_thumbnail.jpg").setSiz
e(Size.SMALL).
setTitle("Javaone Keynote").build()).
build();
return content;
}
Note that the URIs for the two images are different (shorter) in
StdMediaSerializer than they are in the case of the other two. And even
between ThriftSerializer and ProtobufSerializer, the data is not identical
-- the ThriftSerializer is passing two -1 values that for image2 that
ProtobufSerializer is not passing.
It is too much effort to fix the dynamic serializers at the moment, so I
decided just to focus on using the code comparing Thrift and protocol
buffers.
I updated to the trunk of Thrift (rev 773454) and changed the Thrift
serializer to use TCompactProtocol instead of TBinaryProtocol. I also
corrected ThriftSerializer's create() so that the same data was being sent
for image2 as in ProtobufSerializer. Finally, I updated the formatting in
BenchmarkRunner and commented out all the serializers except Thrift and
protocol buffers. Here are the benchmark results from 3 consecutive runs:
, Create, Ser, Deser, Total, Size
thrift , 267.37, 8314.00, 8546.00, 17127.36, 220
protobuf , 412.98, 12642.00, 5217.50, 18272.48, 217
, Create, Ser, Deser, Total, Size
thrift , 266.87, 10905.50, 8526.50, 19698.86, 220
protobuf , 415.21, 11880.50, 4930.00, 17225.71, 217
, Create, Ser, Deser, Total, Size
thrift , 264.95, 11059.50, 8701.50, 20025.95, 220
protobuf , 417.45, 11125.00, 5203.50, 16745.95, 217
Note that TCompactProtocol is almost as compact as protocol buffers and
performs slightly better on serialization (although there is clearly some
variability in the timings). Object creation is significantly faster in
Thrift than in protocol buffers although it is unclear whether that is
really important.
Looking a little further at deserialization since Thrift seemed to be
performing worse than protocol buffers there, the problem may be related to
the fact that protocol buffers provides APIs that support direct
serialization to and deserialization from byte arrays which Thrift does not
provide. The test harness is set up such that the output of serialize() and
the input of deserialize() is a byte array, so this means that Thrift needs
to do more work to match up with the test harness. I am still investigating
this.
Chad
Re: Report on thrift-protobuf-compare
Posted by Alexander Shigin <sh...@rambler-co.ru>.
I haven't own any macbook, but it seems "sysctl hw.cpufreq...." controls
frequency of the cpu.
I've repeat your test. I can do something wrong (I've got warning
message: "serializer 'java (externalizable)' failed round-trip test" +
scala + sbinary). And I've got different serialized size for thrift.
I've switched to protobuf-java-2.1.0, because the 2.0.2 library is
incompatible with 2.1.0 generated source (I won't use protobuf if they
make incompatible changes so often).
Here are results:
------------------------------------------------------------------------
Starting
Object create, Serialization, Deserialization, Total Time, SSize
thrift 930,47903, 7570,38250, 9062,38350, 17563,24503, 314
thrift 903,62937, 7662,11150, 8719,14350, 17284,88437, 314
thrift 891,74183, 7537,20100, 8809,45650, 17238,39933, 314
thrift 898,90696, 7605,01650, 9067,40050, 17571,32396, 314
thrift 904,09775, 7695,88850, 9237,59250, 17837,57875, 314
protobuf 1066,90265, 8909,92600, 5278,93150, 15255,76015, 217
protobuf 1042,76173, 8881,13450, 5313,92500, 15237,82123, 217
protobuf 1065,49403, 9196,53650, 5112,25400, 15374,28453, 217
protobuf 1047,32115, 8983,84900, 5092,27150, 15123,44165, 217
protobuf 1109,47716, 9291,45350, 5354,72300, 15755,65366, 217
------------------------------------------------------------------------
There isn't any major difference in serialization or deserialization
time. The difference between lowest and highest serialization (time?) is
just 2% for thrift and 4% for protobuf (deserialization: 5% for thrift
and protobuf).
------------------------------------------------------------------------
$ java -version
java version "1.6.0_0"
IcedTea6 1.3.1 (6b12-0ubuntu6.4) Runtime Environment (build 1.6.0_0-b12)
OpenJDK Client VM (build 1.6.0_0-b12, mixed mode, sharing)
Genuine Intel(R) CPU 2160 @ 1.80GHz (dual core)
Linux atlas 2.6.27-11-generic #1 SMP Thu Jan 29 19:24:39 UTC 2009 i686
------------------------------------------------------------------------
Should I change something in my setup?
В Птн, 15/05/2009 в 07:24 -0700, Chad Walters пишет:
> Given the fact that the deserialization time is basically constant, I
> am not sure this is the issue.
>
> This is on my MacBook Pro laptop, plugged in and with no particular
> special settings — if you know something I should enable/disable, let
> me know.
>
> Chad
>
> On 5/15/09 12:10 AM, "Alexander Shigin" <sh...@rambler-co.ru> wrote:
>
> В Чтв, 14/05/2009 в 23:43 -0700, Chad Walters пишет:
> > I suspect it may have something to do with GC but haven't proved that
> > out yet. Notice that serialization time is highly variable but
> > deserialization time is almost constant.
> >
> > Any thoughts?
>
> Do you have any power-save technology enabled? I've come across some
> strange results when trying to do benchmarks with "cpufreq" enabled.
>
>
Re: Report on thrift-protobuf-compare
Posted by Chad Walters <Ch...@microsoft.com>.
Given the fact that the deserialization time is basically constant, I am not sure this is the issue.
This is on my MacBook Pro laptop, plugged in and with no particular special settings — if you know something I should enable/disable, let me know.
Chad
On 5/15/09 12:10 AM, "Alexander Shigin" <sh...@rambler-co.ru> wrote:
В Чтв, 14/05/2009 в 23:43 -0700, Chad Walters пишет:
> I suspect it may have something to do with GC but haven't proved that
> out yet. Notice that serialization time is highly variable but
> deserialization time is almost constant.
>
> Any thoughts?
Do you have any power-save technology enabled? I've come across some
strange results when trying to do benchmarks with "cpufreq" enabled.
Re: Report on thrift-protobuf-compare
Posted by Alexander Shigin <sh...@rambler-co.ru>.
В Чтв, 14/05/2009 в 23:43 -0700, Chad Walters пишет:
> I suspect it may have something to do with GC but haven't proved that
> out yet. Notice that serialization time is highly variable but
> deserialization time is almost constant.
>
> Any thoughts?
Do you have any power-save technology enabled? I've come across some
strange results when trying to do benchmarks with "cpufreq" enabled.
Re: Report on thrift-protobuf-compare
Posted by Chad Walters <Ch...@microsoft.com>.
I thought about that so I did multiple runs of each serialization method in a row. Here are the results of running multiple runs in a row for Thrift and protocol buffers.
, Object create, Serialization, Deserialization, Total Time, Serialized Size
thrift , 236.12000, 7585.00000, 6664.00000, 14485.12000, 220
thrift , 235.90500, 7222.50000, 6709.50000, 14167.90500, 220
thrift , 237.42500, 7381.50000, 6688.50000, 14307.42500, 220
thrift , 234.52500, 10247.50000, 6665.50000, 17147.52500, 220
thrift , 241.17000, 7336.50000, 6666.50000, 14244.17000, 220
thrift , 235.93500, 9966.50000, 6683.00000, 16885.43500, 220
thrift , 236.10000, 7133.00000, 6660.50000, 14029.60000, 220
thrift , 235.43000, 10715.00000, 6660.50000, 17610.93000, 220
thrift , 233.83000, 10384.50000, 6690.50000, 17308.83000, 220
protobuf , 375.61000, 10807.00000, 4895.00000, 16077.61000, 217
protobuf , 383.19000, 7112.00000, 5039.50000, 12534.69000, 217
protobuf , 382.63000, 7166.50000, 4857.50000, 12406.63000, 217
protobuf , 375.37000, 9521.50000, 4895.00000, 14791.87000, 217
protobuf , 376.77000, 11855.00000, 4817.00000, 17048.77000, 217
protobuf , 373.11000, 10333.50000, 4854.50000, 15561.11000, 217
I suspect it may have something to do with GC but haven't proved that out yet. Notice that serialization time is highly variable but deserialization time is almost constant.
Any thoughts?
Chad
On 5/14/09 9:49 PM, "Doug Cutting" <cu...@apache.org> wrote:
Chad Walters wrote:
> I can't yet explain why the serialization time can range from ~7500 to ~12000 at times, but it bounces around for protocol buffers too.
Are you throwing away the first run of each benchmark in a given JVM?
I've found that to greatly stabilize many Java benchmarks.
Doug
Re: Report on thrift-protobuf-compare
Posted by Doug Cutting <cu...@apache.org>.
Chad Walters wrote:
> I can't yet explain why the serialization time can range from ~7500 to ~12000 at times, but it bounces around for protocol buffers too.
Are you throwing away the first run of each benchmark in a given JVM?
I've found that to greatly stabilize many Java benchmarks.
Doug
Re: Report on thrift-protobuf-compare
Posted by Chad Walters <Ch...@microsoft.com>.
I am not sure that the JVM can kick in any further optimizations - repeated runs of 10000 serializations and deserializations seem to me like they should trigger anything that could feasibly be done.
I was able to get results as follows by making some minor mods to TMemoryBuffer and then making use of that via some changes to the test's ThriftSerializer and BenchmarkRunner (and using TCompactProtocol again of course):
, Create, Ser, Deser, Total, Size
thrift , 265.73, 11287.50, 6813.50, 18366.73, 220
thrift , 266.33, 10385.50, 6784.00, 17435.83, 220
thrift , 262.85, 8589.50, 6833.00, 15685.35, 220
thrift , 266.49, 9415.50, 6903.50, 16585.49, 220
thrift , 266.21, 10587.00, 6873.00, 17726.21, 220
thrift , 267.14, 7478.50, 6911.50, 14657.14, 220
protobuf , 416.11, 11679.00, 4982.50, 17077.61, 217
protobuf , 438.74, 8211.00, 5434.00, 14083.74, 217
protobuf , 419.93, 11504.00, 5403.50, 17327.43, 217
Deserialization is still slower than protobuf but it is closer. Essentially, the speedups come from not creating new transports and protocols each request. The way the test works kind of messes with Thrift's transport model on deserialization. The test serializes once and then deserializes from that array 10000 times - but Thrift consumes the bytes.
So I had to move the serialization into the loop but outside the timer and also reset the TMemoryBuffer at the start of serialization so that I could reuse the TMemoryBuffer over and over again.
I can't yet explain why the serialization time can range from ~7500 to ~12000 at times, but it bounces around for protocol buffers too.
Still more digging to do...
Chad
On 5/12/09 12:50 PM, "David Reiss" <dr...@facebook.com> wrote:
> Looking a little further at deserialization since Thrift seemed to be
> performing worse than protocol buffers there, the problem may be related to
> the fact that protocol buffers provides APIs that support direct
> serialization to and deserialization from byte arrays which Thrift does not
> provide. The test harness is set up such that the output of serialize() and
> the input of deserialize() is a byte array, so this means that Thrift needs
> to do more work to match up with the test harness. I am still investigating
> this.
My hope is that if you run the benchmark multiple times before the actual run
that you examine, the n will notice that read and write are often being called
with a TCompactProtocol and TMemoryBuffer and inline the calls.
--David
Re: Report on thrift-protobuf-compare
Posted by David Reiss <dr...@facebook.com>.
> Looking a little further at deserialization since Thrift seemed to be
> performing worse than protocol buffers there, the problem may be related to
> the fact that protocol buffers provides APIs that support direct
> serialization to and deserialization from byte arrays which Thrift does not
> provide. The test harness is set up such that the output of serialize() and
> the input of deserialize() is a byte array, so this means that Thrift needs
> to do more work to match up with the test harness. I am still investigating
> this.
My hope is that if you run the benchmark multiple times before the actual run
that you examine, the n will notice that read and write are often being called
with a TCompactProtocol and TMemoryBuffer and inline the calls.
--David
Re: Report on thrift-protobuf-compare
Posted by Bryan Duxbury <br...@rapleaf.com>.
Ah, gotcha.
On May 12, 2009, at 9:59 AM, Chad Walters wrote:
> On 5/12/09 9:20 AM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>>> 1. The performance figures that thrift-protobuf-compare provides
>>> for dynamic
>>> serialization systems like JSON are not currently valid since the
>>> tests do
>>> not really test them as a fully general serialization/
>>> deserialization
>>> framework.
>>
>> You mean that since they coded their json serialization directly to
>> their test data, the performance data is inaccurate? This might be
>> true in the sense that their serialization doesn't do some things
>> that Thrift does, but since we generate code, I don't think it's
>> *that* different.
>
> No, it's much worse than that, at least as I see it. The dynamic
> serializers are quite tightly bound to the data that is provided in
> the create() method, not just to the data structure. This means
> that they are "cheating" in ways that allow them to avoid putting
> In metadata they would need to do valid serialization/
> deserialization in a general way and they also avoid doing a bunch
> of switching that they would be required to do things correctly.
>
> Look at StdMediaSerializer::create():
>
> public final MediaContent create() throws Exception
> {
> Media media = new Media(null, "video/mpg4",
> Media.Player.JAVA, "Javaone Keynote", "htt
> p://javaone.com/keynote.mpg", 1234567, 123, 0, 0, 0);
> media.addToPerson("Bill Gates");
> media.addToPerson("Steve Jobs");
>
> Image image1 = new Image(0, "Javaone Keynote", "A", 0,
> Image.Size.LARGE);
> Image image2 = new Image(0, "Javaone Keynote", "B", 0,
> Image.Size.SMALL);
>
> MediaContent content = new MediaContent(media);
> content.addImage(image1);
> content.addImage(image2);
> return content;
> }
>
> Now look at JsonSerializer::deserialize():
>
> public MediaContent deserialize(byte[] array) throws Exception
> {
> JsonParser parser = _factory.createJsonParser(array);
> parser.nextToken(); // start object
> MediaContent mc = new MediaContent(readMedia(parser));
> mc.addImage(readImage(parser));
> mc.addImage(readImage(parser));
> parser.nextToken(); // end object
> parser.close();
> return mc;
> }
>
> Note that deserialize() assumes that there are 2 images. Suppose
> create were changed to add 3 images instead. Then the roundtrip
> would, at best, discard one of the images.
>
> Digging in a little deeper, these are supposed to be optional
> fields (at least they are marked so in media.thrift). However, the
> JsonSerializer's deserialize() assumes that the exact fields
> initialized in create() will all be present in the exact order they
> were serialized (look at findToken()) at deserialization time.
>
> Does that make sense? Do you agree that this makes the test of the
> dynamic serializers more or less invalid?
>
> Chad
Re: Report on thrift-protobuf-compare
Posted by Chad Walters <Ch...@microsoft.com>.
On 5/12/09 9:20 AM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>> 1. The performance figures that thrift-protobuf-compare provides
>> for dynamic
>> serialization systems like JSON are not currently valid since the
>> tests do
>> not really test them as a fully general serialization/deserialization
>> framework.
>
>You mean that since they coded their json serialization directly to
>their test data, the performance data is inaccurate? This might be
>true in the sense that their serialization doesn't do some things
>that Thrift does, but since we generate code, I don't think it's
>*that* different.
No, it's much worse than that, at least as I see it. The dynamic serializers are quite tightly bound to the data that is provided in the create() method, not just to the data structure. This means that they are "cheating" in ways that allow them to avoid putting In metadata they would need to do valid serialization/deserialization in a general way and they also avoid doing a bunch of switching that they would be required to do things correctly.
Look at StdMediaSerializer::create():
public final MediaContent create() throws Exception
{
Media media = new Media(null, "video/mpg4", Media.Player.JAVA, "Javaone Keynote", "htt
p://javaone.com/keynote.mpg", 1234567, 123, 0, 0, 0);
media.addToPerson("Bill Gates");
media.addToPerson("Steve Jobs");
Image image1 = new Image(0, "Javaone Keynote", "A", 0, Image.Size.LARGE);
Image image2 = new Image(0, "Javaone Keynote", "B", 0, Image.Size.SMALL);
MediaContent content = new MediaContent(media);
content.addImage(image1);
content.addImage(image2);
return content;
}
Now look at JsonSerializer::deserialize():
public MediaContent deserialize(byte[] array) throws Exception
{
JsonParser parser = _factory.createJsonParser(array);
parser.nextToken(); // start object
MediaContent mc = new MediaContent(readMedia(parser));
mc.addImage(readImage(parser));
mc.addImage(readImage(parser));
parser.nextToken(); // end object
parser.close();
return mc;
}
Note that deserialize() assumes that there are 2 images. Suppose create were changed to add 3 images instead. Then the roundtrip would, at best, discard one of the images.
Digging in a little deeper, these are supposed to be optional fields (at least they are marked so in media.thrift). However, the JsonSerializer's deserialize() assumes that the exact fields initialized in create() will all be present in the exact order they were serialized (look at findToken()) at deserialization time.
Does that make sense? Do you agree that this makes the test of the dynamic serializers more or less invalid?
Chad
Re: Report on thrift-protobuf-compare
Posted by Chad Walters <Ch...@microsoft.com>.
On 5/12/09 9:20 AM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>Are you saying that you think there is significant overhead to
>TSerializer? Or rather, how *is* the test going from objects to byte[]?
>
>It sounds like it might be interesting to strap a profiler to this
>test code and see if it shows us anything we can fix.
Compare ProtobufSerializer with ThriftSerializer.
In ProtobufSerializer:
public MediaContent deserialize (byte[] array) throws Exception
{
return MediaContent.parseFrom(array);
}
public byte[] serialize(MediaContent content) throws IOException
{
return content.toByteArray();
}
In ThriftSerializer (I left in original form rather than TCompactProtocol form).
public MediaContent deserialize(byte[] array) throws Exception
{
ByteArrayInputStream bais = new ByteArrayInputStream(array);
TIOStreamTransport trans = new TIOStreamTransport(bais);
TBinaryProtocol oprot = new TBinaryProtocol(trans);
MediaContent content = new MediaContent();
content.read(oprot);
return content;
}
public byte[] serialize(MediaContent content) throws Exception
{
ByteArrayOutputStream baos = new ByteArrayOutputStream(expectedSize);
TIOStreamTransport trans = new TIOStreamTransport(baos);
TBinaryProtocol oprot = new TBinaryProtocol(trans);
content.write(oprot);
byte[] array = baos.toByteArray();
expectedSize = array.length;
return array;
}
Clearly we are doing more work in both cases - creating new transports and protocols, copy byte[] objects, etc. This is basically due to the fact that ProtocolBuffers provides a lower-level API than Thrift that operates on byte[], which is what the test harness is based around.
I am not sure I consider this a real problem for Thrift. In a real world RPC or archiving scenario, you wouldn't need to create new transport or protocol objects on a per request basis. However, it does put us at a disadvantage for naïve bencharking. We could consider adding a lower-level API so that we could avoid doing this for these kind of low-level use cases.
I am not sure this is the whole story on the deserialization performance story though - some preliminary tests I have done to remove the overhead above still shows a similar gap in deserialization performance over protocol buffers. I am going to keep digging if I get time but I am pretty busy this week. If anyone else has time to look at this, particularly someone who is able to contribute code changes back to Thrift and/or thrift-protobuf-compare, would be much appreciated.
Chad
Re: Report on thrift-protobuf-compare
Posted by Bryan Duxbury <br...@rapleaf.com>.
This seems like a pretty thorough reply overall. I'll address some
issues inline.
> 1. The performance figures that thrift-protobuf-compare provides
> for dynamic
> serialization systems like JSON are not currently valid since the
> tests do
> not really test them as a fully general serialization/deserialization
> framework.
You mean that since they coded their json serialization directly to
their test data, the performance data is inaccurate? This might be
true in the sense that their serialization doesn't do some things
that Thrift does, but since we generate code, I don't think it's
*that* different.
> 2. Using TCompactProtocol, Thrift serialization speed and
> serialized size
> are basically equivalent to protocol buffers
Cool! I bet that we could find other datasets that exercise both
protos better and could turn up slight differences one way or another.
> I updated to the trunk of Thrift (rev 773454) and changed the Thrift
> serializer to use TCompactProtocol instead of TBinaryProtocol. I also
> corrected ThriftSerializer's create() so that the same data was
> being sent
> for image2 as in ProtobufSerializer. Finally, I updated the
> formatting in
> BenchmarkRunner and commented out all the serializers except Thrift
> and
> protocol buffers. Here are the benchmark results from 3 consecutive
> runs:
>
> , Create, Ser, Deser,
> Total, Size
> thrift , 267.37, 8314.00, 8546.00,
> 17127.36, 220
> protobuf , 412.98, 12642.00, 5217.50,
> 18272.48, 217
>
> , Create, Ser, Deser,
> Total, Size
> thrift , 266.87, 10905.50, 8526.50,
> 19698.86, 220
> protobuf , 415.21, 11880.50, 4930.00,
> 17225.71, 217
>
> , Create, Ser, Deser,
> Total, Size
> thrift , 264.95, 11059.50, 8701.50,
> 20025.95, 220
> protobuf , 417.45, 11125.00, 5203.50,
> 16745.95, 217
It's interesting that we're that much faster at object creation, but
it would seem from this test that it's basically only like 1% of the
total time of the test, so probably not that significant. We should
definitely address our deserialization time, though.
> Looking a little further at deserialization since Thrift seemed to be
> performing worse than protocol buffers there, the problem may be
> related to
> the fact that protocol buffers provides APIs that support direct
> serialization to and deserialization from byte arrays which Thrift
> does not
> provide. The test harness is set up such that the output of
> serialize() and
> the input of deserialize() is a byte array, so this means that
> Thrift needs
> to do more work to match up with the test harness. I am still
> investigating
> this.
Are you saying that you think there is significant overhead to
TSerializer? Or rather, how *is* the test going from objects to byte[]?
It sounds like it might be interesting to strap a profiler to this
test code and see if it shows us anything we can fix.