Row compatible generated coders for custom classes

[Steven van Rossum via dev <de...@beam.apache.org>]

Hi all,

I was benchmarking the fastjson2 serialization library a few weeks back for
a Java pipeline I was working on and was asked by a colleague to benchmark
binary JSON serialization against Rows for fun. We didn't do any extensive
analysis across different shapes and sizes, but the finding on this
workload was that serialization to binary JSON (tuple representation)
outperformed the SchemaCoder on throughput by ~11x on serialization and ~5x
on deserialization. Additionally, RowCoder outperformed SchemaCoder on
throughput by ~1.3x on serialization and ~1.7x on deserialization. Note
that all benchmarks measured in the millions of ops/sec for this quick
test, so this is already excellent performance obviously.

I'm sure there's stuff to learn from other serialization libraries, but I'd
table that for now. The low hanging fruit improvement would be to skip that
intermediate hop to/from Row and instead generate custom SchemaCoders to
serialize directly into or deserialize from the Row format.
I'd be happy to pick this up at some point in the new year, but would just
like to get some thoughts from this group.

Regards,

Steve

Re: Row compatible generated coders for custom classes

[Steven van Rossum via dev <de...@beam.apache.org>]

> Meaning BSON I presume? What do you mean by "tuple representation"?
> (One downside of JSON is that the field names are redundantly stored
> in each record, so even if you save on CPU it may hurt on the network
> due to the greater data sizes).

Yes, I meant BSON. Tuple or array representation formats the
serialized representation as an array of all field values so the field
names are not stored in the serialized result.

> Sounds like there's a lot of room for improvement! One downside of
> Rows is that they can't (IIRC) store (and encode/decode) unboxed
> representations of their primitive field types. This alone would be
> good to solve, but as mentioned you could probably also skip a Row
> intermediate altogether for encoding/decoding.

If Row were an interface then you could generate a POJO at runtime
from a Schema and have it implement that interface, but I'm not sure
if that improves anything when it comes to serialization since you'd
still use some function with a field index parameter to retrieve
values from the Row instance, but it could be that the deserialized
instance takes up less space in memory.

Mapping out the RowCoderGenerator result into a specialized Coder for
the POJO I was benchmarking resulted in an improvement of
serialization throughput of ~2.2x and an improvement of
deserialization throughput of ~1.6x.

Re: Row compatible generated coders for custom classes

[Robert Bradshaw via dev <de...@beam.apache.org>]

On Fri, Dec 1, 2023 at 9:13 AM Steven van Rossum via dev
<de...@beam.apache.org> wrote:
>
> Hi all,
>
> I was benchmarking the fastjson2 serialization library a few weeks back for a Java pipeline I was working on and was asked by a colleague to benchmark binary JSON serialization against Rows for fun. We didn't do any extensive analysis across different shapes and sizes, but the finding on this workload was that serialization to binary JSON (tuple representation)

Meaning BSON I presume? What do you mean by "tuple representation"?
(One downside of JSON is that the field names are redundantly stored
in each record, so even if you save on CPU it may hurt on the network
due to the greater data sizes).

> outperformed the SchemaCoder on throughput by ~11x on serialization and ~5x on deserialization. Additionally, RowCoder outperformed SchemaCoder on throughput by ~1.3x on serialization and ~1.7x on deserialization. Note that all benchmarks measured in the millions of ops/sec for this quick test, so this is already excellent performance obviously.

Sounds like there's a lot of room for improvement! One downside of
Rows is that they can't (IIRC) store (and encode/decode) unboxed
representations of their primitive field types. This alone would be
good to solve, but as mentioned you could probably also skip a Row
intermediate altogether for encoding/decoding.

> I'm sure there's stuff to learn from other serialization libraries, but I'd table that for now. The low hanging fruit improvement would be to skip that intermediate hop to/from Row and instead generate custom SchemaCoders to serialize directly into or deserialize from the Row format.
> I'd be happy to pick this up at some point in the new year, but would just like to get some thoughts from this group.

+1, this'd be a great addition. I think there was some investigation
into using bytebuddy to auto-generate this kind of thing, but I don't
know how extensive it is.

Re: Row compatible generated coders for custom classes

[Steven van Rossum via dev <de...@beam.apache.org>]

> Out of curiosity, did you add a warmup time before benchmarking? Schema and row coder does codegen, so the first usage is very slow, but subsequent usages should be much faster. I recommend running any test for a warmup period before starting to measure.

Yep, I poked at this using JMH with 3 warmup iterations and 5
iterations. The SchemaCoder is constructed at setup and encode is
invoked once to ensure code generation for the underlying RowCoder has
completed before the iteration begins.

One minor thing I noticed when I was looking at this yesterday is that
an optional field's Coder is wrapped by NullableCoder, but RowCoder
skips encoding of null values entirely and marks their absence in that
field presence BitSet which precedes all field data. It seems like
wrapping optional fields with NullableCoder is a tad redundant, since
either no bytes are written at all or the first byte is always set to
1 to mark the presence of data. The field encoding makes it impossible
to relax a field from required to optional even though Row could
support doing so with the field presence BitSet. Even though that's a
backwards incompatible schema change I can see how some users may
expect support for these changes if they interact with sources/sinks
which support required to optional relaxation like BigQuery.

Re: Row compatible generated coders for custom classes

[Reuven Lax via dev <de...@beam.apache.org>]

Out of curiosity, did you add a warmup time before benchmarking? Schema and
row coder does codegen, so the first usage is very slow, but subsequent
usages should be much faster. I recommend running any test for a warmup
period before starting to measure.

On Fri, Dec 1, 2023, 9:13 AM Steven van Rossum via dev <de...@beam.apache.org>
wrote:

> Hi all,
>
> I was benchmarking the fastjson2 serialization library a few weeks back
> for a Java pipeline I was working on and was asked by a colleague to
> benchmark binary JSON serialization against Rows for fun. We didn't do any
> extensive analysis across different shapes and sizes, but the finding on
> this workload was that serialization to binary JSON (tuple representation)
> outperformed the SchemaCoder on throughput by ~11x on serialization and ~5x
> on deserialization. Additionally, RowCoder outperformed SchemaCoder on
> throughput by ~1.3x on serialization and ~1.7x on deserialization. Note
> that all benchmarks measured in the millions of ops/sec for this quick
> test, so this is already excellent performance obviously.
>
> I'm sure there's stuff to learn from other serialization libraries, but
> I'd table that for now. The low hanging fruit improvement would be to skip
> that intermediate hop to/from Row and instead generate custom SchemaCoders
> to serialize directly into or deserialize from the Row format.
> I'd be happy to pick this up at some point in the new year, but would just
> like to get some thoughts from this group.
>
> Regards,
>
> Steve
>