[jira] [Commented] (SAMZA-317) Serde for Avro-encoded messages

["Roger Hoover (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118395#comment-14118395 ] 

Roger Hoover commented on SAMZA-317:
------------------------------------

I would love to use Avro as well but it seems like the burden of maintaining a repository of all schemas since the beginning of time is painful.  However, I don't know all the consequences of choosing Thrift or PB once you get to Hadoop land.

Here's a crazy idea: what if the Avro project were to define a binary encoding for the schema that could be used in an RPC or streaming context?

As an example, I have an event schema with 21 fields.

Avro schema as JSON - 1126 bytes
Avro schema as JSON (gzipped) - 307 bytes
Estimated Avro schema as binary - ~200 bytes
JSON message - 426 bytes
Avro binary (no compression) - 195 bytes

I'm guessing that the bulk of the binary schema will be the field names.  The types should be very encodable very efficiently.  If so, then sending schema with message in-band is not too bad of an option.  It will roughly double the message size in this case but is comparable with a schema-free JSON version.

> Serde for Avro-encoded messages
> -------------------------------
>
>                 Key: SAMZA-317
>                 URL: https://issues.apache.org/jira/browse/SAMZA-317
>             Project: Samza
>          Issue Type: New Feature
>            Reporter: Martin Kleppmann
>
> [Avro|http://avro.apache.org/] is a popular serialization format with several nice characteristics:
> - Can be read and written by many programming languages
> - Compact and fast to serialize/deserialize
> - Supports Java code generation (in a statically typed language, JSON is annoying to work with, because it parses into dynamically typed HashMaps; it's much nicer to work with objects that have real getters and setters)
> - Deep support for [schema evolution|http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html], so you can make backwards-compatible changes to your data as application requirements change
> It would be nice if Samza came with an Avro serde out of the box, making it easy for Samza jobs to consume and produce Avro-encoded messages. If this serde is built on best practices, we can recommend it as a good default choice for many applications.
> This serde is not entirely straightforward to implement because of Avro's schema handling. In order to accurately deserialize an Avro-encoded message, you need to know the exact version of the schema with which it was serialized. Thus, every message needs to be tagged with a schema version number, and we require a schema registry which translates those version numbers into the schema definition (a JSON string).
> At LinkedIn, the MD5 hash of the schema is used as version number, and the mapping from version number to schema is stored in a separate schema registry service (which provides a HTTP API). In Samza, we could avoid the operational complexity of a separate schema registry service, and instead use a stream (e.g. a Kafka topic) for storing the schemas for all Samza jobs. It can similarly use the hash of the schema as the key, and the schema JSON as the value.
> Any job that wants to consume an Avro-encoded topic would then need to first fully consume the schemas stream (as a [bootstrap stream|http://samza.incubator.apache.org/learn/documentation/0.7.0/container/streams.html#bootstrapping]) in order to learn the complete mapping of version numbers to schemas. It then has all the information it needs to deserialize any messages.
> Any job that wants to produce Avro-encoded messages first needs to publish the schema version it is using to the schemas stream. Kafka log compaction can take care of the fact that the same schema will be submitted repeatedly.
> There is a race condition: when a new schema version is introduced, any consumers need to first receive the new schema version from the schemas stream before they can decode any messages encoded with this new schema. This could be implemented with a custom MessageChooser, which blocks consumption of any stream that contains a message with an unknown schema version number, until the corresponding schema is received. The blocking could have a timeout, to limit the disruption in case a badly-behaved producer sends messages without publishing their schema. (Such messages cannot be deserialized and would have to be dropped.)
> Related stuff:
> - Some discussion appears in [this mailing list thread|http://mail-archives.apache.org/mod_mbox/incubator-samza-dev/201402.mbox/%3C1BBF24B26F58BA4A91B566B7FBEEFF27551F6C7F7D%40EXMBXC01.ms-hosting.nl%3E].
> - We should consider compatibility with [Camus|https://github.com/linkedin/camus], which can take Avro-encoded messages from Kafka topics and load them into HDFS for offline processing.
> - SAMZA-198 discusses an issue that arose in the context of dealing with Avro-serialized messages.
> - AVRO-1124 discusses implementing a standard schema registry service for Avro.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)