Using thrift IDL as the schema for arbitrary JSON messages

[Peter Neumark <pe...@prezi.com>]

Hi all,

At Prezi, we currently use Thrift for a lot of the internal communication
between services. A lot of engineers would like to keep using HTTP + JSON
as means of communicating between services, but with some sort of schema.

After evaluating several alternatives, the best idea we had so far was to
use the thrift IDL as the schema, and beef up the existing
TSimpleJSONProtocol code to read arbitrary JSON messages.


What I'm most interested in is whether or not someone else is already using
the Thrift IDL as schema for JSON communication?

I have a very early stage python implementation of such a protocol here:
https://github.com/neumark/thriftmodel/blob/master/thriftmodel/TFlexibleJSONProtocol.py

We'll probably create an implementation for Java as well.
The most difficult issue with reading JSON is that a lot of existing JSON
protocols allow for several different types in a field. My solution was to
have an "unboxed union". To illustrate this with an example, if we accept
the following messages:

{"id": 1, "expires": 32324234234}
{"id": 1, "expires": "2014.10.07 10:34"}
{"id": 1, "expires": false}

Then our Thrift definition would be

// Note: this is an unboxed union
union expiry { // or struct in my implementation because python has no
union yet
  1: i64 unix_timestamp,
  2: string date_string,
  3: bool expiration_switch
}

struct msg {
  1: i64 id,
  2: expiry expires
}

The "expires" field must be read "unboxed" so to speak.
What's the best option for encoding the fact that the union is unboxed in
the thrift IDL?
Has anyone considered adding annotations to the IDL for such purposes?

Please note that I posted this message to user@ yesterday, but I'm
crossposting because this seems like a more appropriate forum. Roger Meier
pointed out that there is a patch for a JSON generator, which is very
useful, but it does not in itself allow one to use Thrift as a schema for
JSON messages, which is our goal.

Thanks,
Peter

-- 

*Peter Neumark*
DevOps guy @Prezi <http://prezi.com>