You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Peter Neumark <ne...@gmail.com> on 2014/10/08 00:12:40 UTC
Using thrift IDL as the schema for arbitrary JSON messages
Hi all,
At Prezi, we currently use Thrift for a lot of the internal communication
between services. A lot of engineers would like to keep using HTTP + JSON
as means of communicating between services, but with some sort of schema.
After evaluating several alternatives, the best idea we had so far was to
use the thrift IDL as the schema, and beef up the existing
TSimpleJSONProtocol code to read arbitrary JSON messages.
What I'm most interested in is whether or not someone else is already doing
this?
I have a very early stage python implementation of such a protocol here:
https://github.com/neumark/thriftmodel/blob/master/thriftmodel/TFlexibleJSONProtocol.py
We'll probably create an implementation for Java as well.
The most difficult issue with reading JSON is that a lot of existing JSON
protocols allow for several different types in a field. My solution was to
have an "unboxed union". To illustrate this with an example, if we accept
the following messages:
{"id": 1, "expires": 32324234234}
{"id": 1, "expires": "2014.10.07 10:34"}
{"id": 1, "expires": false}
Then our Thrift definition would be
// Note: this is an unboxed union
union { // or struct in my implementation because python has no union yet
1: i64 unix_timestamp,
2: string date_string,
3: bool expiration_switch
}
struct msg {
1: i64 id,
2: expiry expires
}
The "expires" field must be read "unboxed" so to speak.
What's the best option for encoding this in the thrift IDL?
Has anyone considered adding annotations to the IDL for similar purposes?
Thanks,
Peter
RE: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Craig Peterson <cp...@ancestry.com>.
I made that json generator in jira as a way to make the idl available to other platforms without having to write a full parser. My intention was really to provide an alternate view of the idl that is a bit easier to consume. I am not clear exactly how it would be relevant to using json as a wire protocol. It is not the same as the facebook thrift version which appears to be a bit simpler. My intention was really to dump the entire parsed data structure the compiler uses internally and make it available via json for other tools to use.
I have since found a different way to do what I was working on at the time and stopped trying to push on getting that patch merged. It would still be nice to have, since I can think of a number of tools I would like to make, but I don't really want to write a custom IDL parser for my platform of choice. I am not sure how well it will work with various IDL features like imported files or complex default values or things like that.
-----Original Message-----
From: Roger Meier [mailto:roger@bufferoverflow.ch]
Sent: 08 October 2014 4:49
To: Peter Neumark; dev@thrift.apache.org
Cc: user@thrift.apache.org
Subject: Re: Using thrift IDL as the schema for arbitrary JSON messages
Hi Peter
Quoting Peter Neumark <ne...@gmail.com>:
> Thanks, Roger!
>
> I'm also posting this to the dev list, because I think that may be
> more appropriate than user@.
Yes, that's why I'm cross posting now.
> The json generator is awesome (any estimate on when it will be
> merged?),
there are two versions and I like to have one...
I'm ready to commit it as soon as I get proper response on my questions on that issue. Json Schema is what i would like to have for the future.
> but that's not really what I was working on.
> The goal here is to ready an aribrary JSON message as thrift data.
Yes, I know. However using a schema would simplify this and that's why I started to talk about the json generator.
-roger
>
> Pete
>
> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
> wrote:
>
>> Hi Peter
>>
>> Yes, JSON is worth to add and I see an option by using json schema as
>> mentioned here:
>> https://issues.apache.org/jira/browse/THRIFT-2360
>>
>> best!
>> -roger
>>
>>
>> Quoting Peter Neumark <ne...@gmail.com>:
>>
>> Hi all,
>>>
>>> At Prezi, we currently use Thrift for a lot of the internal
>>> communication between services. A lot of engineers would like to
>>> keep using HTTP + JSON as means of communicating between services, but with some sort of schema.
>>>
>>> After evaluating several alternatives, the best idea we had so far
>>> was to use the thrift IDL as the schema, and beef up the existing
>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>
>>> What I'm most interested in is whether or not someone else is
>>> already doing this?
>>>
>>> I have a very early stage python implementation of such a protocol here:
>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>> TFlexibleJSONProtocol.py
>>>
>>> We'll probably create an implementation for Java as well.
>>> The most difficult issue with reading JSON is that a lot of existing
>>> JSON protocols allow for several different types in a field. My
>>> solution was to have an "unboxed union". To illustrate this with an
>>> example, if we accept the following messages:
>>>
>>> {"id": 1, "expires": 32324234234}
>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>> {"id": 1, "expires": false}
>>>
>>> Then our Thrift definition would be
>>>
>>> // Note: this is an unboxed union
>>> union { // or struct in my implementation because python has no union yet
>>> 1: i64 unix_timestamp,
>>> 2: string date_string,
>>> 3: bool expiration_switch
>>> }
>>>
>>> struct msg {
>>> 1: i64 id,
>>> 2: expiry expires
>>> }
>>>
>>> The "expires" field must be read "unboxed" so to speak.
>>> What's the best option for encoding this in the thrift IDL?
>>> Has anyone considered adding annotations to the IDL for similar purposes?
>>>
>>> Thanks,
>>> Peter
>>>
>>
>>
>>
Re: Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Peter Neumark <pe...@prezi.com>.
Done: https://github.com/neumark/thrift/compare/THRIFT-2360?expand=1
On Thu, Oct 9, 2014 at 2:03 PM, Roger Meier <ro...@bufferoverflow.ch> wrote:
>
> Quoting Peter Neumark <pe...@prezi.com>:
>
> On Thu, Oct 9, 2014 at 12:49 AM, Roger Meier <ro...@bufferoverflow.ch>
>> wrote:
>>
>> Hi Peter
>>>
>>> Quoting Peter Neumark <ne...@gmail.com>:
>>>
>>> Thanks, Roger!
>>>
>>>>
>>>> I'm also posting this to the dev list, because I think that may be more
>>>> appropriate than user@.
>>>>
>>>> Yes, that's why I'm cross posting now.
>>>
>>> The json generator is awesome (any estimate on when it will be merged?),
>>>
>>>>
>>>> there are two versions and I like to have one...
>>> I'm ready to commit it as soon as I get proper response on my questions
>>> on that issue. Json Schema is what i would like to have for the future.
>>>
>>>
>> What can I do to get this into the master branch?
>>
>
> test the patch from Peterson:
> https://issues.apache.org/jira/browse/THRIFT-2360
>
> fix warnings, add it to CMakeLists.txt and Makefile.am
> submit reworked patch.
>
> thanks
> -roger
>
>>
>>
>> but that's not really what I was working on.
>>>
>>>> The goal here is to ready an aribrary JSON message as thrift data.
>>>>
>>>> Yes, I know. However using a schema would simplify this and that's why
>>> I
>>> started to talk about the json generator.
>>> -roger
>>>
>>>
>>> Pete
>>>>
>>>> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
>>>> wrote:
>>>>
>>>> Hi Peter
>>>>
>>>>>
>>>>> Yes, JSON is worth to add and I see an option by using json schema as
>>>>> mentioned here:
>>>>> https://issues.apache.org/jira/browse/THRIFT-2360
>>>>>
>>>>> best!
>>>>> -roger
>>>>>
>>>>>
>>>>> Quoting Peter Neumark <ne...@gmail.com>:
>>>>>
>>>>> Hi all,
>>>>>
>>>>>
>>>>>> At Prezi, we currently use Thrift for a lot of the internal
>>>>>> communication
>>>>>> between services. A lot of engineers would like to keep using HTTP +
>>>>>> JSON
>>>>>> as means of communicating between services, but with some sort of
>>>>>> schema.
>>>>>>
>>>>>> After evaluating several alternatives, the best idea we had so far was
>>>>>> to
>>>>>> use the thrift IDL as the schema, and beef up the existing
>>>>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>>>>
>>>>>> What I'm most interested in is whether or not someone else is already
>>>>>> doing
>>>>>> this?
>>>>>>
>>>>>> I have a very early stage python implementation of such a protocol
>>>>>> here:
>>>>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>>>>> TFlexibleJSONProtocol.py
>>>>>>
>>>>>> We'll probably create an implementation for Java as well.
>>>>>> The most difficult issue with reading JSON is that a lot of existing
>>>>>> JSON
>>>>>> protocols allow for several different types in a field. My solution
>>>>>> was
>>>>>> to
>>>>>> have an "unboxed union". To illustrate this with an example, if we
>>>>>> accept
>>>>>> the following messages:
>>>>>>
>>>>>> {"id": 1, "expires": 32324234234}
>>>>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>>>>> {"id": 1, "expires": false}
>>>>>>
>>>>>> Then our Thrift definition would be
>>>>>>
>>>>>> // Note: this is an unboxed union
>>>>>> union { // or struct in my implementation because python has no union
>>>>>> yet
>>>>>> 1: i64 unix_timestamp,
>>>>>> 2: string date_string,
>>>>>> 3: bool expiration_switch
>>>>>> }
>>>>>>
>>>>>> struct msg {
>>>>>> 1: i64 id,
>>>>>> 2: expiry expires
>>>>>> }
>>>>>>
>>>>>> The "expires" field must be read "unboxed" so to speak.
>>>>>> What's the best option for encoding this in the thrift IDL?
>>>>>> Has anyone considered adding annotations to the IDL for similar
>>>>>> purposes?
>>>>>>
>>>>>> Thanks,
>>>>>> Peter
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>> --
>>
>> *Peter Neumark*
>> DevOps guy @Prezi <http://prezi.com>
>>
>
>
>
--
*Peter Neumark*
DevOps guy @Prezi <http://prezi.com>
Re: Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Quoting Peter Neumark <pe...@prezi.com>:
> On Thu, Oct 9, 2014 at 12:49 AM, Roger Meier <ro...@bufferoverflow.ch>
> wrote:
>
>> Hi Peter
>>
>> Quoting Peter Neumark <ne...@gmail.com>:
>>
>> Thanks, Roger!
>>>
>>> I'm also posting this to the dev list, because I think that may be more
>>> appropriate than user@.
>>>
>> Yes, that's why I'm cross posting now.
>>
>> The json generator is awesome (any estimate on when it will be merged?),
>>>
>> there are two versions and I like to have one...
>> I'm ready to commit it as soon as I get proper response on my questions
>> on that issue. Json Schema is what i would like to have for the future.
>>
>
> What can I do to get this into the master branch?
test the patch from Peterson:
https://issues.apache.org/jira/browse/THRIFT-2360
fix warnings, add it to CMakeLists.txt and Makefile.am
submit reworked patch.
thanks
-roger
>
>
>> but that's not really what I was working on.
>>> The goal here is to ready an aribrary JSON message as thrift data.
>>>
>> Yes, I know. However using a schema would simplify this and that's why I
>> started to talk about the json generator.
>> -roger
>>
>>
>>> Pete
>>>
>>> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
>>> wrote:
>>>
>>> Hi Peter
>>>>
>>>> Yes, JSON is worth to add and I see an option by using json schema as
>>>> mentioned here:
>>>> https://issues.apache.org/jira/browse/THRIFT-2360
>>>>
>>>> best!
>>>> -roger
>>>>
>>>>
>>>> Quoting Peter Neumark <ne...@gmail.com>:
>>>>
>>>> Hi all,
>>>>
>>>>>
>>>>> At Prezi, we currently use Thrift for a lot of the internal
>>>>> communication
>>>>> between services. A lot of engineers would like to keep using HTTP +
>>>>> JSON
>>>>> as means of communicating between services, but with some sort of
>>>>> schema.
>>>>>
>>>>> After evaluating several alternatives, the best idea we had so far was
>>>>> to
>>>>> use the thrift IDL as the schema, and beef up the existing
>>>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>>>
>>>>> What I'm most interested in is whether or not someone else is already
>>>>> doing
>>>>> this?
>>>>>
>>>>> I have a very early stage python implementation of such a protocol here:
>>>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>>>> TFlexibleJSONProtocol.py
>>>>>
>>>>> We'll probably create an implementation for Java as well.
>>>>> The most difficult issue with reading JSON is that a lot of existing
>>>>> JSON
>>>>> protocols allow for several different types in a field. My solution was
>>>>> to
>>>>> have an "unboxed union". To illustrate this with an example, if we
>>>>> accept
>>>>> the following messages:
>>>>>
>>>>> {"id": 1, "expires": 32324234234}
>>>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>>>> {"id": 1, "expires": false}
>>>>>
>>>>> Then our Thrift definition would be
>>>>>
>>>>> // Note: this is an unboxed union
>>>>> union { // or struct in my implementation because python has no union
>>>>> yet
>>>>> 1: i64 unix_timestamp,
>>>>> 2: string date_string,
>>>>> 3: bool expiration_switch
>>>>> }
>>>>>
>>>>> struct msg {
>>>>> 1: i64 id,
>>>>> 2: expiry expires
>>>>> }
>>>>>
>>>>> The "expires" field must be read "unboxed" so to speak.
>>>>> What's the best option for encoding this in the thrift IDL?
>>>>> Has anyone considered adding annotations to the IDL for similar
>>>>> purposes?
>>>>>
>>>>> Thanks,
>>>>> Peter
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>
>>
>
>
> --
>
> *Peter Neumark*
> DevOps guy @Prezi <http://prezi.com>
Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Peter Neumark <pe...@prezi.com>.
On Thu, Oct 9, 2014 at 12:49 AM, Roger Meier <ro...@bufferoverflow.ch>
wrote:
> Hi Peter
>
> Quoting Peter Neumark <ne...@gmail.com>:
>
> Thanks, Roger!
>>
>> I'm also posting this to the dev list, because I think that may be more
>> appropriate than user@.
>>
> Yes, that's why I'm cross posting now.
>
> The json generator is awesome (any estimate on when it will be merged?),
>>
> there are two versions and I like to have one...
> I'm ready to commit it as soon as I get proper response on my questions
> on that issue. Json Schema is what i would like to have for the future.
>
What can I do to get this into the master branch?
> but that's not really what I was working on.
>> The goal here is to ready an aribrary JSON message as thrift data.
>>
> Yes, I know. However using a schema would simplify this and that's why I
> started to talk about the json generator.
> -roger
>
>
>> Pete
>>
>> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
>> wrote:
>>
>> Hi Peter
>>>
>>> Yes, JSON is worth to add and I see an option by using json schema as
>>> mentioned here:
>>> https://issues.apache.org/jira/browse/THRIFT-2360
>>>
>>> best!
>>> -roger
>>>
>>>
>>> Quoting Peter Neumark <ne...@gmail.com>:
>>>
>>> Hi all,
>>>
>>>>
>>>> At Prezi, we currently use Thrift for a lot of the internal
>>>> communication
>>>> between services. A lot of engineers would like to keep using HTTP +
>>>> JSON
>>>> as means of communicating between services, but with some sort of
>>>> schema.
>>>>
>>>> After evaluating several alternatives, the best idea we had so far was
>>>> to
>>>> use the thrift IDL as the schema, and beef up the existing
>>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>>
>>>> What I'm most interested in is whether or not someone else is already
>>>> doing
>>>> this?
>>>>
>>>> I have a very early stage python implementation of such a protocol here:
>>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>>> TFlexibleJSONProtocol.py
>>>>
>>>> We'll probably create an implementation for Java as well.
>>>> The most difficult issue with reading JSON is that a lot of existing
>>>> JSON
>>>> protocols allow for several different types in a field. My solution was
>>>> to
>>>> have an "unboxed union". To illustrate this with an example, if we
>>>> accept
>>>> the following messages:
>>>>
>>>> {"id": 1, "expires": 32324234234}
>>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>>> {"id": 1, "expires": false}
>>>>
>>>> Then our Thrift definition would be
>>>>
>>>> // Note: this is an unboxed union
>>>> union { // or struct in my implementation because python has no union
>>>> yet
>>>> 1: i64 unix_timestamp,
>>>> 2: string date_string,
>>>> 3: bool expiration_switch
>>>> }
>>>>
>>>> struct msg {
>>>> 1: i64 id,
>>>> 2: expiry expires
>>>> }
>>>>
>>>> The "expires" field must be read "unboxed" so to speak.
>>>> What's the best option for encoding this in the thrift IDL?
>>>> Has anyone considered adding annotations to the IDL for similar
>>>> purposes?
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>>
>>>
>>>
>>>
>
>
--
*Peter Neumark*
DevOps guy @Prezi <http://prezi.com>
Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Hi Peter
Quoting Peter Neumark <ne...@gmail.com>:
> Thanks, Roger!
>
> I'm also posting this to the dev list, because I think that may be more
> appropriate than user@.
Yes, that's why I'm cross posting now.
> The json generator is awesome (any estimate on when it will be merged?),
there are two versions and I like to have one...
I'm ready to commit it as soon as I get proper response on my questions
on that issue. Json Schema is what i would like to have for the future.
> but that's not really what I was working on.
> The goal here is to ready an aribrary JSON message as thrift data.
Yes, I know. However using a schema would simplify this and that's why I
started to talk about the json generator.
-roger
>
> Pete
>
> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
> wrote:
>
>> Hi Peter
>>
>> Yes, JSON is worth to add and I see an option by using json schema as
>> mentioned here:
>> https://issues.apache.org/jira/browse/THRIFT-2360
>>
>> best!
>> -roger
>>
>>
>> Quoting Peter Neumark <ne...@gmail.com>:
>>
>> Hi all,
>>>
>>> At Prezi, we currently use Thrift for a lot of the internal communication
>>> between services. A lot of engineers would like to keep using HTTP + JSON
>>> as means of communicating between services, but with some sort of schema.
>>>
>>> After evaluating several alternatives, the best idea we had so far was to
>>> use the thrift IDL as the schema, and beef up the existing
>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>
>>> What I'm most interested in is whether or not someone else is already
>>> doing
>>> this?
>>>
>>> I have a very early stage python implementation of such a protocol here:
>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>> TFlexibleJSONProtocol.py
>>>
>>> We'll probably create an implementation for Java as well.
>>> The most difficult issue with reading JSON is that a lot of existing JSON
>>> protocols allow for several different types in a field. My solution was to
>>> have an "unboxed union". To illustrate this with an example, if we accept
>>> the following messages:
>>>
>>> {"id": 1, "expires": 32324234234}
>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>> {"id": 1, "expires": false}
>>>
>>> Then our Thrift definition would be
>>>
>>> // Note: this is an unboxed union
>>> union { // or struct in my implementation because python has no union yet
>>> 1: i64 unix_timestamp,
>>> 2: string date_string,
>>> 3: bool expiration_switch
>>> }
>>>
>>> struct msg {
>>> 1: i64 id,
>>> 2: expiry expires
>>> }
>>>
>>> The "expires" field must be read "unboxed" so to speak.
>>> What's the best option for encoding this in the thrift IDL?
>>> Has anyone considered adding annotations to the IDL for similar purposes?
>>>
>>> Thanks,
>>> Peter
>>>
>>
>>
>>
Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Hi Peter
Quoting Peter Neumark <ne...@gmail.com>:
> Thanks, Roger!
>
> I'm also posting this to the dev list, because I think that may be more
> appropriate than user@.
Yes, that's why I'm cross posting now.
> The json generator is awesome (any estimate on when it will be merged?),
there are two versions and I like to have one...
I'm ready to commit it as soon as I get proper response on my questions
on that issue. Json Schema is what i would like to have for the future.
> but that's not really what I was working on.
> The goal here is to ready an aribrary JSON message as thrift data.
Yes, I know. However using a schema would simplify this and that's why I
started to talk about the json generator.
-roger
>
> Pete
>
> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
> wrote:
>
>> Hi Peter
>>
>> Yes, JSON is worth to add and I see an option by using json schema as
>> mentioned here:
>> https://issues.apache.org/jira/browse/THRIFT-2360
>>
>> best!
>> -roger
>>
>>
>> Quoting Peter Neumark <ne...@gmail.com>:
>>
>> Hi all,
>>>
>>> At Prezi, we currently use Thrift for a lot of the internal communication
>>> between services. A lot of engineers would like to keep using HTTP + JSON
>>> as means of communicating between services, but with some sort of schema.
>>>
>>> After evaluating several alternatives, the best idea we had so far was to
>>> use the thrift IDL as the schema, and beef up the existing
>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>
>>> What I'm most interested in is whether or not someone else is already
>>> doing
>>> this?
>>>
>>> I have a very early stage python implementation of such a protocol here:
>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>> TFlexibleJSONProtocol.py
>>>
>>> We'll probably create an implementation for Java as well.
>>> The most difficult issue with reading JSON is that a lot of existing JSON
>>> protocols allow for several different types in a field. My solution was to
>>> have an "unboxed union". To illustrate this with an example, if we accept
>>> the following messages:
>>>
>>> {"id": 1, "expires": 32324234234}
>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>> {"id": 1, "expires": false}
>>>
>>> Then our Thrift definition would be
>>>
>>> // Note: this is an unboxed union
>>> union { // or struct in my implementation because python has no union yet
>>> 1: i64 unix_timestamp,
>>> 2: string date_string,
>>> 3: bool expiration_switch
>>> }
>>>
>>> struct msg {
>>> 1: i64 id,
>>> 2: expiry expires
>>> }
>>>
>>> The "expires" field must be read "unboxed" so to speak.
>>> What's the best option for encoding this in the thrift IDL?
>>> Has anyone considered adding annotations to the IDL for similar purposes?
>>>
>>> Thanks,
>>> Peter
>>>
>>
>>
>>
Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Peter Neumark <ne...@gmail.com>.
Thanks, Roger!
I'm also posting this to the dev list, because I think that may be more
appropriate than user@.
The json generator is awesome (any estimate on when it will be merged?),
but that's not really what I was working on.
The goal here is to ready an aribrary JSON message as thrift data.
Pete
On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <ro...@bufferoverflow.ch>
wrote:
> Hi Peter
>
> Yes, JSON is worth to add and I see an option by using json schema as
> mentioned here:
> https://issues.apache.org/jira/browse/THRIFT-2360
>
> best!
> -roger
>
>
> Quoting Peter Neumark <ne...@gmail.com>:
>
> Hi all,
>>
>> At Prezi, we currently use Thrift for a lot of the internal communication
>> between services. A lot of engineers would like to keep using HTTP + JSON
>> as means of communicating between services, but with some sort of schema.
>>
>> After evaluating several alternatives, the best idea we had so far was to
>> use the thrift IDL as the schema, and beef up the existing
>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>
>> What I'm most interested in is whether or not someone else is already
>> doing
>> this?
>>
>> I have a very early stage python implementation of such a protocol here:
>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>> TFlexibleJSONProtocol.py
>>
>> We'll probably create an implementation for Java as well.
>> The most difficult issue with reading JSON is that a lot of existing JSON
>> protocols allow for several different types in a field. My solution was to
>> have an "unboxed union". To illustrate this with an example, if we accept
>> the following messages:
>>
>> {"id": 1, "expires": 32324234234}
>> {"id": 1, "expires": "2014.10.07 10:34"}
>> {"id": 1, "expires": false}
>>
>> Then our Thrift definition would be
>>
>> // Note: this is an unboxed union
>> union { // or struct in my implementation because python has no union yet
>> 1: i64 unix_timestamp,
>> 2: string date_string,
>> 3: bool expiration_switch
>> }
>>
>> struct msg {
>> 1: i64 id,
>> 2: expiry expires
>> }
>>
>> The "expires" field must be read "unboxed" so to speak.
>> What's the best option for encoding this in the thrift IDL?
>> Has anyone considered adding annotations to the IDL for similar purposes?
>>
>> Thanks,
>> Peter
>>
>
>
>
Re: Using thrift IDL as the schema for arbitrary JSON messages
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Hi Peter
Yes, JSON is worth to add and I see an option by using json schema as
mentioned here:
https://issues.apache.org/jira/browse/THRIFT-2360
best!
-roger
Quoting Peter Neumark <ne...@gmail.com>:
> Hi all,
>
> At Prezi, we currently use Thrift for a lot of the internal communication
> between services. A lot of engineers would like to keep using HTTP + JSON
> as means of communicating between services, but with some sort of schema.
>
> After evaluating several alternatives, the best idea we had so far was to
> use the thrift IDL as the schema, and beef up the existing
> TSimpleJSONProtocol code to read arbitrary JSON messages.
>
> What I'm most interested in is whether or not someone else is already doing
> this?
>
> I have a very early stage python implementation of such a protocol here:
> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/TFlexibleJSONProtocol.py
>
> We'll probably create an implementation for Java as well.
> The most difficult issue with reading JSON is that a lot of existing JSON
> protocols allow for several different types in a field. My solution was to
> have an "unboxed union". To illustrate this with an example, if we accept
> the following messages:
>
> {"id": 1, "expires": 32324234234}
> {"id": 1, "expires": "2014.10.07 10:34"}
> {"id": 1, "expires": false}
>
> Then our Thrift definition would be
>
> // Note: this is an unboxed union
> union { // or struct in my implementation because python has no union yet
> 1: i64 unix_timestamp,
> 2: string date_string,
> 3: bool expiration_switch
> }
>
> struct msg {
> 1: i64 id,
> 2: expiry expires
> }
>
> The "expires" field must be read "unboxed" so to speak.
> What's the best option for encoding this in the thrift IDL?
> Has anyone considered adding annotations to the IDL for similar purposes?
>
> Thanks,
> Peter