You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Bryan Duxbury <br...@rapleaf.com> on 2009/03/25 23:05:06 UTC

Addition of union type

Hey all,

I wanted to propose an idea I've been toying around with for a little  
while. At Rapleaf, we've developed a sort of pattern for creating  
Thrift objects that are all of the same type, but by convention only  
contain one of many fields. That is, every one of our objects has two  
required fields and ~80 optional fields, of which only one is ever  
set. This gives us union-like functionality, so our object could  
conceivably be of any one the many subtypes. One of the two required  
fields is an i32 that contains the Thrift field id of the field that  
should be set.

There are a lot of things that are good about this approach: it's  
simple, it's pretty sparse on the wire, and its very flexible.  
However, there are some things about it that aren't so great: nothing  
in Thrift validates the relationship between the field type specifier  
field and the data field or guarantees only one of the data fields is  
set. We've been able to work around these limitations for the most  
part, but it's been something we've had to deal with at the  
application level. Another limitation is that the cpu impact  
attempting to serialize 79 unset fields seems to be as much as 100%  
overhead to serialization performance.

What I would really like is for Thrift to support this behavior  
natively. I was thinking that we could add a "union" construct to  
Thrift, which would exist parallel to "struct". In some languages,  
this could potentially map to an actual union; in languages like Java  
and Ruby, we'd probably have to make some sort of TUnion class to  
mimic the behavior. I don't think we'd need a new wire type or  
anything. All of the behavior changes would be in the generated code,  
making sure that it didn't read two values for the same union field,  
etc.

What do people think about this idea? If we like it we can start to  
flesh it out some more and then open a ticket to get implementations  
going.

-Bryan

Re: Addition of union type

Posted by Doug Daniels <da...@gmail.com>.
I like the idea of efficient support for union's as it would allow me to use
Thrift as a traditional game network protocol where the server and client
just send "messages" (which would be any struct defined to be in your
Client2Server protocol and Server2ClientProtocol). The Client2Server union
would define messages client send to the server, and the Server2Client
defines what the server sends to client.

The Client2Server union is very similiar to RPC service definitions, the
biggest difference is the Server2Client protocol I'd want it to be like a
"reverse RPC" that the server would initiate sending messages to a connected
client transport (the server response message to the client would not have
to be directly related to any client sent request or RPC).

I worked with a closed-source message protocol IDL system in C++/Java
similiar to Thrift and the way we'd do this is define the structs like you
do in Thrift and additionally each struct is given an Message ID and each
language is given a lookup table so when they receive a msg they're given
it's ID and it's size. The input/output transport for client and server
would work the same, they'd read a message Type and size off the network
then read in the struct defined by that message ID.

I think the concept of "messages" is similiar to how Thrift implements RPC
services the trouble is I can't seem to find support to allow a serevr
respond to a connected client unless that client initiates an RPC call and
then it would only be able to respond with the particular defined response
type (which could be made to be "anything" if you use the Union message as a
response), but still that doesn't allow the server to initiate a message to
the client, the client would have to poll for messages, also the trouble
with Unions is it feels like you're repeating yourself (but I guess you do
need some way to indicate which messages are Client2Server and which are
Server2Client)



On Thu, Mar 26, 2009 at 6:57 AM, mark weisser
<ma...@diligentcns.com>wrote:

> maybe the solution is to create the C++ code and then the thrift code
>
> Bryan Duxbury wrote:
> > Hey all,
> >
> > I wanted to propose an idea I've been toying around with for a little
> > while. At Rapleaf, we've developed a sort of pattern for creating
> > Thrift objects that are all of the same type, but by convention only
> > contain one of many fields. That is, every one of our objects has two
> > required fields and ~80 optional fields, of which only one is ever
> > set. This gives us union-like functionality, so our object could
> > conceivably be of any one the many subtypes. One of the two required
> > fields is an i32 that contains the Thrift field id of the field that
> > should be set.
> >
> > There are a lot of things that are good about this approach: it's
> > simple, it's pretty sparse on the wire, and its very flexible.
> > However, there are some things about it that aren't so great: nothing
> > in Thrift validates the relationship between the field type specifier
> > field and the data field or guarantees only one of the data fields is
> > set. We've been able to work around these limitations for the most
> > part, but it's been something we've had to deal with at the
> > application level. Another limitation is that the cpu impact
> > attempting to serialize 79 unset fields seems to be as much as 100%
> > overhead to serialization performance.
> >
> > What I would really like is for Thrift to support this behavior
> > natively. I was thinking that we could add a "union" construct to
> > Thrift, which would exist parallel to "struct". In some languages,
> > this could potentially map to an actual union; in languages like Java
> > and Ruby, we'd probably have to make some sort of TUnion class to
> > mimic the behavior. I don't think we'd need a new wire type or
> > anything. All of the behavior changes would be in the generated code,
> > making sure that it didn't read two values for the same union field, etc.
> >
> > What do people think about this idea? If we like it we can start to
> > flesh it out some more and then open a ticket to get implementations
> > going.
> >
> > -Bryan
> > ------------------------------------------------------------------------
> >
> >
> > No virus found in this incoming message.
> > Checked by AVG - www.avg.com
> > Version: 8.0.238 / Virus Database: 270.11.28/2022 - Release Date:
> 03/25/09 07:16:00
> >
> >
>
>

Re: Addition of union type

Posted by mark weisser <ma...@diligentcns.com>.
maybe the solution is to create the C++ code and then the thrift code

Bryan Duxbury wrote:
> Hey all,
>
> I wanted to propose an idea I've been toying around with for a little
> while. At Rapleaf, we've developed a sort of pattern for creating
> Thrift objects that are all of the same type, but by convention only
> contain one of many fields. That is, every one of our objects has two
> required fields and ~80 optional fields, of which only one is ever
> set. This gives us union-like functionality, so our object could
> conceivably be of any one the many subtypes. One of the two required
> fields is an i32 that contains the Thrift field id of the field that
> should be set.
>
> There are a lot of things that are good about this approach: it's
> simple, it's pretty sparse on the wire, and its very flexible.
> However, there are some things about it that aren't so great: nothing
> in Thrift validates the relationship between the field type specifier
> field and the data field or guarantees only one of the data fields is
> set. We've been able to work around these limitations for the most
> part, but it's been something we've had to deal with at the
> application level. Another limitation is that the cpu impact
> attempting to serialize 79 unset fields seems to be as much as 100%
> overhead to serialization performance.
>
> What I would really like is for Thrift to support this behavior
> natively. I was thinking that we could add a "union" construct to
> Thrift, which would exist parallel to "struct". In some languages,
> this could potentially map to an actual union; in languages like Java
> and Ruby, we'd probably have to make some sort of TUnion class to
> mimic the behavior. I don't think we'd need a new wire type or
> anything. All of the behavior changes would be in the generated code,
> making sure that it didn't read two values for the same union field, etc.
>
> What do people think about this idea? If we like it we can start to
> flesh it out some more and then open a ticket to get implementations
> going.
>
> -Bryan
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com 
> Version: 8.0.238 / Virus Database: 270.11.28/2022 - Release Date: 03/25/09 07:16:00
>
>   


Re: Addition of union type

Posted by David Reiss <dr...@facebook.com>.
I agree.

Mark Slee wrote:
> Oh, but to the original question about unions, that sounds pretty handy, and 
> probably a pretty common use case. Basically just syntactic sugar for the 
> "all-optional-only-one-present" case. That's definitely the kind of code 
> that can and should be generated, IMO. Go for it.
> 
> ----- Original Message ----- 
> From: "Ted Dunning" <te...@gmail.com>
> To: <th...@incubator.apache.org>
> Sent: Wednesday, March 25, 2009 6:50 PM
> Subject: Re: Addition of union type
> 
> 
>> It seems pretty reasonable and not all that hard.  If you got all 
>> ambitious,
>> you could rewrite the entire compiler to allow forward references and we
>> would suddenly have recursive data structures!
>>
>> (or not)
>>
>> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>>
>>> What do people think about this idea? If we like it we can start to flesh
>>> it out some more and then open a ticket to get implementations going.
>>
>>
>>
>> -- 
>> Ted Dunning, CTO
>> DeepDyve
>>
> 

Re: Addition of union type

Posted by Mark Slee <mc...@facebook.com>.
Oh, but to the original question about unions, that sounds pretty handy, and 
probably a pretty common use case. Basically just syntactic sugar for the 
"all-optional-only-one-present" case. That's definitely the kind of code 
that can and should be generated, IMO. Go for it.

----- Original Message ----- 
From: "Ted Dunning" <te...@gmail.com>
To: <th...@incubator.apache.org>
Sent: Wednesday, March 25, 2009 6:50 PM
Subject: Re: Addition of union type


> It seems pretty reasonable and not all that hard.  If you got all 
> ambitious,
> you could rewrite the entire compiler to allow forward references and we
> would suddenly have recursive data structures!
>
> (or not)
>
> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>
>> What do people think about this idea? If we like it we can start to flesh
>> it out some more and then open a ticket to get implementations going.
>
>
>
>
> -- 
> Ted Dunning, CTO
> DeepDyve
> 


Re: Addition of union type

Posted by Ted Dunning <te...@gmail.com>.
Btw... I should have put little <joke> markups on my original suggestion.

The union suggestion is a good one and simple.  The result is likely to be
scary, just as you say.

On Wed, Mar 25, 2009 at 7:03 PM, Mark Slee <mc...@facebook.com> wrote:

> Whoa scary-town.




-- 
Ted Dunning, CTO
DeepDyve

Re: Addition of union type

Posted by Ted Dunning <te...@gmail.com>.
I knew I didn't want to think about C++.

Thanks for doing it for me.

Wow.

On Wed, Mar 25, 2009 at 7:03 PM, Mark Slee <mc...@facebook.com> wrote:

> The reason for not having forward references/recursive structures in the
> first place is largely due to the nightmare they induce in C++




-- 
Ted Dunning, CTO
DeepDyve

Re: Addition of union type

Posted by Mark Slee <mc...@facebook.com>.
Whoa scary-town. The reason for not having forward references/recursive 
structures in the first place is largely due to the nightmare they induce in 
C++. For the languages where all objects are actually pointers, it's 
straightforward enough to toss in some nulls all over the place, but to pull 
this off in C++ means we need to start making every object a pointer, which 
means implementing dynamic memory management in generated code, which means 
very-scary-contracts-with-application-code, etc. etc.

Here's the super-scary case:

class MyObject {
  MyObject *otherObject;
}

MyObject a = someCodeThatReadsOneOfThese(); // automatically allocates 
nested object
MyObject *b = a.otherObject; // fair enough, you want to use it too
a.otherObject = ohActuallyIWantThisOne; // fair enough, you want to modify 
the object

OH GOD, WHO OWNS THE DYNAMICALLY ALLOCATED MEMORY?

There is no one hard-and-fast rule. Sometimes, you want the memory 
dynamically owned. Other times, you want to yank it out and transfer 
ownership to your application code (without having to do a deep copy).

Basically, to pull this off we'd have to make every Thrift object a 
smart_ptr or reference-counted storage of some type, which goes against 
Thrift's design principle of simplicity and light-weightness for the common 
base cases.

Cheers,
mcslee

----- Original Message ----- 
From: "Ted Dunning" <te...@gmail.com>
To: <th...@incubator.apache.org>
Sent: Wednesday, March 25, 2009 6:50 PM
Subject: Re: Addition of union type


> It seems pretty reasonable and not all that hard.  If you got all 
> ambitious,
> you could rewrite the entire compiler to allow forward references and we
> would suddenly have recursive data structures!
>
> (or not)
>
> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>
>> What do people think about this idea? If we like it we can start to flesh
>> it out some more and then open a ticket to get implementations going.
>
>
>
>
> -- 
> Ted Dunning, CTO
> DeepDyve
> 


Re: Addition of union type

Posted by Ted Dunning <te...@gmail.com>.
It seems pretty reasonable and not all that hard.  If you got all ambitious,
you could rewrite the entire compiler to allow forward references and we
would suddenly have recursive data structures!

(or not)

On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:

> What do people think about this idea? If we like it we can start to flesh
> it out some more and then open a ticket to get implementations going.




-- 
Ted Dunning, CTO
DeepDyve