You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Bryan Duxbury <br...@rapleaf.com> on 2009/03/25 23:05:06 UTC
Addition of union type
Hey all,
I wanted to propose an idea I've been toying around with for a little
while. At Rapleaf, we've developed a sort of pattern for creating
Thrift objects that are all of the same type, but by convention only
contain one of many fields. That is, every one of our objects has two
required fields and ~80 optional fields, of which only one is ever
set. This gives us union-like functionality, so our object could
conceivably be of any one the many subtypes. One of the two required
fields is an i32 that contains the Thrift field id of the field that
should be set.
There are a lot of things that are good about this approach: it's
simple, it's pretty sparse on the wire, and its very flexible.
However, there are some things about it that aren't so great: nothing
in Thrift validates the relationship between the field type specifier
field and the data field or guarantees only one of the data fields is
set. We've been able to work around these limitations for the most
part, but it's been something we've had to deal with at the
application level. Another limitation is that the cpu impact
attempting to serialize 79 unset fields seems to be as much as 100%
overhead to serialization performance.
What I would really like is for Thrift to support this behavior
natively. I was thinking that we could add a "union" construct to
Thrift, which would exist parallel to "struct". In some languages,
this could potentially map to an actual union; in languages like Java
and Ruby, we'd probably have to make some sort of TUnion class to
mimic the behavior. I don't think we'd need a new wire type or
anything. All of the behavior changes would be in the generated code,
making sure that it didn't read two values for the same union field,
etc.
What do people think about this idea? If we like it we can start to
flesh it out some more and then open a ticket to get implementations
going.
-Bryan
Re: Addition of union type
Posted by Doug Daniels <da...@gmail.com>.
I like the idea of efficient support for union's as it would allow me to use
Thrift as a traditional game network protocol where the server and client
just send "messages" (which would be any struct defined to be in your
Client2Server protocol and Server2ClientProtocol). The Client2Server union
would define messages client send to the server, and the Server2Client
defines what the server sends to client.
The Client2Server union is very similiar to RPC service definitions, the
biggest difference is the Server2Client protocol I'd want it to be like a
"reverse RPC" that the server would initiate sending messages to a connected
client transport (the server response message to the client would not have
to be directly related to any client sent request or RPC).
I worked with a closed-source message protocol IDL system in C++/Java
similiar to Thrift and the way we'd do this is define the structs like you
do in Thrift and additionally each struct is given an Message ID and each
language is given a lookup table so when they receive a msg they're given
it's ID and it's size. The input/output transport for client and server
would work the same, they'd read a message Type and size off the network
then read in the struct defined by that message ID.
I think the concept of "messages" is similiar to how Thrift implements RPC
services the trouble is I can't seem to find support to allow a serevr
respond to a connected client unless that client initiates an RPC call and
then it would only be able to respond with the particular defined response
type (which could be made to be "anything" if you use the Union message as a
response), but still that doesn't allow the server to initiate a message to
the client, the client would have to poll for messages, also the trouble
with Unions is it feels like you're repeating yourself (but I guess you do
need some way to indicate which messages are Client2Server and which are
Server2Client)
On Thu, Mar 26, 2009 at 6:57 AM, mark weisser
<ma...@diligentcns.com>wrote:
> maybe the solution is to create the C++ code and then the thrift code
>
> Bryan Duxbury wrote:
> > Hey all,
> >
> > I wanted to propose an idea I've been toying around with for a little
> > while. At Rapleaf, we've developed a sort of pattern for creating
> > Thrift objects that are all of the same type, but by convention only
> > contain one of many fields. That is, every one of our objects has two
> > required fields and ~80 optional fields, of which only one is ever
> > set. This gives us union-like functionality, so our object could
> > conceivably be of any one the many subtypes. One of the two required
> > fields is an i32 that contains the Thrift field id of the field that
> > should be set.
> >
> > There are a lot of things that are good about this approach: it's
> > simple, it's pretty sparse on the wire, and its very flexible.
> > However, there are some things about it that aren't so great: nothing
> > in Thrift validates the relationship between the field type specifier
> > field and the data field or guarantees only one of the data fields is
> > set. We've been able to work around these limitations for the most
> > part, but it's been something we've had to deal with at the
> > application level. Another limitation is that the cpu impact
> > attempting to serialize 79 unset fields seems to be as much as 100%
> > overhead to serialization performance.
> >
> > What I would really like is for Thrift to support this behavior
> > natively. I was thinking that we could add a "union" construct to
> > Thrift, which would exist parallel to "struct". In some languages,
> > this could potentially map to an actual union; in languages like Java
> > and Ruby, we'd probably have to make some sort of TUnion class to
> > mimic the behavior. I don't think we'd need a new wire type or
> > anything. All of the behavior changes would be in the generated code,
> > making sure that it didn't read two values for the same union field, etc.
> >
> > What do people think about this idea? If we like it we can start to
> > flesh it out some more and then open a ticket to get implementations
> > going.
> >
> > -Bryan
> > ------------------------------------------------------------------------
> >
> >
> > No virus found in this incoming message.
> > Checked by AVG - www.avg.com
> > Version: 8.0.238 / Virus Database: 270.11.28/2022 - Release Date:
> 03/25/09 07:16:00
> >
> >
>
>
Re: Addition of union type
Posted by mark weisser <ma...@diligentcns.com>.
maybe the solution is to create the C++ code and then the thrift code
Bryan Duxbury wrote:
> Hey all,
>
> I wanted to propose an idea I've been toying around with for a little
> while. At Rapleaf, we've developed a sort of pattern for creating
> Thrift objects that are all of the same type, but by convention only
> contain one of many fields. That is, every one of our objects has two
> required fields and ~80 optional fields, of which only one is ever
> set. This gives us union-like functionality, so our object could
> conceivably be of any one the many subtypes. One of the two required
> fields is an i32 that contains the Thrift field id of the field that
> should be set.
>
> There are a lot of things that are good about this approach: it's
> simple, it's pretty sparse on the wire, and its very flexible.
> However, there are some things about it that aren't so great: nothing
> in Thrift validates the relationship between the field type specifier
> field and the data field or guarantees only one of the data fields is
> set. We've been able to work around these limitations for the most
> part, but it's been something we've had to deal with at the
> application level. Another limitation is that the cpu impact
> attempting to serialize 79 unset fields seems to be as much as 100%
> overhead to serialization performance.
>
> What I would really like is for Thrift to support this behavior
> natively. I was thinking that we could add a "union" construct to
> Thrift, which would exist parallel to "struct". In some languages,
> this could potentially map to an actual union; in languages like Java
> and Ruby, we'd probably have to make some sort of TUnion class to
> mimic the behavior. I don't think we'd need a new wire type or
> anything. All of the behavior changes would be in the generated code,
> making sure that it didn't read two values for the same union field, etc.
>
> What do people think about this idea? If we like it we can start to
> flesh it out some more and then open a ticket to get implementations
> going.
>
> -Bryan
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.0.238 / Virus Database: 270.11.28/2022 - Release Date: 03/25/09 07:16:00
>
>
Re: Addition of union type
Posted by David Reiss <dr...@facebook.com>.
I agree.
Mark Slee wrote:
> Oh, but to the original question about unions, that sounds pretty handy, and
> probably a pretty common use case. Basically just syntactic sugar for the
> "all-optional-only-one-present" case. That's definitely the kind of code
> that can and should be generated, IMO. Go for it.
>
> ----- Original Message -----
> From: "Ted Dunning" <te...@gmail.com>
> To: <th...@incubator.apache.org>
> Sent: Wednesday, March 25, 2009 6:50 PM
> Subject: Re: Addition of union type
>
>
>> It seems pretty reasonable and not all that hard. If you got all
>> ambitious,
>> you could rewrite the entire compiler to allow forward references and we
>> would suddenly have recursive data structures!
>>
>> (or not)
>>
>> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>>
>>> What do people think about this idea? If we like it we can start to flesh
>>> it out some more and then open a ticket to get implementations going.
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>
Re: Addition of union type
Posted by Mark Slee <mc...@facebook.com>.
Oh, but to the original question about unions, that sounds pretty handy, and
probably a pretty common use case. Basically just syntactic sugar for the
"all-optional-only-one-present" case. That's definitely the kind of code
that can and should be generated, IMO. Go for it.
----- Original Message -----
From: "Ted Dunning" <te...@gmail.com>
To: <th...@incubator.apache.org>
Sent: Wednesday, March 25, 2009 6:50 PM
Subject: Re: Addition of union type
> It seems pretty reasonable and not all that hard. If you got all
> ambitious,
> you could rewrite the entire compiler to allow forward references and we
> would suddenly have recursive data structures!
>
> (or not)
>
> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>
>> What do people think about this idea? If we like it we can start to flesh
>> it out some more and then open a ticket to get implementations going.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: Addition of union type
Posted by Ted Dunning <te...@gmail.com>.
Btw... I should have put little <joke> markups on my original suggestion.
The union suggestion is a good one and simple. The result is likely to be
scary, just as you say.
On Wed, Mar 25, 2009 at 7:03 PM, Mark Slee <mc...@facebook.com> wrote:
> Whoa scary-town.
--
Ted Dunning, CTO
DeepDyve
Re: Addition of union type
Posted by Ted Dunning <te...@gmail.com>.
I knew I didn't want to think about C++.
Thanks for doing it for me.
Wow.
On Wed, Mar 25, 2009 at 7:03 PM, Mark Slee <mc...@facebook.com> wrote:
> The reason for not having forward references/recursive structures in the
> first place is largely due to the nightmare they induce in C++
--
Ted Dunning, CTO
DeepDyve
Re: Addition of union type
Posted by Mark Slee <mc...@facebook.com>.
Whoa scary-town. The reason for not having forward references/recursive
structures in the first place is largely due to the nightmare they induce in
C++. For the languages where all objects are actually pointers, it's
straightforward enough to toss in some nulls all over the place, but to pull
this off in C++ means we need to start making every object a pointer, which
means implementing dynamic memory management in generated code, which means
very-scary-contracts-with-application-code, etc. etc.
Here's the super-scary case:
class MyObject {
MyObject *otherObject;
}
MyObject a = someCodeThatReadsOneOfThese(); // automatically allocates
nested object
MyObject *b = a.otherObject; // fair enough, you want to use it too
a.otherObject = ohActuallyIWantThisOne; // fair enough, you want to modify
the object
OH GOD, WHO OWNS THE DYNAMICALLY ALLOCATED MEMORY?
There is no one hard-and-fast rule. Sometimes, you want the memory
dynamically owned. Other times, you want to yank it out and transfer
ownership to your application code (without having to do a deep copy).
Basically, to pull this off we'd have to make every Thrift object a
smart_ptr or reference-counted storage of some type, which goes against
Thrift's design principle of simplicity and light-weightness for the common
base cases.
Cheers,
mcslee
----- Original Message -----
From: "Ted Dunning" <te...@gmail.com>
To: <th...@incubator.apache.org>
Sent: Wednesday, March 25, 2009 6:50 PM
Subject: Re: Addition of union type
> It seems pretty reasonable and not all that hard. If you got all
> ambitious,
> you could rewrite the entire compiler to allow forward references and we
> would suddenly have recursive data structures!
>
> (or not)
>
> On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>
>> What do people think about this idea? If we like it we can start to flesh
>> it out some more and then open a ticket to get implementations going.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: Addition of union type
Posted by Ted Dunning <te...@gmail.com>.
It seems pretty reasonable and not all that hard. If you got all ambitious,
you could rewrite the entire compiler to allow forward references and we
would suddenly have recursive data structures!
(or not)
On Wed, Mar 25, 2009 at 3:05 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
> What do people think about this idea? If we like it we can start to flesh
> it out some more and then open a ticket to get implementations going.
--
Ted Dunning, CTO
DeepDyve