You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Bryan Duxbury <br...@rapleaf.com> on 2008/08/21 22:56:26 UTC
Adding a "vint" type to Thrift
All,
We've been following the discussion on THRIFT-110. I think as a first
step we should add a "vint" (variable integer) to Thrift.
My general proposal is that we add vint as a new type. Protocols that
don't do variable length integers can treat vints as i64s. Protocols
that do implement variable length integers can choose encodings that
make sense for them.
We'd have to change the compiler and all the generators and libraries
to recognize vint, if for no other reason than to treat it as an i64.
However, this would make implementing other protocols that included
variable length integers very easy.
Are there any objections to this proposal? If not, I'll make an
issue, and Rapleaf is willing to commit time to this enhancement.
-Bryan
Re: Adding a "vint" type to Thrift
Posted by David Reiss <dr...@facebook.com>.
That's fine, but if we do that, I'd prefer to see it implemented
with a flexible syntax and then check validity after parsing.
--David
Mark Slee wrote:
> I'd propose starting with a fixed set of type hints that we agree upon.
> We could certainly implement a dynamic any-string goes type of thing.
> But here's my concern with that:
>
> 1/ If we agree upon a fixed set of modifiers, we're more likely to get
> them implemented across more languages since deciding on the set will
> involve getting some basic agreement amongst the Thrift community. We're
> likely to retain more overall compatibility.
>
> 2/ If the modifiers are dynamic, it implies that people would make their
> own local changes, and write their own custom Protocol implementations.
> We're more likely to duplicate this work or disagree on things. This
> will make custom protocol submissions back into the trunk more
> difficult, and the core project becomes more likely to lose quality
> work.
>
> So, while a dynamic implementation is arguably more elegant/flexible, I
> think having the community agree upon a reasonable set of type
> annotations is most healthy for the coherence of Thrift in the long run.
>
> Cheers,
> Mark
>
> -----Original Message-----
> From: Bryan Duxbury [mailto:bryan@rapleaf.com]
> Sent: Thursday, August 21, 2008 3:40 PM
> To: thrift-dev@incubator.apache.org
> Subject: Re: Adding a "vint" type to Thrift
>
> I'm not opposed to using a type modifier approach instead of a separate
> type. Will we really be adding "arbitrary" annotations to the objects,
> or will they be of a specific nature? That is, will we have modifiers
> like "fixed", "variable", "extern", etc. be compiler tokens, or will you
> just be allowed to add any string you want as a modifier before the
> type? The latter would certainly be more flexible, but would be more
> complicated as well.
>
> -Bryan
>
> On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
>
>> First, I don't understand why it would be limited to 64-bit. What if
>> I have an integer that I want represented as 32 bits in memory but
>> serialized with a variable-length encoding.
>>
>> Second, I wonder if creating a new type is the best solution. I've
>> been thinking for a while that it might be useful to add arbitrary
>> annotations to types. We already have a few special cases of this
>> (cpp_type in C++, slist in Erlang), and vint would be a third. I
>> think it might be time to bite the bullet and implement annotations so
>
>> we can stop adding new syntactic constructs for each minor extension
>> to the data model.
>>
>> --David
>>
>> Bryan Duxbury wrote:
>>> All,
>>>
>>> We've been following the discussion on THRIFT-110. I think as a first
>
>>> step we should add a "vint" (variable integer) to Thrift.
>>>
>>> My general proposal is that we add vint as a new type. Protocols that
>
>>> don't do variable length integers can treat vints as i64s. Protocols
>>> that do implement variable length integers can choose encodings that
>>> make sense for them.
>>>
>>> We'd have to change the compiler and all the generators and libraries
>
>>> to recognize vint, if for no other reason than to treat it as an i64.
>>> However, this would make implementing other protocols that included
>>> variable length integers very easy.
>>>
>>> Are there any objections to this proposal? If not, I'll make an
>>> issue, and Rapleaf is willing to commit time to this enhancement.
>>>
>>> -Bryan
>>>
>
Re: Adding a "vint" type to Thrift
Posted by Chad Walters <ch...@powerset.com>.
Works for me.
Chad
On 8/22/08 3:39 AM, "Noble Paul നോബിള് नोब्ळ्" <no...@gmail.com> wrote:
Do we really need an extra byte to encode boolean?
We can have a type modifier of '1' for false and 2 for 'true'
On Fri, Aug 22, 2008 at 6:05 AM, Bryan Duxbury <br...@rapleaf.com> wrote:
> Yeah, sounds better than way :)
>
> -Bryan
>
> On Aug 21, 2008, at 5:28 PM, Chad Walters wrote:
>
>>
>> Would you be happier if, instead of saying a number of bits should be
>> reserved, I said that there should be a maximum of four different possible
>> modification values, with one of them being a default (such that it could be
>> represented in a maximum of 2 bits if a protocol so desired)?
>>
>> The JSON protocol would clearly ignore the hints.
>>
>> Chad
>>
>>
>> On 8/21/08 5:23 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>>
>> Clearly the protocols and the modification types need to be in sync,
>> but saying that a certain number of bits should be reserved is too
>> specific. How would that map to the JSON protocol, for instance?
>>
>> It seems like we've arrived at some consensus. Should we create a new
>> issue for making type modifiers and get to work?
>>
>> -Bryan
>
>
--
--Noble Paul
Re: Adding a "vint" type to Thrift
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
Do we really need an extra byte to encode boolean?
We can have a type modifier of '1' for false and 2 for 'true'
On Fri, Aug 22, 2008 at 6:05 AM, Bryan Duxbury <br...@rapleaf.com> wrote:
> Yeah, sounds better than way :)
>
> -Bryan
>
> On Aug 21, 2008, at 5:28 PM, Chad Walters wrote:
>
>>
>> Would you be happier if, instead of saying a number of bits should be
>> reserved, I said that there should be a maximum of four different possible
>> modification values, with one of them being a default (such that it could be
>> represented in a maximum of 2 bits if a protocol so desired)?
>>
>> The JSON protocol would clearly ignore the hints.
>>
>> Chad
>>
>>
>> On 8/21/08 5:23 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>>
>> Clearly the protocols and the modification types need to be in sync,
>> but saying that a certain number of bits should be reserved is too
>> specific. How would that map to the JSON protocol, for instance?
>>
>> It seems like we've arrived at some consensus. Should we create a new
>> issue for making type modifiers and get to work?
>>
>> -Bryan
>
>
--
--Noble Paul
Re: Adding a "vint" type to Thrift
Posted by Bryan Duxbury <br...@rapleaf.com>.
Yeah, sounds better than way :)
-Bryan
On Aug 21, 2008, at 5:28 PM, Chad Walters wrote:
>
> Would you be happier if, instead of saying a number of bits should
> be reserved, I said that there should be a maximum of four
> different possible modification values, with one of them being a
> default (such that it could be represented in a maximum of 2 bits
> if a protocol so desired)?
>
> The JSON protocol would clearly ignore the hints.
>
> Chad
>
>
> On 8/21/08 5:23 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>
> Clearly the protocols and the modification types need to be in sync,
> but saying that a certain number of bits should be reserved is too
> specific. How would that map to the JSON protocol, for instance?
>
> It seems like we've arrived at some consensus. Should we create a new
> issue for making type modifiers and get to work?
>
> -Bryan
Re: Adding a "vint" type to Thrift
Posted by Chad Walters <ch...@powerset.com>.
Would you be happier if, instead of saying a number of bits should be reserved, I said that there should be a maximum of four different possible modification values, with one of them being a default (such that it could be represented in a maximum of 2 bits if a protocol so desired)?
The JSON protocol would clearly ignore the hints.
Chad
On 8/21/08 5:23 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
Clearly the protocols and the modification types need to be in sync,
but saying that a certain number of bits should be reserved is too
specific. How would that map to the JSON protocol, for instance?
It seems like we've arrived at some consensus. Should we create a new
issue for making type modifiers and get to work?
-Bryan
Re: Adding a "vint" type to Thrift
Posted by Bryan Duxbury <br...@rapleaf.com>.
Clearly the protocols and the modification types need to be in sync,
but saying that a certain number of bits should be reserved is too
specific. How would that map to the JSON protocol, for instance?
It seems like we've arrived at some consensus. Should we create a new
issue for making type modifiers and get to work?
-Bryan
On Aug 21, 2008, at 5:17 PM, Chad Walters wrote:
>
>> I'm fine with suggesting we keep the list short, but isn't saying we
>> keep 2 bits free an implementation concern for protocols?
>
> It is and it isn't. I think its reasonable for the framework to
> concern itself with how the protocols might implement the
> requirements that the framework is imposing. If we have more than
> 256 possible modified types, then it becomes impossible for any
> protocol to represent a type and its modifiers in a single byte.
>
>> I presume this would mean that the default modifier should be
>> interpreted as "variable", and that protocols can feel free to
>> compress all ints by default.
>
> Yes, that was the intent of the scheme I was proposing.
>
> Chad
>
> On 8/21/08 5:06 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
>
> Comments inline.
>
> On Aug 21, 2008, at 4:57 PM, Chad Walters wrote:
>
>> Looking at things from the standpoint of efficiency also argues for
>> a limited set of modifiers, since we need to actually pass this
>> information on the wire.
>>
>> In THRIFT-110, I suggest that we reserve 2 bits at the top of the
>> type byte for modifiers. This allows for 3 different modifiers for
>> any given primary type (plus a default).
> I'm fine with suggesting we keep the list short, but isn't saying we
> keep 2 bits free an implementation concern for protocols?
>
>> For integer types, I suggested that we add "fixed" and
>> "zipper"/"zigzag". The default assumption would be that a variable
>> encoding could be used. "fixed" would mean that variable encoding
>> is unlikely to be helpful, and "zipper"/"zigzag" would suggest a
>> variable encoding suitable for fields likely to contain a mix of
>> negative and positive values.
> I presume this would mean that the default modifier should be
> interpreted as "variable", and that protocols can feel free to
> compress all ints by default.
>
>> For string types, I suggested that we add "extern". Strings marked
>> with this modifier could be stored in some separate indexed table
>> and referenced by index when passed through the protocol.
>>
>> Does this sound like a good starting point? Are there other
>> modifiers we'd like to add?
>>
>> Chad
>
>
>
>
Re: Adding a "vint" type to Thrift
Posted by Chad Walters <ch...@powerset.com>.
>I'm fine with suggesting we keep the list short, but isn't saying we
>keep 2 bits free an implementation concern for protocols?
It is and it isn't. I think its reasonable for the framework to concern itself with how the protocols might implement the requirements that the framework is imposing. If we have more than 256 possible modified types, then it becomes impossible for any protocol to represent a type and its modifiers in a single byte.
>I presume this would mean that the default modifier should be
>interpreted as "variable", and that protocols can feel free to
>compress all ints by default.
Yes, that was the intent of the scheme I was proposing.
Chad
On 8/21/08 5:06 PM, "Bryan Duxbury" <br...@rapleaf.com> wrote:
Comments inline.
On Aug 21, 2008, at 4:57 PM, Chad Walters wrote:
> Looking at things from the standpoint of efficiency also argues for
> a limited set of modifiers, since we need to actually pass this
> information on the wire.
>
> In THRIFT-110, I suggest that we reserve 2 bits at the top of the
> type byte for modifiers. This allows for 3 different modifiers for
> any given primary type (plus a default).
I'm fine with suggesting we keep the list short, but isn't saying we
keep 2 bits free an implementation concern for protocols?
> For integer types, I suggested that we add "fixed" and
> "zipper"/"zigzag". The default assumption would be that a variable
> encoding could be used. "fixed" would mean that variable encoding
> is unlikely to be helpful, and "zipper"/"zigzag" would suggest a
> variable encoding suitable for fields likely to contain a mix of
> negative and positive values.
I presume this would mean that the default modifier should be
interpreted as "variable", and that protocols can feel free to
compress all ints by default.
> For string types, I suggested that we add "extern". Strings marked
> with this modifier could be stored in some separate indexed table
> and referenced by index when passed through the protocol.
>
> Does this sound like a good starting point? Are there other
> modifiers we'd like to add?
>
> Chad
RE: Adding a "vint" type to Thrift
Posted by Mark Slee <ms...@facebook.com>.
>> I'm fine with suggesting we keep the list short, but isn't saying we
keep 2 bits free an implementation concern for protocols?
I'd take Chad's suggestion here as an implementation recommendation, not
a requirement. I forgot to mention in my API change email that we also
need to change the signatures of readFieldBegin and readFieldEnd. Both
of those methods will also need to pass/return type hints. How they send
the hints over the wire is up to the protocol implementation. Using the
top 2 bits of the type field seems a reasonable suggestion in the
TBinaryProtocol world.
>> I presume this would mean that the default modifier should be
interpreted as "variable", and that protocols can feel free to compress
all ints by default.
Yes, the default behavior should be totally up to the Protocol
implementation.
-----Original Message-----
From: Bryan Duxbury [mailto:bryan@rapleaf.com]
Sent: Thursday, August 21, 2008 5:07 PM
To: thrift-dev@incubator.apache.org
Subject: Re: Adding a "vint" type to Thrift
Comments inline.
On Aug 21, 2008, at 4:57 PM, Chad Walters wrote:
> Looking at things from the standpoint of efficiency also argues for a
> limited set of modifiers, since we need to actually pass this
> information on the wire.
>
> In THRIFT-110, I suggest that we reserve 2 bits at the top of the type
> byte for modifiers. This allows for 3 different modifiers for any
> given primary type (plus a default).
I'm fine with suggesting we keep the list short, but isn't saying we
keep 2 bits free an implementation concern for protocols?
> For integer types, I suggested that we add "fixed" and
> "zipper"/"zigzag". The default assumption would be that a variable
> encoding could be used. "fixed" would mean that variable encoding is
> unlikely to be helpful, and "zipper"/"zigzag" would suggest a variable
> encoding suitable for fields likely to contain a mix of negative and
> positive values.
I presume this would mean that the default modifier should be
interpreted as "variable", and that protocols can feel free to compress
all ints by default.
> For string types, I suggested that we add "extern". Strings marked
> with this modifier could be stored in some separate indexed table and
> referenced by index when passed through the protocol.
>
> Does this sound like a good starting point? Are there other modifiers
> we'd like to add?
>
> Chad
>
> On 8/21/08 4:41 PM, "Mark Slee" <ms...@facebook.com> wrote:
>
> I'd propose starting with a fixed set of type hints that we agree
> upon.
> We could certainly implement a dynamic any-string goes type of thing.
> But here's my concern with that:
>
> 1/ If we agree upon a fixed set of modifiers, we're more likely to get
> them implemented across more languages since deciding on the set will
> involve getting some basic agreement amongst the Thrift community.
> We're
> likely to retain more overall compatibility.
>
> 2/ If the modifiers are dynamic, it implies that people would make
> their own local changes, and write their own custom Protocol
> implementations.
> We're more likely to duplicate this work or disagree on things. This
> will make custom protocol submissions back into the trunk more
> difficult, and the core project becomes more likely to lose quality
> work.
>
> So, while a dynamic implementation is arguably more elegant/ flexible,
> I think having the community agree upon a reasonable set of type
> annotations is most healthy for the coherence of Thrift in the long
> run.
>
> Cheers,
> Mark
>
> -----Original Message-----
> From: Bryan Duxbury [mailto:bryan@rapleaf.com]
> Sent: Thursday, August 21, 2008 3:40 PM
> To: thrift-dev@incubator.apache.org
> Subject: Re: Adding a "vint" type to Thrift
>
> I'm not opposed to using a type modifier approach instead of a
> separate type. Will we really be adding "arbitrary" annotations to the
> objects, or will they be of a specific nature? That is, will we have
> modifiers like "fixed", "variable", "extern", etc. be compiler tokens,
> or will you just be allowed to add any string you want as a modifier
> before the type? The latter would certainly be more flexible, but
> would be more complicated as well.
>
> -Bryan
>
> On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
>
>> First, I don't understand why it would be limited to 64-bit. What if
>> I have an integer that I want represented as 32 bits in memory but
>> serialized with a variable-length encoding.
>>
>> Second, I wonder if creating a new type is the best solution. I've
>> been thinking for a while that it might be useful to add arbitrary
>> annotations to types. We already have a few special cases of this
>> (cpp_type in C++, slist in Erlang), and vint would be a third. I
>> think it might be time to bite the bullet and implement annotations
>> so
>
>> we can stop adding new syntactic constructs for each minor extension
>> to the data model.
>>
>> --David
>>
>> Bryan Duxbury wrote:
>>> All,
>>>
>>> We've been following the discussion on THRIFT-110. I think as a
>>> first
>
>>> step we should add a "vint" (variable integer) to Thrift.
>>>
>>> My general proposal is that we add vint as a new type. Protocols
>>> that
>
>>> don't do variable length integers can treat vints as i64s. Protocols
>>> that do implement variable length integers can choose encodings that
>>> make sense for them.
>>>
>>> We'd have to change the compiler and all the generators and
>>> libraries
>
>>> to recognize vint, if for no other reason than to treat it as an
>>> i64.
>>> However, this would make implementing other protocols that included
>>> variable length integers very easy.
>>>
>>> Are there any objections to this proposal? If not, I'll make an
>>> issue, and Rapleaf is willing to commit time to this enhancement.
>>>
>>> -Bryan
>>>
>
>
>
Re: Adding a "vint" type to Thrift
Posted by Bryan Duxbury <br...@rapleaf.com>.
Comments inline.
On Aug 21, 2008, at 4:57 PM, Chad Walters wrote:
> Looking at things from the standpoint of efficiency also argues for
> a limited set of modifiers, since we need to actually pass this
> information on the wire.
>
> In THRIFT-110, I suggest that we reserve 2 bits at the top of the
> type byte for modifiers. This allows for 3 different modifiers for
> any given primary type (plus a default).
I'm fine with suggesting we keep the list short, but isn't saying we
keep 2 bits free an implementation concern for protocols?
> For integer types, I suggested that we add "fixed" and
> "zipper"/"zigzag". The default assumption would be that a variable
> encoding could be used. "fixed" would mean that variable encoding
> is unlikely to be helpful, and "zipper"/"zigzag" would suggest a
> variable encoding suitable for fields likely to contain a mix of
> negative and positive values.
I presume this would mean that the default modifier should be
interpreted as "variable", and that protocols can feel free to
compress all ints by default.
> For string types, I suggested that we add "extern". Strings marked
> with this modifier could be stored in some separate indexed table
> and referenced by index when passed through the protocol.
>
> Does this sound like a good starting point? Are there other
> modifiers we'd like to add?
>
> Chad
>
> On 8/21/08 4:41 PM, "Mark Slee" <ms...@facebook.com> wrote:
>
> I'd propose starting with a fixed set of type hints that we agree
> upon.
> We could certainly implement a dynamic any-string goes type of thing.
> But here's my concern with that:
>
> 1/ If we agree upon a fixed set of modifiers, we're more likely to get
> them implemented across more languages since deciding on the set will
> involve getting some basic agreement amongst the Thrift community.
> We're
> likely to retain more overall compatibility.
>
> 2/ If the modifiers are dynamic, it implies that people would make
> their
> own local changes, and write their own custom Protocol
> implementations.
> We're more likely to duplicate this work or disagree on things. This
> will make custom protocol submissions back into the trunk more
> difficult, and the core project becomes more likely to lose quality
> work.
>
> So, while a dynamic implementation is arguably more elegant/
> flexible, I
> think having the community agree upon a reasonable set of type
> annotations is most healthy for the coherence of Thrift in the long
> run.
>
> Cheers,
> Mark
>
> -----Original Message-----
> From: Bryan Duxbury [mailto:bryan@rapleaf.com]
> Sent: Thursday, August 21, 2008 3:40 PM
> To: thrift-dev@incubator.apache.org
> Subject: Re: Adding a "vint" type to Thrift
>
> I'm not opposed to using a type modifier approach instead of a
> separate
> type. Will we really be adding "arbitrary" annotations to the objects,
> or will they be of a specific nature? That is, will we have modifiers
> like "fixed", "variable", "extern", etc. be compiler tokens, or
> will you
> just be allowed to add any string you want as a modifier before the
> type? The latter would certainly be more flexible, but would be more
> complicated as well.
>
> -Bryan
>
> On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
>
>> First, I don't understand why it would be limited to 64-bit. What if
>> I have an integer that I want represented as 32 bits in memory but
>> serialized with a variable-length encoding.
>>
>> Second, I wonder if creating a new type is the best solution. I've
>> been thinking for a while that it might be useful to add arbitrary
>> annotations to types. We already have a few special cases of this
>> (cpp_type in C++, slist in Erlang), and vint would be a third. I
>> think it might be time to bite the bullet and implement
>> annotations so
>
>> we can stop adding new syntactic constructs for each minor extension
>> to the data model.
>>
>> --David
>>
>> Bryan Duxbury wrote:
>>> All,
>>>
>>> We've been following the discussion on THRIFT-110. I think as a
>>> first
>
>>> step we should add a "vint" (variable integer) to Thrift.
>>>
>>> My general proposal is that we add vint as a new type. Protocols
>>> that
>
>>> don't do variable length integers can treat vints as i64s. Protocols
>>> that do implement variable length integers can choose encodings that
>>> make sense for them.
>>>
>>> We'd have to change the compiler and all the generators and
>>> libraries
>
>>> to recognize vint, if for no other reason than to treat it as an
>>> i64.
>>> However, this would make implementing other protocols that included
>>> variable length integers very easy.
>>>
>>> Are there any objections to this proposal? If not, I'll make an
>>> issue, and Rapleaf is willing to commit time to this enhancement.
>>>
>>> -Bryan
>>>
>
>
>
Re: Adding a "vint" type to Thrift
Posted by Chad Walters <ch...@powerset.com>.
Looking at things from the standpoint of efficiency also argues for a limited set of modifiers, since we need to actually pass this information on the wire.
In THRIFT-110, I suggest that we reserve 2 bits at the top of the type byte for modifiers. This allows for 3 different modifiers for any given primary type (plus a default).
For integer types, I suggested that we add "fixed" and "zipper"/"zigzag". The default assumption would be that a variable encoding could be used. "fixed" would mean that variable encoding is unlikely to be helpful, and "zipper"/"zigzag" would suggest a variable encoding suitable for fields likely to contain a mix of negative and positive values.
For string types, I suggested that we add "extern". Strings marked with this modifier could be stored in some separate indexed table and referenced by index when passed through the protocol.
Does this sound like a good starting point? Are there other modifiers we'd like to add?
Chad
On 8/21/08 4:41 PM, "Mark Slee" <ms...@facebook.com> wrote:
I'd propose starting with a fixed set of type hints that we agree upon.
We could certainly implement a dynamic any-string goes type of thing.
But here's my concern with that:
1/ If we agree upon a fixed set of modifiers, we're more likely to get
them implemented across more languages since deciding on the set will
involve getting some basic agreement amongst the Thrift community. We're
likely to retain more overall compatibility.
2/ If the modifiers are dynamic, it implies that people would make their
own local changes, and write their own custom Protocol implementations.
We're more likely to duplicate this work or disagree on things. This
will make custom protocol submissions back into the trunk more
difficult, and the core project becomes more likely to lose quality
work.
So, while a dynamic implementation is arguably more elegant/flexible, I
think having the community agree upon a reasonable set of type
annotations is most healthy for the coherence of Thrift in the long run.
Cheers,
Mark
-----Original Message-----
From: Bryan Duxbury [mailto:bryan@rapleaf.com]
Sent: Thursday, August 21, 2008 3:40 PM
To: thrift-dev@incubator.apache.org
Subject: Re: Adding a "vint" type to Thrift
I'm not opposed to using a type modifier approach instead of a separate
type. Will we really be adding "arbitrary" annotations to the objects,
or will they be of a specific nature? That is, will we have modifiers
like "fixed", "variable", "extern", etc. be compiler tokens, or will you
just be allowed to add any string you want as a modifier before the
type? The latter would certainly be more flexible, but would be more
complicated as well.
-Bryan
On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
> First, I don't understand why it would be limited to 64-bit. What if
> I have an integer that I want represented as 32 bits in memory but
> serialized with a variable-length encoding.
>
> Second, I wonder if creating a new type is the best solution. I've
> been thinking for a while that it might be useful to add arbitrary
> annotations to types. We already have a few special cases of this
> (cpp_type in C++, slist in Erlang), and vint would be a third. I
> think it might be time to bite the bullet and implement annotations so
> we can stop adding new syntactic constructs for each minor extension
> to the data model.
>
> --David
>
> Bryan Duxbury wrote:
>> All,
>>
>> We've been following the discussion on THRIFT-110. I think as a first
>> step we should add a "vint" (variable integer) to Thrift.
>>
>> My general proposal is that we add vint as a new type. Protocols that
>> don't do variable length integers can treat vints as i64s. Protocols
>> that do implement variable length integers can choose encodings that
>> make sense for them.
>>
>> We'd have to change the compiler and all the generators and libraries
>> to recognize vint, if for no other reason than to treat it as an i64.
>> However, this would make implementing other protocols that included
>> variable length integers very easy.
>>
>> Are there any objections to this proposal? If not, I'll make an
>> issue, and Rapleaf is willing to commit time to this enhancement.
>>
>> -Bryan
>>
RE: Adding a "vint" type to Thrift
Posted by Mark Slee <ms...@facebook.com>.
I'd propose starting with a fixed set of type hints that we agree upon.
We could certainly implement a dynamic any-string goes type of thing.
But here's my concern with that:
1/ If we agree upon a fixed set of modifiers, we're more likely to get
them implemented across more languages since deciding on the set will
involve getting some basic agreement amongst the Thrift community. We're
likely to retain more overall compatibility.
2/ If the modifiers are dynamic, it implies that people would make their
own local changes, and write their own custom Protocol implementations.
We're more likely to duplicate this work or disagree on things. This
will make custom protocol submissions back into the trunk more
difficult, and the core project becomes more likely to lose quality
work.
So, while a dynamic implementation is arguably more elegant/flexible, I
think having the community agree upon a reasonable set of type
annotations is most healthy for the coherence of Thrift in the long run.
Cheers,
Mark
-----Original Message-----
From: Bryan Duxbury [mailto:bryan@rapleaf.com]
Sent: Thursday, August 21, 2008 3:40 PM
To: thrift-dev@incubator.apache.org
Subject: Re: Adding a "vint" type to Thrift
I'm not opposed to using a type modifier approach instead of a separate
type. Will we really be adding "arbitrary" annotations to the objects,
or will they be of a specific nature? That is, will we have modifiers
like "fixed", "variable", "extern", etc. be compiler tokens, or will you
just be allowed to add any string you want as a modifier before the
type? The latter would certainly be more flexible, but would be more
complicated as well.
-Bryan
On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
> First, I don't understand why it would be limited to 64-bit. What if
> I have an integer that I want represented as 32 bits in memory but
> serialized with a variable-length encoding.
>
> Second, I wonder if creating a new type is the best solution. I've
> been thinking for a while that it might be useful to add arbitrary
> annotations to types. We already have a few special cases of this
> (cpp_type in C++, slist in Erlang), and vint would be a third. I
> think it might be time to bite the bullet and implement annotations so
> we can stop adding new syntactic constructs for each minor extension
> to the data model.
>
> --David
>
> Bryan Duxbury wrote:
>> All,
>>
>> We've been following the discussion on THRIFT-110. I think as a first
>> step we should add a "vint" (variable integer) to Thrift.
>>
>> My general proposal is that we add vint as a new type. Protocols that
>> don't do variable length integers can treat vints as i64s. Protocols
>> that do implement variable length integers can choose encodings that
>> make sense for them.
>>
>> We'd have to change the compiler and all the generators and libraries
>> to recognize vint, if for no other reason than to treat it as an i64.
>> However, this would make implementing other protocols that included
>> variable length integers very easy.
>>
>> Are there any objections to this proposal? If not, I'll make an
>> issue, and Rapleaf is willing to commit time to this enhancement.
>>
>> -Bryan
>>
Re: Adding a "vint" type to Thrift
Posted by Bryan Duxbury <br...@rapleaf.com>.
I'm not opposed to using a type modifier approach instead of a
separate type. Will we really be adding "arbitrary" annotations to
the objects, or will they be of a specific nature? That is, will we
have modifiers like "fixed", "variable", "extern", etc. be compiler
tokens, or will you just be allowed to add any string you want as a
modifier before the type? The latter would certainly be more
flexible, but would be more complicated as well.
-Bryan
On Aug 21, 2008, at 2:07 PM, David Reiss wrote:
> First, I don't understand why it would be limited to 64-bit. What if
> I have an integer that I want represented as 32 bits in memory but
> serialized with a variable-length encoding.
>
> Second, I wonder if creating a new type is the best solution. I've
> been thinking for a while that it might be useful to add arbitrary
> annotations to types. We already have a few special cases of this
> (cpp_type in C++, slist in Erlang), and vint would be a third. I
> think it might be time to bite the bullet and implement annotations
> so we can stop adding new syntactic constructs for each minor
> extension
> to the data model.
>
> --David
>
> Bryan Duxbury wrote:
>> All,
>>
>> We've been following the discussion on THRIFT-110. I think as a first
>> step we should add a "vint" (variable integer) to Thrift.
>>
>> My general proposal is that we add vint as a new type. Protocols that
>> don't do variable length integers can treat vints as i64s. Protocols
>> that do implement variable length integers can choose encodings that
>> make sense for them.
>>
>> We'd have to change the compiler and all the generators and libraries
>> to recognize vint, if for no other reason than to treat it as an i64.
>> However, this would make implementing other protocols that included
>> variable length integers very easy.
>>
>> Are there any objections to this proposal? If not, I'll make an
>> issue, and Rapleaf is willing to commit time to this enhancement.
>>
>> -Bryan
>>
RE: Adding a "vint" type to Thrift
Posted by Mark Slee <ms...@facebook.com>.
I agree that a new first-order type is not the best way to go there. A
"variable integer" isn't a different type than an i32 or an i64, rather
it's a specification of how you want a protocol to handle transporting
the integer. How to encode/decode a type is different from what the type
is. I personally feel very strongly that we should strive to avoid
type-bloat in Thrift. Adding one new type or custom field here and there
seems very simple to those of us already familiar with the project, but
it all adds up over time and costs the project by making it less
approachable for new users/developers.
I agree with David (and I think Chad has also been suggesting the same
thing) that we should extend the Protocol abstraction to support type
annotations. This way, we keep the number of types small and
approachable, but allow for greater flexibility in protocol
implementations.
The API would change from something like:
void writeI32(int)
int readI32()
To:
void writeI32(int, hints)
int readi32(hints)
The default implementations can just ignore the hints entirely, keeping
things as simple as they are today. Advanced implementations can require
usage of the hints.
Cheers,
Mark
-----Original Message-----
From: David Reiss [mailto:dreiss@facebook.com]
Sent: Thursday, August 21, 2008 2:07 PM
To: thrift-dev@incubator.apache.org
Subject: Re: Adding a "vint" type to Thrift
First, I don't understand why it would be limited to 64-bit. What if I
have an integer that I want represented as 32 bits in memory but
serialized with a variable-length encoding.
Second, I wonder if creating a new type is the best solution. I've been
thinking for a while that it might be useful to add arbitrary
annotations to types. We already have a few special cases of this
(cpp_type in C++, slist in Erlang), and vint would be a third. I think
it might be time to bite the bullet and implement annotations so we can
stop adding new syntactic constructs for each minor extension to the
data model.
--David
Bryan Duxbury wrote:
> All,
>
> We've been following the discussion on THRIFT-110. I think as a first
> step we should add a "vint" (variable integer) to Thrift.
>
> My general proposal is that we add vint as a new type. Protocols that
> don't do variable length integers can treat vints as i64s. Protocols
> that do implement variable length integers can choose encodings that
> make sense for them.
>
> We'd have to change the compiler and all the generators and libraries
> to recognize vint, if for no other reason than to treat it as an i64.
> However, this would make implementing other protocols that included
> variable length integers very easy.
>
> Are there any objections to this proposal? If not, I'll make an issue,
> and Rapleaf is willing to commit time to this enhancement.
>
> -Bryan
>
Re: Adding a "vint" type to Thrift
Posted by David Reiss <dr...@facebook.com>.
First, I don't understand why it would be limited to 64-bit. What if
I have an integer that I want represented as 32 bits in memory but
serialized with a variable-length encoding.
Second, I wonder if creating a new type is the best solution. I've
been thinking for a while that it might be useful to add arbitrary
annotations to types. We already have a few special cases of this
(cpp_type in C++, slist in Erlang), and vint would be a third. I
think it might be time to bite the bullet and implement annotations
so we can stop adding new syntactic constructs for each minor extension
to the data model.
--David
Bryan Duxbury wrote:
> All,
>
> We've been following the discussion on THRIFT-110. I think as a first
> step we should add a "vint" (variable integer) to Thrift.
>
> My general proposal is that we add vint as a new type. Protocols that
> don't do variable length integers can treat vints as i64s. Protocols
> that do implement variable length integers can choose encodings that
> make sense for them.
>
> We'd have to change the compiler and all the generators and libraries
> to recognize vint, if for no other reason than to treat it as an i64.
> However, this would make implementing other protocols that included
> variable length integers very easy.
>
> Are there any objections to this proposal? If not, I'll make an
> issue, and Rapleaf is willing to commit time to this enhancement.
>
> -Bryan
>