You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Nevo Hed <nh...@aereo.com> on 2012/04/18 00:08:28 UTC
Do new (optional) fields really "pass through" when a message is
re-serialized in code compiled with old definition?
Hi All
In The Missing Guide
(http://diwakergupta.github.com/thrift-missing-guide/)<http://diwakergupta.github.com/thrift-missing-guide/>
,
Under "Best Practices/Versioning/Compatibility", at the end of 2nd bullet
it states:
> However, the unknown fields are not discarded, and if the message is later
> serialized, the unknown fields are serialized along with it — so* if the
> message is passed on to new code, the new fields are still available*.
This is really good theory, but is this really true? If so, is it true for
all languages?
For one I don't see how this would be possible if I look at the generated
C++ code. For y type "MyType", the generated MyType::write()
only references known fields from the thrift file used at compile time. I
also ran a ruby test that showed similar results (write file with old
version, read with new, add new optional field, write file, confirm by
reread, then read in old version, rewrite file, and new fields is gone).
As there is only storage for the fields known at compile time, I can only
see this as plausible for languages offering
reflection<http://en.wikipedia.org/wiki/Reflection_(computer_programming)>
.
Of course, if one maintains the bitstream (for example by keeping the
original serialized buffer, say in a TMemoryBuffer), then it can be
transmitted along to another system element, but the current element would
only be able to act as a message router, and not actually augment the
message.
Am I missing anything?
Thanks
-Nevo
Re: Do new (optional) fields really "pass through" when a message is
re-serialized in code compiled with old definition?
Posted by Bryan Duxbury <br...@rapleaf.com>.
We don't currently have a way to verify that the struct versions are the
same. If it's important for your structs' schemas to match, you should use
required fields, or consider making a protocol that adds the hash of the
thrift file (or some equivalent fingerprint) to the messages.
On Mon, Apr 30, 2012 at 12:14 PM, Nevo Hed <nh...@aereo.com> wrote:
> Thanks Bryan
>
> I have a followup to this question ... as there is no pass-through, and new
> fields will be obliterated, is there any way for a thrift server to check
> that the connected client is compiled with the same IDL?
>
> I was thinking of using adding a fingerprint field to the RPC call in
> question, where the client would include the
> fingerprint from the generated file, but then saw that these are only
> generated for C++, and the client I am concerned with is written in ruby,
> and the server is in C++.
>
> (I assume that since fingerprints are only used in the dense protocol an
> since I only see the dense proto
> mentioned in the cpp libs, that this would explain why they are emitted
> only into the cpp generated files. I assume that it would be trivial to
> emit them into other generated files, but wonder if this is reasonable)
>
> Thanks
> -Nevo
>
>
> On Tue, Apr 17, 2012 at 7:57 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
>
> > This is definitely not true, at least in ruby and java. I would be
> > surprised if any of the other libraries supported this.
> >
> > On Tue, Apr 17, 2012 at 3:08 PM, Nevo Hed <nh...@aereo.com> wrote:
> >
> > > Hi All
> > >
> > > In The Missing Guide
> > > (http://diwakergupta.github.com/thrift-missing-guide/)<
> > > http://diwakergupta.github.com/thrift-missing-guide/>
> > > ,
> > > Under "Best Practices/Versioning/Compatibility", at the end of 2nd
> bullet
> > > it states:
> > >
> > > > However, the unknown fields are not discarded, and if the message is
> > > later
> > > > serialized, the unknown fields are serialized along with it — so* if
> > the
> > > > message is passed on to new code, the new fields are still
> available*.
> > >
> > >
> > > This is really good theory, but is this really true? If so, is it true
> > for
> > > all languages?
> > >
> > > For one I don't see how this would be possible if I look at the
> generated
> > > C++ code. For y type "MyType", the generated MyType::write()
> > > only references known fields from the thrift file used at compile time.
> > I
> > > also ran a ruby test that showed similar results (write file with old
> > > version, read with new, add new optional field, write file, confirm by
> > > reread, then read in old version, rewrite file, and new fields is
> gone).
> > >
> > > As there is only storage for the fields known at compile time, I can
> only
> > > see this as plausible for languages offering
> > > reflection<
> > http://en.wikipedia.org/wiki/Reflection_(computer_programming)>
> > > .
> > >
> > > Of course, if one maintains the bitstream (for example by keeping the
> > > original serialized buffer, say in a TMemoryBuffer), then it can be
> > > transmitted along to another system element, but the current element
> > would
> > > only be able to act as a message router, and not actually augment the
> > > message.
> > >
> > > Am I missing anything?
> > >
> > > Thanks
> > >
> > > -Nevo
> > >
> >
>
Re: Do new (optional) fields really "pass through" when a message is
re-serialized in code compiled with old definition?
Posted by Nevo Hed <nh...@aereo.com>.
Thanks Bryan
I have a followup to this question ... as there is no pass-through, and new
fields will be obliterated, is there any way for a thrift server to check
that the connected client is compiled with the same IDL?
I was thinking of using adding a fingerprint field to the RPC call in
question, where the client would include the
fingerprint from the generated file, but then saw that these are only
generated for C++, and the client I am concerned with is written in ruby,
and the server is in C++.
(I assume that since fingerprints are only used in the dense protocol an
since I only see the dense proto
mentioned in the cpp libs, that this would explain why they are emitted
only into the cpp generated files. I assume that it would be trivial to
emit them into other generated files, but wonder if this is reasonable)
Thanks
-Nevo
On Tue, Apr 17, 2012 at 7:57 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
> This is definitely not true, at least in ruby and java. I would be
> surprised if any of the other libraries supported this.
>
> On Tue, Apr 17, 2012 at 3:08 PM, Nevo Hed <nh...@aereo.com> wrote:
>
> > Hi All
> >
> > In The Missing Guide
> > (http://diwakergupta.github.com/thrift-missing-guide/)<
> > http://diwakergupta.github.com/thrift-missing-guide/>
> > ,
> > Under "Best Practices/Versioning/Compatibility", at the end of 2nd bullet
> > it states:
> >
> > > However, the unknown fields are not discarded, and if the message is
> > later
> > > serialized, the unknown fields are serialized along with it — so* if
> the
> > > message is passed on to new code, the new fields are still available*.
> >
> >
> > This is really good theory, but is this really true? If so, is it true
> for
> > all languages?
> >
> > For one I don't see how this would be possible if I look at the generated
> > C++ code. For y type "MyType", the generated MyType::write()
> > only references known fields from the thrift file used at compile time.
> I
> > also ran a ruby test that showed similar results (write file with old
> > version, read with new, add new optional field, write file, confirm by
> > reread, then read in old version, rewrite file, and new fields is gone).
> >
> > As there is only storage for the fields known at compile time, I can only
> > see this as plausible for languages offering
> > reflection<
> http://en.wikipedia.org/wiki/Reflection_(computer_programming)>
> > .
> >
> > Of course, if one maintains the bitstream (for example by keeping the
> > original serialized buffer, say in a TMemoryBuffer), then it can be
> > transmitted along to another system element, but the current element
> would
> > only be able to act as a message router, and not actually augment the
> > message.
> >
> > Am I missing anything?
> >
> > Thanks
> >
> > -Nevo
> >
>
Re: Do new (optional) fields really "pass through" when a message is
re-serialized in code compiled with old definition?
Posted by Bryan Duxbury <br...@rapleaf.com>.
This is definitely not true, at least in ruby and java. I would be
surprised if any of the other libraries supported this.
On Tue, Apr 17, 2012 at 3:08 PM, Nevo Hed <nh...@aereo.com> wrote:
> Hi All
>
> In The Missing Guide
> (http://diwakergupta.github.com/thrift-missing-guide/)<
> http://diwakergupta.github.com/thrift-missing-guide/>
> ,
> Under "Best Practices/Versioning/Compatibility", at the end of 2nd bullet
> it states:
>
> > However, the unknown fields are not discarded, and if the message is
> later
> > serialized, the unknown fields are serialized along with it — so* if the
> > message is passed on to new code, the new fields are still available*.
>
>
> This is really good theory, but is this really true? If so, is it true for
> all languages?
>
> For one I don't see how this would be possible if I look at the generated
> C++ code. For y type "MyType", the generated MyType::write()
> only references known fields from the thrift file used at compile time. I
> also ran a ruby test that showed similar results (write file with old
> version, read with new, add new optional field, write file, confirm by
> reread, then read in old version, rewrite file, and new fields is gone).
>
> As there is only storage for the fields known at compile time, I can only
> see this as plausible for languages offering
> reflection<http://en.wikipedia.org/wiki/Reflection_(computer_programming)>
> .
>
> Of course, if one maintains the bitstream (for example by keeping the
> original serialized buffer, say in a TMemoryBuffer), then it can be
> transmitted along to another system element, but the current element would
> only be able to act as a message router, and not actually augment the
> message.
>
> Am I missing anything?
>
> Thanks
>
> -Nevo
>