You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Tom Deering <to...@workiva.com> on 2015/11/09 17:32:38 UTC
Code Generation With Distributed, Versioned Dependencies
Does Thrift have a solution for generating code with distributed, versioned
dependencies (includes)? For example, suppose we have 3 repositories with
Thrift definitions:
*Foo:* Defines types, FooService.
*Bar (includes Foo):* Defines types, BarService extends FooService.
*Baz (includes Bar, Foo):* Defines BazService extends FooService, uses
types in Foo, Bar.
Suppose we want to generate code for Baz, which was written when Foo and
Bar were at commits X and Y. Is there a standard way to express pinned
Thrift dependencies and generate code with the necessary artifacts at the
necessary versions? This is a solved problem for most programming
languages, but how about Thrift IDL?
--
Tom Deering, PhD | (563) 249-9277 | Software Engineer, Workiva
Re: Code Generation With Distributed, Versioned Dependencies
Posted by Tom Deering <to...@workiva.com>.
Sorry if I wasn't clear. It sounds like you understand the problem I'm
asking about.
> When you are generating code with "external dependencies", how are you
"including" those files if you don't have them locally?
Exactly :) That's what I was hoping to answer. It would be great to be able
to generate code for a service without needing every service to have its
own local copy of the full transitive closure of included Thrift files.
Local copies of dependencies can get stale, may tempt you to edit them, etc.
> Perhaps you could give a more concrete example of what you mean.
Good idea. Consider Maven for Java projects, wherein a pom.xml file
describes a project's dependencies and from where they can be retrieved.
You *do* need a local copy of these in order to compile or run your
program. But you certainly wouldn't manually fetch the forward closure of
dependencies and check them all into your project. Instead this happens
lazily as-needed at build time. It would be nice if Thrift code generation
worked similarly. My question was whether such a thing exists, or if the
standard "solution" is to make a copy of your dependencies and keep them
around locally with your own service definition.
On Mon, Nov 9, 2015 at 1:50 PM, BCG <bg...@hushmail.com> wrote:
> On 11/9/2015 at 1:32 PM, "Tom Deering" <to...@workiva.com> wrote:
> >
> >Thanks, Ben. I understand the "be liberal in what you accept"
> >philosophy
> >that allows mismatched versions of a Thrift client and server to
> >talk
> >(regardless of whether that's a good idea
> ><https://techblog.workiva.com/tech-blog/postel%E2%80%99s-cycle>).
> >It seems
> >that you're suggesting, therefore, that pinning specific versions
> >of
> >external Thrift dependencies is unnecessary because due to
> >interoperability.
> >
> >However, I don't see that this solves the problem of generating
> >code from
> >Thrift files that have external includes (unless I keep a copy of
> >them
> >locally like you suggest).
>
> Maybe I'm not understanding your problem...
>
> I'm assuming by "external dependencies" you mean structs, enums, unions,
> typedefs, exceptions, or parent services contained is some other thrift
> file than the one you are compiling against... please correct me if I am
> wrong
>
> When you are generating code with "external dependencies", how are you
> "including" those files if you don't have them locally? AFAIK when you
> invoke the Thrift compiler you're operating on local files. So when you
> generate either your clients or your servers in whatever language, you're
> operating on some local file.
>
> >I was hoping for a solution that lets
> >each
> >service be the source of truth for the single copy of its own
> >Thrift
> >definition instead of keeping copies within dependent services.
> >
>
> Perhaps you could give a more concrete example of what you mean... you
> said this problem is solved in most programming languages already... maybe
> framing it in terms of another language that has solved this already would
> be helpful
>
> Thanks
>
> Ben
>
>
--
Tom Deering, PhD | (563) 249-9277 | Software Engineer, Workiva
Re: Code Generation With Distributed, Versioned Dependencies
Posted by BCG <bg...@hushmail.com>.
On 11/9/2015 at 1:32 PM, "Tom Deering" <to...@workiva.com> wrote:
>
>Thanks, Ben. I understand the "be liberal in what you accept"
>philosophy
>that allows mismatched versions of a Thrift client and server to
>talk
>(regardless of whether that's a good idea
><https://techblog.workiva.com/tech-blog/postel%E2%80%99s-cycle>).
>It seems
>that you're suggesting, therefore, that pinning specific versions
>of
>external Thrift dependencies is unnecessary because due to
>interoperability.
>
>However, I don't see that this solves the problem of generating
>code from
>Thrift files that have external includes (unless I keep a copy of
>them
>locally like you suggest).
Maybe I'm not understanding your problem...
I'm assuming by "external dependencies" you mean structs, enums, unions, typedefs, exceptions, or parent services contained is some other thrift file than the one you are compiling against... please correct me if I am wrong
When you are generating code with "external dependencies", how are you "including" those files if you don't have them locally? AFAIK when you invoke the Thrift compiler you're operating on local files. So when you generate either your clients or your servers in whatever language, you're operating on some local file.
>I was hoping for a solution that lets
>each
>service be the source of truth for the single copy of its own
>Thrift
>definition instead of keeping copies within dependent services.
>
Perhaps you could give a more concrete example of what you mean... you said this problem is solved in most programming languages already... maybe framing it in terms of another language that has solved this already would be helpful
Thanks
Ben
Re: Code Generation With Distributed, Versioned Dependencies
Posted by Tom Deering <to...@workiva.com>.
Thanks, Ben. I understand the "be liberal in what you accept" philosophy
that allows mismatched versions of a Thrift client and server to talk
(regardless of whether that's a good idea
<https://techblog.workiva.com/tech-blog/postel%E2%80%99s-cycle>). It seems
that you're suggesting, therefore, that pinning specific versions of
external Thrift dependencies is unnecessary because due to interoperability.
However, I don't see that this solves the problem of generating code from
Thrift files that have external includes (unless I keep a copy of them
locally like you suggest). I was hoping for a solution that lets each
service be the source of truth for the single copy of its own Thrift
definition instead of keeping copies within dependent services.
On Mon, Nov 9, 2015 at 10:49 AM, BCG <bg...@hushmail.com> wrote:
> On 11/9/2015 at 11:32 AM, "Tom Deering" <to...@workiva.com> wrote:
> >
> >Does Thrift have a solution for generating code with distributed,
> >versioned
> >dependencies (includes)? For example, suppose we have 3
> >repositories with
> >Thrift definitions:
> >
> >*Foo:* Defines types, FooService.
> >*Bar (includes Foo):* Defines types, BarService extends FooService.
> >*Baz (includes Bar, Foo):* Defines BazService extends FooService,
> >uses
> >types in Foo, Bar.
> >
> >Suppose we want to generate code for Baz, which was written when
> >Foo and
> >Bar were at commits X and Y. Is there a standard way to express
> >pinned
> >Thrift dependencies and generate code with the necessary artifacts
> >at the
> >necessary versions? This is a solved problem for most programming
> >languages, but how about Thrift IDL?
> >
>
> If you read section 5 of the Thrift whitepaper (
> https://thrift.apache.org/static/files/thrift-20070401.pdf), Thrift's
> versioning solution is outlined there. Generated code handles data
> definition changes robustly... there is a case analysis in there though
> that can alert you to the pitfalls.
>
> In summary your best bet is probably to implement default behavior for new
> fields when you update your servers. Don't remove fields if you can avoid
> it because new clients won't send the field when interacting with old
> servers that might not have a default behavior defined.
>
> To me, playing nice with Thrift's versioning solution is better than
> pinning to a specific version of an interface... if you need to do that
> though, you could simply change the namespace of your IDL to force clients
> to recompile. The other thing that comes to mind is to create a repository
> for the IDL dependencies you need in your version control system, and
> create a tag for the specific combination of versions that you need.
> Truthfully though, if you follow the strategy in the Thrift whitepaper,
> neither of these measures should be necessary.
>
> HTH
>
> Ben
>
>
--
Tom Deering, PhD | (563) 249-9277 | Software Engineer, Workiva
Re: Code Generation With Distributed, Versioned Dependencies
Posted by BCG <bg...@hushmail.com>.
On 11/9/2015 at 11:32 AM, "Tom Deering" <to...@workiva.com> wrote:
>
>Does Thrift have a solution for generating code with distributed,
>versioned
>dependencies (includes)? For example, suppose we have 3
>repositories with
>Thrift definitions:
>
>*Foo:* Defines types, FooService.
>*Bar (includes Foo):* Defines types, BarService extends FooService.
>*Baz (includes Bar, Foo):* Defines BazService extends FooService,
>uses
>types in Foo, Bar.
>
>Suppose we want to generate code for Baz, which was written when
>Foo and
>Bar were at commits X and Y. Is there a standard way to express
>pinned
>Thrift dependencies and generate code with the necessary artifacts
>at the
>necessary versions? This is a solved problem for most programming
>languages, but how about Thrift IDL?
>
If you read section 5 of the Thrift whitepaper (https://thrift.apache.org/static/files/thrift-20070401.pdf), Thrift's versioning solution is outlined there. Generated code handles data definition changes robustly... there is a case analysis in there though that can alert you to the pitfalls.
In summary your best bet is probably to implement default behavior for new fields when you update your servers. Don't remove fields if you can avoid it because new clients won't send the field when interacting with old servers that might not have a default behavior defined.
To me, playing nice with Thrift's versioning solution is better than pinning to a specific version of an interface... if you need to do that though, you could simply change the namespace of your IDL to force clients to recompile. The other thing that comes to mind is to create a repository for the IDL dependencies you need in your version control system, and create a tag for the specific combination of versions that you need. Truthfully though, if you follow the strategy in the Thrift whitepaper, neither of these measures should be necessary.
HTH
Ben