Implementing Thrift language libraries by embedding C

[Mark Slee <ms...@facebook.com>]

I'm forking this part of the e-mail thread with a new subject line, as this has gone in the direction of a specific technical discussion.

On this topic, here are my opinions as a Thrift user:
- Ease-of-use is absolutely critical
 -- Being required to install/maintain PHP/Python/Ruby/etc. extensions is very annoying and a huge barrier to entry, I would much rather just have source that I can use directly
 -- When evaluating painkillers (like RPC systems), I first evaluate the option that's easiest for me to evaluate, and if it effectively makes my pain go away, I stick with it and move on with my real work
 -- Being able to easily understand generated code in my language of choice is very nice, I like learning/understanding how things work in domains that I understand
- I want good enough performance, sometimes I may need really high performance
 -- Having extensions available to enable this is great

My thoughts as a Thrift developer/community-member:
- Offering extensions for greater performance generally makes a lot of sense
 -- Sharing core code across these extensions is clearly a good thing
- Extension frameworks are often quite complex to get right
 -- We'll likely need to find N people with expertise in language-specific extension-writing to develop and maintain them
 -- My sense is that these people are harder to find than experts in the languages themselves
 -- There may be just as much tricky code required to effectively generate/wrap a common C library for each target language (I think this is what Bryan is getting at)
- Seems like extensions would probably only address the binary protocol, we don't want to force all custom protocol implementation into C

Balancing these two, I'm supportive of work on C/extension code if there's enough community to support it (which I think will be the hard part... as fewer people truly love programming in C these days *and* deeply understand other language extension models).

Either way, I feel quite strongly that we should always continue offering pure language libraries. I'd be really turned off as a user if I had to install language extensions just to try Thrift out. I'm quite confident that imposing this as a requirement would nontrivially stifle growth of the Thrift userbase, which would in turn stifle advancement of the project as a whole (Thrift users are programmers, most of our community members start out as users).

Cheers,
mcslee


-----Original Message-----
From: Mayan Moudgill [mailto:mayan@bestweb.net] 
Sent: Tuesday, August 17, 2010 2:43 AM
To: thrift-dev@incubator.apache.org
Subject: Re: sharing knowledge means sharing control



Bryan Duxbury wrote:
> I feel that this suggestion is a bit of a red herring. Sure, you could
> implement a tiny fraction of the Thrift "library" in C - maybe just the
> protocol? - but then you'd still be stuck needing the full code generator
> and a set of related language-specific tools. I'm thinking that even if you
> coded up the core of the protocol in C, you'd probably have to put a
> language-specific C-extension wrapper on it in order for it to be usable in
> the way that all the languages operate.
> 
> Bottom line, I don't think there's near enough benefit to putting the "core"
> in a common C blob. I could be convinced otherwise if someone has some
> strong ideas.
> 

Why is (was) there a C-version of the Python run-time? Performance maybe?

Part of the problem with having N languages is that any fast, optimized, 
but hard-to-write implementations/features will either have to be 
written (and debugged!) N times or will end up not being available to 
everyone. This, among other things, lowers the benefits (and therefore 
attractiveness) of writing any high-performance code. Which may be why 
the Thrift run-timess (even the C++ runtime) are fat & slow compared to 
what it could be.

Re: Implementing Thrift language libraries by embedding C

[Anthony Molinaro <an...@alumni.caltech.edu>]

I'm assuming most people have used SWIG in some capacity.  Since it
supports wrapping C++, it might work with the current C++ library.
With lwes we used it for perl/python/php wrappers, while ruby, dotnet
and java (and of course C) have their own bindings.  This was mostly based
on who wanted to do what development, when I originally wrote it in 1998
it was purely java, a C port followed after that, and then perl but using
xs.  The switch to swig cleaned up quite a bit (although it was Michael Lum
who did most of that).  Now lwes doesn't actually have clients/servers as
its a fire and forget logging mechanism, but for the most part if you
have clean library functions it's pretty easy to get setup.  Once things
slow down a little at OpenX I might have some time to experiment with
it.

The one nice thing about having a C/C++ library that plays nice with SWIG
is it opens support to many languages that you might not otherwise want
to support (see http://www.swig.org/compat.html#SupportedLanguages).

-Anthony

On Tue, Aug 17, 2010 at 01:06:42PM -0700, Bryan Duxbury wrote:
> Yeah, this is basically what I was trying to convey. I think it's absolutely
> critical to have native versions for situations when the extension cannot be
> used, and while I support the idea of sharing a common implementation for
> extensions, I do think we'll end up needing nearly as much wrapper code to
> support the common extension connecting to various languages. If it still
> seems like it's worth pursuing this path, by all means, have at it, and
> we'll see what it ends up looking like.
> 
> On Tue, Aug 17, 2010 at 12:45 PM, Mark Slee <ms...@facebook.com> wrote:
> 
> > I'm forking this part of the e-mail thread with a new subject line, as this
> > has gone in the direction of a specific technical discussion.
> >
> > On this topic, here are my opinions as a Thrift user:
> > - Ease-of-use is absolutely critical
> >  -- Being required to install/maintain PHP/Python/Ruby/etc. extensions is
> > very annoying and a huge barrier to entry, I would much rather just have
> > source that I can use directly
> >  -- When evaluating painkillers (like RPC systems), I first evaluate the
> > option that's easiest for me to evaluate, and if it effectively makes my
> > pain go away, I stick with it and move on with my real work
> >  -- Being able to easily understand generated code in my language of choice
> > is very nice, I like learning/understanding how things work in domains that
> > I understand
> > - I want good enough performance, sometimes I may need really high
> > performance
> >  -- Having extensions available to enable this is great
> >
> > My thoughts as a Thrift developer/community-member:
> > - Offering extensions for greater performance generally makes a lot of
> > sense
> >  -- Sharing core code across these extensions is clearly a good thing
> > - Extension frameworks are often quite complex to get right
> >  -- We'll likely need to find N people with expertise in language-specific
> > extension-writing to develop and maintain them
> >  -- My sense is that these people are harder to find than experts in the
> > languages themselves
> >  -- There may be just as much tricky code required to effectively
> > generate/wrap a common C library for each target language (I think this is
> > what Bryan is getting at)
> > - Seems like extensions would probably only address the binary protocol, we
> > don't want to force all custom protocol implementation into C
> >
> > Balancing these two, I'm supportive of work on C/extension code if there's
> > enough community to support it (which I think will be the hard part... as
> > fewer people truly love programming in C these days *and* deeply understand
> > other language extension models).
> >
> > Either way, I feel quite strongly that we should always continue offering
> > pure language libraries. I'd be really turned off as a user if I had to
> > install language extensions just to try Thrift out. I'm quite confident that
> > imposing this as a requirement would nontrivially stifle growth of the
> > Thrift userbase, which would in turn stifle advancement of the project as a
> > whole (Thrift users are programmers, most of our community members start out
> > as users).
> >
> > Cheers,
> > mcslee
> >
> >
> > -----Original Message-----
> > From: Mayan Moudgill [mailto:mayan@bestweb.net]
> > Sent: Tuesday, August 17, 2010 2:43 AM
> > To: thrift-dev@incubator.apache.org
> > Subject: Re: sharing knowledge means sharing control
> >
> >
> >
> > Bryan Duxbury wrote:
> > > I feel that this suggestion is a bit of a red herring. Sure, you could
> > > implement a tiny fraction of the Thrift "library" in C - maybe just the
> > > protocol? - but then you'd still be stuck needing the full code generator
> > > and a set of related language-specific tools. I'm thinking that even if
> > you
> > > coded up the core of the protocol in C, you'd probably have to put a
> > > language-specific C-extension wrapper on it in order for it to be usable
> > in
> > > the way that all the languages operate.
> > >
> > > Bottom line, I don't think there's near enough benefit to putting the
> > "core"
> > > in a common C blob. I could be convinced otherwise if someone has some
> > > strong ideas.
> > >
> >
> > Why is (was) there a C-version of the Python run-time? Performance maybe?
> >
> > Part of the problem with having N languages is that any fast, optimized,
> > but hard-to-write implementations/features will either have to be
> > written (and debugged!) N times or will end up not being available to
> > everyone. This, among other things, lowers the benefits (and therefore
> > attractiveness) of writing any high-performance code. Which may be why
> > the Thrift run-timess (even the C++ runtime) are fat & slow compared to
> > what it could be.
> >

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <an...@alumni.caltech.edu>

Re: Implementing Thrift language libraries by embedding C

[Bryan Duxbury <br...@rapleaf.com>]

Yeah, this is basically what I was trying to convey. I think it's absolutely
critical to have native versions for situations when the extension cannot be
used, and while I support the idea of sharing a common implementation for
extensions, I do think we'll end up needing nearly as much wrapper code to
support the common extension connecting to various languages. If it still
seems like it's worth pursuing this path, by all means, have at it, and
we'll see what it ends up looking like.

On Tue, Aug 17, 2010 at 12:45 PM, Mark Slee <ms...@facebook.com> wrote:

> I'm forking this part of the e-mail thread with a new subject line, as this
> has gone in the direction of a specific technical discussion.
>
> On this topic, here are my opinions as a Thrift user:
> - Ease-of-use is absolutely critical
>  -- Being required to install/maintain PHP/Python/Ruby/etc. extensions is
> very annoying and a huge barrier to entry, I would much rather just have
> source that I can use directly
>  -- When evaluating painkillers (like RPC systems), I first evaluate the
> option that's easiest for me to evaluate, and if it effectively makes my
> pain go away, I stick with it and move on with my real work
>  -- Being able to easily understand generated code in my language of choice
> is very nice, I like learning/understanding how things work in domains that
> I understand
> - I want good enough performance, sometimes I may need really high
> performance
>  -- Having extensions available to enable this is great
>
> My thoughts as a Thrift developer/community-member:
> - Offering extensions for greater performance generally makes a lot of
> sense
>  -- Sharing core code across these extensions is clearly a good thing
> - Extension frameworks are often quite complex to get right
>  -- We'll likely need to find N people with expertise in language-specific
> extension-writing to develop and maintain them
>  -- My sense is that these people are harder to find than experts in the
> languages themselves
>  -- There may be just as much tricky code required to effectively
> generate/wrap a common C library for each target language (I think this is
> what Bryan is getting at)
> - Seems like extensions would probably only address the binary protocol, we
> don't want to force all custom protocol implementation into C
>
> Balancing these two, I'm supportive of work on C/extension code if there's
> enough community to support it (which I think will be the hard part... as
> fewer people truly love programming in C these days *and* deeply understand
> other language extension models).
>
> Either way, I feel quite strongly that we should always continue offering
> pure language libraries. I'd be really turned off as a user if I had to
> install language extensions just to try Thrift out. I'm quite confident that
> imposing this as a requirement would nontrivially stifle growth of the
> Thrift userbase, which would in turn stifle advancement of the project as a
> whole (Thrift users are programmers, most of our community members start out
> as users).
>
> Cheers,
> mcslee
>
>
> -----Original Message-----
> From: Mayan Moudgill [mailto:mayan@bestweb.net]
> Sent: Tuesday, August 17, 2010 2:43 AM
> To: thrift-dev@incubator.apache.org
> Subject: Re: sharing knowledge means sharing control
>
>
>
> Bryan Duxbury wrote:
> > I feel that this suggestion is a bit of a red herring. Sure, you could
> > implement a tiny fraction of the Thrift "library" in C - maybe just the
> > protocol? - but then you'd still be stuck needing the full code generator
> > and a set of related language-specific tools. I'm thinking that even if
> you
> > coded up the core of the protocol in C, you'd probably have to put a
> > language-specific C-extension wrapper on it in order for it to be usable
> in
> > the way that all the languages operate.
> >
> > Bottom line, I don't think there's near enough benefit to putting the
> "core"
> > in a common C blob. I could be convinced otherwise if someone has some
> > strong ideas.
> >
>
> Why is (was) there a C-version of the Python run-time? Performance maybe?
>
> Part of the problem with having N languages is that any fast, optimized,
> but hard-to-write implementations/features will either have to be
> written (and debugged!) N times or will end up not being available to
> everyone. This, among other things, lowers the benefits (and therefore
> attractiveness) of writing any high-performance code. Which may be why
> the Thrift run-timess (even the C++ runtime) are fat & slow compared to
> what it could be.
>