You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Bjorn Borud <bb...@gmail.com> on 2010/08/26 22:54:08 UTC
Thrift compiler in Java (have parser, willing to hack)
on the project I currently work on we have a (ANTLR-based) parser for
the Thrift IDL language in order to generate code for a proprietary
serialization library.
it struck me that perhaps we could use this parser the implement the
Thrift compiler in Java instead. this would mean that the thrift
compiler itself could be built as a platform independent artifact --
which should make it a lot more elegant to write Maven plugins for
Thrift. it would also eliminate the need (for us) to maintain Thrift
compiler binaries for all platforms and versions of the compiler.
currently the parser lacks some minor features, but this could easily be
rectified. the real job is to add the code generation for various
languages.
if anyone is interested in this, I am going to talk to some people
tomorrow to get formal approval for open sourcing it.
any thoughts?
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
Okay. I'll try to bring that branch back to life in the next few days.
It was integrated with a lot of other changes to simplify the build.
Don't forget: autoconf isn't required to build from a release tarball.
--David
On 08/26/2010 04:17 PM, Todd Lipcon wrote:
> On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dreiss@facebook.com
> <ma...@facebook.com>> wrote:
>
> Roger, you misunderstood that patch. It just lets you run configure
> and make
> to build the code generator and java/python/etc. libraries without
> having to
> build the C++ runtime library.
>
> Also, I think I have a patch lying around to make the code generator
> not rely
> on boost.
>
>
> Not relying on boost would be a great start! That's probably the biggest
> obstacle - autoconf, etc, is annoying but boost is just the worst.
>
> -Todd
>
>
> On 08/26/2010 04:03 PM, Todd Lipcon wrote:
> > On Thu, Aug 26, 2010 at 3:58 PM, Roger Meier
> <roger@bufferoverflow.ch <ma...@bufferoverflow.ch>>wrote:
> >
> >> Yes, that's an important issue!
> >>
> >> what do you think about a compiler that only needs C as described in
> >> https://issues.apache.org/jira/browse/THRIFT-506
> >>
> >> compile that one for different platforms and make it available
> for each
> >> version on the download web site?
> >>
> >> That gets rid of the boost dependency, but we're still forced to
> push out
> > built binaries for a bunch of platforms, which is kind of a pain
> compared to
> > Java.
> >
> > However, porting over all our existing generators will take some
> time, for
> > sure.
> >
> > -Todd
> >
> >>
> >> Am 26.08.2010 23:56, schrieb Todd Lipcon:
> >>
> >> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bborud@gmail.com
> <ma...@gmail.com>> wrote:
> >>>
> >>>
> >>>
> >>>> on the project I currently work on we have a (ANTLR-based)
> parser for
> >>>> the Thrift IDL language in order to generate code for a proprietary
> >>>> serialization library.
> >>>>
> >>>> it struck me that perhaps we could use this parser the
> implement the
> >>>> Thrift compiler in Java instead. this would mean that the thrift
> >>>> compiler itself could be built as a platform independent
> artifact --
> >>>> which should make it a lot more elegant to write Maven plugins for
> >>>> Thrift. it would also eliminate the need (for us) to maintain
> Thrift
> >>>> compiler binaries for all platforms and versions of the compiler.
> >>>>
> >>>> currently the parser lacks some minor features, but this could
> easily be
> >>>> rectified. the real job is to add the code generation for various
> >>>> languages.
> >>>>
> >>>> if anyone is interested in this, I am going to talk to some people
> >>>> tomorrow to get formal approval for open sourcing it.
> >>>>
> >>>>
> >>>>
> >>>>
> >>> I'm interested - in my experience, the C++ compiler has been one
> of the
> >>> major barriers of Thrift adoption, since it's often a pain to
> get running
> >>> on
> >>> OSX, Windows, etc. A java code generator would be so much easier
> to get up
> >>> and going, easier to add "plugin" support, and easier for a larger
> >>> community
> >>> of developers to contribute to.
> >>>
> >>> -Todd
> >>>
> >>>
> >>
> >>
> >
> >
>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 26, 2010 at 19:47, Greg Stein <gs...@gmail.com> wrote:
>...
> And regarding my Python thing... that was more "yah. got one of those.
> not really offering it as a possibility."
Oh. Heh. I *did* check it into a branch. <self-pat-on-back/>
http://svn.apache.org/repos/asf/incubator/thrift/branches/py-compiler/
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
+1 for keeping C++. I still haven't heard what the compiler build and
packaging issues have been, other than boost, which can be removed with
a patch.
On 8/26/2010 4:52 PM, David Reiss wrote:
>> This has been
>> a hassle for httpd and svn... those two projects generally do *not*
>> provide binaries, and rely on downstream packagers. Of course, we
>> could also rely on packagers.
> That sounds fine to me.
>
>> I'm more comfortable with C++ than Java, but it seems easier to use
>> compiler subsystems (like ANTLR) and other tools in Java.
> My understanding was that ANTLR serves the same purpose as lex and yacc,
> which are already used by Thrift.
>
>> And back to
>> the deployment question regarding dependencies: will the packagers
>> have all the deps available? Compare that to shipping one .jar.
> The Thrift compiler doesn't have any runtime dependencies other than
> system libraries. I think the few compile-time dependencies it has
> can be eliminated far more easily than rewriting several thousand
> lines of code.
>
> --David
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> Is introducing a core dependency on having a java executable
> in a user's path really a critical issue nowadays?
As usual, I can only speak for myself, but it would be a reasonably
painful transition at Facebook.
> This little initiative sounds perfect for a branch/sandbox effort
> by interested committers. And if folks want to work on it,
> why say no?
I never said no. I also never said no when Greg started working
on his Python version last year. I'm just pointing out that if
Boost is the problem, there are simpler solutions that Java.
If people want to start committing code to trunk/compiler/java,
they can go nuts.
--David
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Joe Schaefer <jo...@yahoo.com>.
----- Original Message ----
> From: David Reiss <dr...@facebook.com>
> To: thrift-dev@incubator.apache.org
> Cc: Greg Stein <gs...@gmail.com>; Bryan Duxbury <br...@rapleaf.com>
> Sent: Thu, August 26, 2010 7:52:24 PM
> Subject: Re: Thrift compiler in Java (have parser, willing to hack)
>
> > This has been
> > a hassle for httpd and svn... those two projects generally do *not*
> > provide binaries, and rely on downstream packagers. Of course, we
> > could also rely on packagers.
> That sounds fine to me.
>
> > I'm more comfortable with C++ than Java, but it seems easier to use
> > compiler subsystems (like ANTLR) and other tools in Java.
> My understanding was that ANTLR serves the same purpose as lex and yacc,
> which are already used by Thrift.
>
> > And back to
> > the deployment question regarding dependencies: will the packagers
> > have all the deps available? Compare that to shipping one .jar.
> The Thrift compiler doesn't have any runtime dependencies other than
> system libraries. I think the few compile-time dependencies it has
> can be eliminated far more easily than rewriting several thousand
> lines of code.
Well someone other than you has offered to do it, so how easy the
task would be isn't all that relevant here. It's whether or not
changing to Java is in the best interests of the project, and
portability concerns alone are far more significant than the
relative ease with which the task can be accomplished.
Is introducing a core dependency on having a java executable
in a user's path really a critical issue nowadays?
This little initiative sounds perfect for a branch/sandbox effort
by interested committers. And if folks want to work on it,
why say no? You can always wait for the work to be finished
to compare it with the C++ version, and either collectively pick
the better option, or go with both.
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
I'll get rid of Boost. (From the code generator. Not the C++ runtime library.)
On 08/26/2010 05:05 PM, Greg Stein wrote:
> On Thu, Aug 26, 2010 at 19:52, David Reiss <dr...@facebook.com> wrote:
>>> This has been
>>> a hassle for httpd and svn... those two projects generally do *not*
>>> provide binaries, and rely on downstream packagers. Of course, we
>>> could also rely on packagers.
>>
>> That sounds fine to me.
>>
>>> I'm more comfortable with C++ than Java, but it seems easier to use
>>> compiler subsystems (like ANTLR) and other tools in Java.
>>
>> My understanding was that ANTLR serves the same purpose as lex and yacc,
>> which are already used by Thrift.
>>
>>> And back to
>>> the deployment question regarding dependencies: will the packagers
>>> have all the deps available? Compare that to shipping one .jar.
>>
>> The Thrift compiler doesn't have any runtime dependencies other than
>> system libraries. I think the few compile-time dependencies it has
>> can be eliminated far more easily than rewriting several thousand
>> lines of code.
>
> I tend to agree, but a Java implementation *is* intriguing.
>
> I would suggest that the Java-based compiler be checked into a new
> subdir for people to look at an experiment with. (I like to avoid a
> branch if experimental code can safely live compartmentalized on
> trunk) Then we can make a better-informed decision as a community on
> where to go.
>
> If somebody can say, "I'll get rid of Boost", then I'd be +1 on sticking to C++.
>
> Cheers,
> -g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by "Florentine, Justin F." <Ju...@espn.com>.
+1 for a java implementation. We'd love to distribute the thrift generator as a maven plugin that projects could simply
depend on and use during the "generate-sources" phase in the lifecycle.
Right now, that is technically possible, though cumbersome.
On Aug 26, 2010, at 8:05 PM, Greg Stein wrote:
> On Thu, Aug 26, 2010 at 19:52, David Reiss <dr...@facebook.com> wrote:
>>> This has been
>>> a hassle for httpd and svn... those two projects generally do *not*
>>> provide binaries, and rely on downstream packagers. Of course, we
>>> could also rely on packagers.
>>
>> That sounds fine to me.
>>
>>> I'm more comfortable with C++ than Java, but it seems easier to use
>>> compiler subsystems (like ANTLR) and other tools in Java.
>>
>> My understanding was that ANTLR serves the same purpose as lex and yacc,
>> which are already used by Thrift.
>>
>>> And back to
>>> the deployment question regarding dependencies: will the packagers
>>> have all the deps available? Compare that to shipping one .jar.
>>
>> The Thrift compiler doesn't have any runtime dependencies other than
>> system libraries. I think the few compile-time dependencies it has
>> can be eliminated far more easily than rewriting several thousand
>> lines of code.
>
> I tend to agree, but a Java implementation *is* intriguing.
>
> I would suggest that the Java-based compiler be checked into a new
> subdir for people to look at an experiment with. (I like to avoid a
> branch if experimental code can safely live compartmentalized on
> trunk) Then we can make a better-informed decision as a community on
> where to go.
>
> If somebody can say, "I'll get rid of Boost", then I'd be +1 on sticking to C++.
>
> Cheers,
> -g
>
> Please consider the environment before printing this e-mail.
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 26, 2010 at 19:52, David Reiss <dr...@facebook.com> wrote:
>> This has been
>> a hassle for httpd and svn... those two projects generally do *not*
>> provide binaries, and rely on downstream packagers. Of course, we
>> could also rely on packagers.
>
> That sounds fine to me.
>
>> I'm more comfortable with C++ than Java, but it seems easier to use
>> compiler subsystems (like ANTLR) and other tools in Java.
>
> My understanding was that ANTLR serves the same purpose as lex and yacc,
> which are already used by Thrift.
>
>> And back to
>> the deployment question regarding dependencies: will the packagers
>> have all the deps available? Compare that to shipping one .jar.
>
> The Thrift compiler doesn't have any runtime dependencies other than
> system libraries. I think the few compile-time dependencies it has
> can be eliminated far more easily than rewriting several thousand
> lines of code.
I tend to agree, but a Java implementation *is* intriguing.
I would suggest that the Java-based compiler be checked into a new
subdir for people to look at an experiment with. (I like to avoid a
branch if experimental code can safely live compartmentalized on
trunk) Then we can make a better-informed decision as a community on
where to go.
If somebody can say, "I'll get rid of Boost", then I'd be +1 on sticking to C++.
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> This has been
> a hassle for httpd and svn... those two projects generally do *not*
> provide binaries, and rely on downstream packagers. Of course, we
> could also rely on packagers.
That sounds fine to me.
> I'm more comfortable with C++ than Java, but it seems easier to use
> compiler subsystems (like ANTLR) and other tools in Java.
My understanding was that ANTLR serves the same purpose as lex and yacc,
which are already used by Thrift.
> And back to
> the deployment question regarding dependencies: will the packagers
> have all the deps available? Compare that to shipping one .jar.
The Thrift compiler doesn't have any runtime dependencies other than
system libraries. I think the few compile-time dependencies it has
can be eliminated far more easily than rewriting several thousand
lines of code.
--David
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Greg Stein <gs...@gmail.com> writes:
>
> I'm more comfortable with C++ than Java, but it seems easier to use
> compiler subsystems (like ANTLR) and other tools in Java. And back to
> the deployment question regarding dependencies: will the packagers
> have all the deps available? Compare that to shipping one .jar.
you would typically want the build of a Java-based compiler to produce
two artifacts. one artifact containing just the compiler classes and
one artifact that is the "application jar" with all dependencies (such
as ANTLR) embedded so that you can offer people a single JAR that just
does what it says on the tin without the excitement and joy of the
easter-egg hunt of yore.
(most of my Maven projects produce two JAR files. one used by other
Maven projects that wish to resolve the transitive dependencies
themselves and one JAR-file that is the stand-alone command line
version with everything stuffed inside like a sausage.)
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 26, 2010 at 19:40, David Reiss <dr...@facebook.com> wrote:
>> Otherwise, I think we should take the path of least resistance and port to
>> Java
>
> Isn't the path of least resistance just continuing to use C++?
hahah... nope.
I'm no fan of Java, but I would agree that it is easier to
develop/maintain than C++ in the long run. Also, delivering a single
.jar is a better story than platform-specific binaries. This has been
a hassle for httpd and svn... those two projects generally do *not*
provide binaries, and rely on downstream packagers. Of course, we
could also rely on packagers.
I'm more comfortable with C++ than Java, but it seems easier to use
compiler subsystems (like ANTLR) and other tools in Java. And back to
the deployment question regarding dependencies: will the packagers
have all the deps available? Compare that to shipping one .jar.
And regarding my Python thing... that was more "yah. got one of those.
not really offering it as a possibility."
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
On 8/26/2010 4:40 PM, David Reiss wrote:
>> Otherwise, I think we should take the path of least resistance and port to
>> Java
> Isn't the path of least resistance just continuing to use C++?
I found generating Thrift C code from C++ to only be mildly painful, but
not to the point where I felt a compelling desire to re-write the
emitter in something else.
Other than getting boost, what other build and packaging issues have
come up that are difficult to solve? It can't be any more painful than
trying to build, say, Firefox 4 nightlies with WebGL.
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> Otherwise, I think we should take the path of least resistance and port to
> Java
Isn't the path of least resistance just continuing to use C++?
On 08/26/2010 04:33 PM, Bryan Duxbury wrote:
> I don't think that it's going to matter how *fast* the Thrift compiler is -
> it's going to be a trivial amount of time one way or another.
>
> However, I do feel strongly that we should use a language with a compiler to
> write our compiler, simply because I don't think there's any way that we're
> going to avoid having a bajillion lines of code, and all things considered,
> I'd rather have a bajillion lines of Java than Python, if for no other
> reason that for the refactoring tools.
>
> I don't want to get mired in quibbling about our favorite language. If
> someone has a really clear alternative that would be orders of magnitude
> more productive due to magical templating or something, then speak up.
> Otherwise, I think we should take the path of least resistance and port to
> Java - it's accessible, more capable, and C-like enough that we'll be able
> to copy/paste/fix rather than starting over from scratch.
>
> On Thu, Aug 26, 2010 at 4:28 PM, Greg Stein <gs...@gmail.com> wrote:
>
>> On Thu, Aug 26, 2010 at 19:17, Todd Lipcon <to...@cloudera.com> wrote:
>>> On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dr...@facebook.com>
>> wrote:
>>>
>>>> Roger, you misunderstood that patch. It just lets you run configure and
>>>> make
>>>> to build the code generator and java/python/etc. libraries without
>> having
>>>> to
>>>> build the C++ runtime library.
>>>>
>>>> Also, I think I have a patch lying around to make the code generator not
>>>> rely
>>>> on boost.
>>>>
>>>>
>>> Not relying on boost would be a great start! That's probably the biggest
>>> obstacle - autoconf, etc, is annoying but boost is just the worst.
>>
>> Agreed. I *loathe* Boost.
>>
>> When I first went to build Thrift, I downloaded Boost and
>> built/installed it. I didn't know the magic incantations to limit the
>> thing, but figured "how bad can it be?"
>>
>> Well, I learned as soon as it said something about dealing 6000 header
>> files. Seriously. My include/boost-1_37/boost directory contains 6639
>> header files. That's just fucked.
>>
>> Regarding switch to Java? Meh. I've never found Java-based command
>> line tools to be all that useful or performant. They always seem to
>> have very bizarre idiosyncrasies.
>>
>> That said, I have most of a Python-based Thrift compiler sitting around.
>> :-P
>>
>> Cheers,
>> -g
>>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
I don't think that it's going to matter how *fast* the Thrift compiler is -
it's going to be a trivial amount of time one way or another.
However, I do feel strongly that we should use a language with a compiler to
write our compiler, simply because I don't think there's any way that we're
going to avoid having a bajillion lines of code, and all things considered,
I'd rather have a bajillion lines of Java than Python, if for no other
reason that for the refactoring tools.
I don't want to get mired in quibbling about our favorite language. If
someone has a really clear alternative that would be orders of magnitude
more productive due to magical templating or something, then speak up.
Otherwise, I think we should take the path of least resistance and port to
Java - it's accessible, more capable, and C-like enough that we'll be able
to copy/paste/fix rather than starting over from scratch.
On Thu, Aug 26, 2010 at 4:28 PM, Greg Stein <gs...@gmail.com> wrote:
> On Thu, Aug 26, 2010 at 19:17, Todd Lipcon <to...@cloudera.com> wrote:
> > On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dr...@facebook.com>
> wrote:
> >
> >> Roger, you misunderstood that patch. It just lets you run configure and
> >> make
> >> to build the code generator and java/python/etc. libraries without
> having
> >> to
> >> build the C++ runtime library.
> >>
> >> Also, I think I have a patch lying around to make the code generator not
> >> rely
> >> on boost.
> >>
> >>
> > Not relying on boost would be a great start! That's probably the biggest
> > obstacle - autoconf, etc, is annoying but boost is just the worst.
>
> Agreed. I *loathe* Boost.
>
> When I first went to build Thrift, I downloaded Boost and
> built/installed it. I didn't know the magic incantations to limit the
> thing, but figured "how bad can it be?"
>
> Well, I learned as soon as it said something about dealing 6000 header
> files. Seriously. My include/boost-1_37/boost directory contains 6639
> header files. That's just fucked.
>
> Regarding switch to Java? Meh. I've never found Java-based command
> line tools to be all that useful or performant. They always seem to
> have very bizarre idiosyncrasies.
>
> That said, I have most of a Python-based Thrift compiler sitting around.
> :-P
>
> Cheers,
> -g
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 5:10 PM, Roger Meier <ro...@bufferoverflow.ch>wrote:
> fully agree! hudson is a super great piece of software. => easy to use is
> key!
>
> however, another question is time.
> - how long does it take to get a java compiler up and running with all the
> featuresand tests?
> - what about these patches from David? => have a solution very quickly and
> simple
>
> please remember C and C++ is avaiable on any platform and the compiler can
> be compiled without dependencies.
>
> So what's the compilers mission?
> - just create the files
> - do things like hudson does
>
In my opinion the most important end goal is simple: my coworker who has
stock OSX with no special junk installed beyond the "Developer Tools" can
download a tarball and get thrift generating code in their favorite
language, without having to go and install anything else first.
If we can do that by killing the boost dependency, that's great.
Converting to Java gives us that plus a bit more - easier development, no
compilation step at all, easier integration in Java-based builds. But it's a
lot of work and no one has volunteered to get us there. Like David said,
it's thousands of lines of code.
>
> Am 27.08.2010 01:57, schrieb Bjorn Borud:
>
> Todd Lipcon<to...@cloudera.com> writes:
>>
>>
>>> Not to say I don't like coding in python plenty, but Java at least has a
>>> pretty sane deployment story.
>>>
>>>
>> my favorite is Hudson. *that* is how Java applications are assembled
>> and distributed properly.
>>
>> (for those who haven't set up Hudson: you download a hudson.war file.
>> then you can decide if you are in a particularly masochistic mood and
>> set up a Java web server and wrestle it into accepting the WAR file OR
>> you can just run it from the command line with:
>>
>> java -jar hudson.war
>>
>> and it just does what it is supposed to with no dangly bits (external
>> dependencies) hanging off to the side. beautiful)
>>
>> -Bjørn
>>
>>
>>
>>
>>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Roger Meier <ro...@bufferoverflow.ch>.
fully agree! hudson is a super great piece of software. => easy to use
is key!
however, another question is time.
- how long does it take to get a java compiler up and running with all
the featuresand tests?
- what about these patches from David? => have a solution very quickly
and simple
please remember C and C++ is avaiable on any platform and the compiler
can be compiled without dependencies.
So what's the compilers mission?
- just create the files
- do things like hudson does
Am 27.08.2010 01:57, schrieb Bjorn Borud:
> Todd Lipcon<to...@cloudera.com> writes:
>
>> Not to say I don't like coding in python plenty, but Java at least has a
>> pretty sane deployment story.
>>
> my favorite is Hudson. *that* is how Java applications are assembled
> and distributed properly.
>
> (for those who haven't set up Hudson: you download a hudson.war file.
> then you can decide if you are in a particularly masochistic mood and
> set up a Java web server and wrestle it into accepting the WAR file OR
> you can just run it from the command line with:
>
> java -jar hudson.war
>
> and it just does what it is supposed to with no dangly bits (external
> dependencies) hanging off to the side. beautiful)
>
> -Bjørn
>
>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Todd Lipcon <to...@cloudera.com> writes:
>
> Not to say I don't like coding in python plenty, but Java at least has a
> pretty sane deployment story.
my favorite is Hudson. *that* is how Java applications are assembled
and distributed properly.
(for those who haven't set up Hudson: you download a hudson.war file.
then you can decide if you are in a particularly masochistic mood and
set up a Java web server and wrestle it into accepting the WAR file OR
you can just run it from the command line with:
java -jar hudson.war
and it just does what it is supposed to with no dangly bits (external
dependencies) hanging off to the side. beautiful)
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 4:28 PM, Greg Stein <gs...@gmail.com> wrote:
> On Thu, Aug 26, 2010 at 19:17, Todd Lipcon <to...@cloudera.com> wrote:
> > On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dr...@facebook.com>
> wrote:
> >
> >> Roger, you misunderstood that patch. It just lets you run configure and
> >> make
> >> to build the code generator and java/python/etc. libraries without
> having
> >> to
> >> build the C++ runtime library.
> >>
> >> Also, I think I have a patch lying around to make the code generator not
> >> rely
> >> on boost.
> >>
> >>
> > Not relying on boost would be a great start! That's probably the biggest
> > obstacle - autoconf, etc, is annoying but boost is just the worst.
>
> Agreed. I *loathe* Boost.
>
> When I first went to build Thrift, I downloaded Boost and
> built/installed it. I didn't know the magic incantations to limit the
> thing, but figured "how bad can it be?"
>
> Well, I learned as soon as it said something about dealing 6000 header
> files. Seriously. My include/boost-1_37/boost directory contains 6639
> header files. That's just fucked.
>
> Regarding switch to Java? Meh. I've never found Java-based command
> line tools to be all that useful or performant. They always seem to
> have very bizarre idiosyncrasies.
>
> That said, I have most of a Python-based Thrift compiler sitting around.
> :-P
>
Python has its own set of idiosyncrasies - if you start depending on outside
modules, you either require users to go through lots of hoops to create
virtualenvironments, or you make them easy_install stuff system-wide,
usually causing hernias for sysadmins :)
Not to say I don't like coding in python plenty, but Java at least has a
pretty sane deployment story.
-Todd
>
> Cheers,
> -g
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Greg Stein <gs...@gmail.com> writes:
>
> Regarding switch to Java? Meh. I've never found Java-based command
> line tools to be all that useful or performant. They always seem to
> have very bizarre idiosyncrasies.
the startup time I can live with. :)
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 26, 2010 at 19:17, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dr...@facebook.com> wrote:
>
>> Roger, you misunderstood that patch. It just lets you run configure and
>> make
>> to build the code generator and java/python/etc. libraries without having
>> to
>> build the C++ runtime library.
>>
>> Also, I think I have a patch lying around to make the code generator not
>> rely
>> on boost.
>>
>>
> Not relying on boost would be a great start! That's probably the biggest
> obstacle - autoconf, etc, is annoying but boost is just the worst.
Agreed. I *loathe* Boost.
When I first went to build Thrift, I downloaded Boost and
built/installed it. I didn't know the magic incantations to limit the
thing, but figured "how bad can it be?"
Well, I learned as soon as it said something about dealing 6000 header
files. Seriously. My include/boost-1_37/boost directory contains 6639
header files. That's just fucked.
Regarding switch to Java? Meh. I've never found Java-based command
line tools to be all that useful or performant. They always seem to
have very bizarre idiosyncrasies.
That said, I have most of a Python-based Thrift compiler sitting around. :-P
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 4:12 PM, David Reiss <dr...@facebook.com> wrote:
> Roger, you misunderstood that patch. It just lets you run configure and
> make
> to build the code generator and java/python/etc. libraries without having
> to
> build the C++ runtime library.
>
> Also, I think I have a patch lying around to make the code generator not
> rely
> on boost.
>
>
Not relying on boost would be a great start! That's probably the biggest
obstacle - autoconf, etc, is annoying but boost is just the worst.
-Todd
> On 08/26/2010 04:03 PM, Todd Lipcon wrote:
> > On Thu, Aug 26, 2010 at 3:58 PM, Roger Meier <roger@bufferoverflow.ch
> >wrote:
> >
> >> Yes, that's an important issue!
> >>
> >> what do you think about a compiler that only needs C as described in
> >> https://issues.apache.org/jira/browse/THRIFT-506
> >>
> >> compile that one for different platforms and make it available for each
> >> version on the download web site?
> >>
> >> That gets rid of the boost dependency, but we're still forced to push
> out
> > built binaries for a bunch of platforms, which is kind of a pain compared
> to
> > Java.
> >
> > However, porting over all our existing generators will take some time,
> for
> > sure.
> >
> > -Todd
> >
> >>
> >> Am 26.08.2010 23:56, schrieb Todd Lipcon:
> >>
> >> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com> wrote:
> >>>
> >>>
> >>>
> >>>> on the project I currently work on we have a (ANTLR-based) parser for
> >>>> the Thrift IDL language in order to generate code for a proprietary
> >>>> serialization library.
> >>>>
> >>>> it struck me that perhaps we could use this parser the implement the
> >>>> Thrift compiler in Java instead. this would mean that the thrift
> >>>> compiler itself could be built as a platform independent artifact --
> >>>> which should make it a lot more elegant to write Maven plugins for
> >>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
> >>>> compiler binaries for all platforms and versions of the compiler.
> >>>>
> >>>> currently the parser lacks some minor features, but this could easily
> be
> >>>> rectified. the real job is to add the code generation for various
> >>>> languages.
> >>>>
> >>>> if anyone is interested in this, I am going to talk to some people
> >>>> tomorrow to get formal approval for open sourcing it.
> >>>>
> >>>>
> >>>>
> >>>>
> >>> I'm interested - in my experience, the C++ compiler has been one of the
> >>> major barriers of Thrift adoption, since it's often a pain to get
> running
> >>> on
> >>> OSX, Windows, etc. A java code generator would be so much easier to get
> up
> >>> and going, easier to add "plugin" support, and easier for a larger
> >>> community
> >>> of developers to contribute to.
> >>>
> >>> -Todd
> >>>
> >>>
> >>
> >>
> >
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
See! I told you guys I had patches for it!
It still requires a C++ compiler, though. Not just C.
Other committers, please don't check this in yet.
I need to update it along with my other build diffs.
--David
On 08/26/2010 05:45 PM, Roger Meier wrote:
> I took the wrong issue on that...but I did know you had a patch on for
> that topic;-)
>
> 506 *+1* => https://issues.apache.org/jira/browse/THRIFT-507
>
>
> my vote on 507 is *+1*
>
>
> Am 27.08.2010 01:12, schrieb David Reiss:
>> Roger, you misunderstood that patch. It just lets you run configure
>> and make
>> to build the code generator and java/python/etc. libraries without
>> having to
>> build the C++ runtime library.
>>
>> Also, I think I have a patch lying around to make the code generator
>> not rely
>> on boost.
>>
>> --David
>>
>> On 08/26/2010 04:03 PM, Todd Lipcon wrote:
>>
>>> On Thu, Aug 26, 2010 at 3:58 PM, Roger
>>> Meier<ro...@bufferoverflow.ch>wrote:
>>>
>>>
>>>> Yes, that's an important issue!
>>>>
>>>> what do you think about a compiler that only needs C as described in
>>>> https://issues.apache.org/jira/browse/THRIFT-506
>>>>
>>>> compile that one for different platforms and make it available for each
>>>> version on the download web site?
>>>>
>>>> That gets rid of the boost dependency, but we're still forced to
>>>> push out
>>>>
>>> built binaries for a bunch of platforms, which is kind of a pain
>>> compared to
>>> Java.
>>>
>>> However, porting over all our existing generators will take some
>>> time, for
>>> sure.
>>>
>>> -Todd
>>>
>>>
>>>> Am 26.08.2010 23:56, schrieb Todd Lipcon:
>>>>
>>>> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> on the project I currently work on we have a (ANTLR-based) parser for
>>>>>> the Thrift IDL language in order to generate code for a proprietary
>>>>>> serialization library.
>>>>>>
>>>>>> it struck me that perhaps we could use this parser the implement the
>>>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>>>> compiler itself could be built as a platform independent artifact --
>>>>>> which should make it a lot more elegant to write Maven plugins for
>>>>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>>>>> compiler binaries for all platforms and versions of the compiler.
>>>>>>
>>>>>> currently the parser lacks some minor features, but this could
>>>>>> easily be
>>>>>> rectified. the real job is to add the code generation for various
>>>>>> languages.
>>>>>>
>>>>>> if anyone is interested in this, I am going to talk to some people
>>>>>> tomorrow to get formal approval for open sourcing it.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> I'm interested - in my experience, the C++ compiler has been one of
>>>>> the
>>>>> major barriers of Thrift adoption, since it's often a pain to get
>>>>> running
>>>>> on
>>>>> OSX, Windows, etc. A java code generator would be so much easier to
>>>>> get up
>>>>> and going, easier to add "plugin" support, and easier for a larger
>>>>> community
>>>>> of developers to contribute to.
>>>>>
>>>>> -Todd
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Roger Meier <ro...@bufferoverflow.ch>.
I took the wrong issue on that...but I did know you had a patch on for
that topic;-)
506 *+1* => https://issues.apache.org/jira/browse/THRIFT-507
my vote on 507 is *+1*
Am 27.08.2010 01:12, schrieb David Reiss:
> Roger, you misunderstood that patch. It just lets you run configure and make
> to build the code generator and java/python/etc. libraries without having to
> build the C++ runtime library.
>
> Also, I think I have a patch lying around to make the code generator not rely
> on boost.
>
> --David
>
> On 08/26/2010 04:03 PM, Todd Lipcon wrote:
>
>> On Thu, Aug 26, 2010 at 3:58 PM, Roger Meier<ro...@bufferoverflow.ch>wrote:
>>
>>
>>> Yes, that's an important issue!
>>>
>>> what do you think about a compiler that only needs C as described in
>>> https://issues.apache.org/jira/browse/THRIFT-506
>>>
>>> compile that one for different platforms and make it available for each
>>> version on the download web site?
>>>
>>> That gets rid of the boost dependency, but we're still forced to push out
>>>
>> built binaries for a bunch of platforms, which is kind of a pain compared to
>> Java.
>>
>> However, porting over all our existing generators will take some time, for
>> sure.
>>
>> -Todd
>>
>>
>>> Am 26.08.2010 23:56, schrieb Todd Lipcon:
>>>
>>> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>>> on the project I currently work on we have a (ANTLR-based) parser for
>>>>> the Thrift IDL language in order to generate code for a proprietary
>>>>> serialization library.
>>>>>
>>>>> it struck me that perhaps we could use this parser the implement the
>>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>>> compiler itself could be built as a platform independent artifact --
>>>>> which should make it a lot more elegant to write Maven plugins for
>>>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>>>> compiler binaries for all platforms and versions of the compiler.
>>>>>
>>>>> currently the parser lacks some minor features, but this could easily be
>>>>> rectified. the real job is to add the code generation for various
>>>>> languages.
>>>>>
>>>>> if anyone is interested in this, I am going to talk to some people
>>>>> tomorrow to get formal approval for open sourcing it.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> I'm interested - in my experience, the C++ compiler has been one of the
>>>> major barriers of Thrift adoption, since it's often a pain to get running
>>>> on
>>>> OSX, Windows, etc. A java code generator would be so much easier to get up
>>>> and going, easier to add "plugin" support, and easier for a larger
>>>> community
>>>> of developers to contribute to.
>>>>
>>>> -Todd
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
Roger, you misunderstood that patch. It just lets you run configure and make
to build the code generator and java/python/etc. libraries without having to
build the C++ runtime library.
Also, I think I have a patch lying around to make the code generator not rely
on boost.
--David
On 08/26/2010 04:03 PM, Todd Lipcon wrote:
> On Thu, Aug 26, 2010 at 3:58 PM, Roger Meier <ro...@bufferoverflow.ch>wrote:
>
>> Yes, that's an important issue!
>>
>> what do you think about a compiler that only needs C as described in
>> https://issues.apache.org/jira/browse/THRIFT-506
>>
>> compile that one for different platforms and make it available for each
>> version on the download web site?
>>
>> That gets rid of the boost dependency, but we're still forced to push out
> built binaries for a bunch of platforms, which is kind of a pain compared to
> Java.
>
> However, porting over all our existing generators will take some time, for
> sure.
>
> -Todd
>
>>
>> Am 26.08.2010 23:56, schrieb Todd Lipcon:
>>
>> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com> wrote:
>>>
>>>
>>>
>>>> on the project I currently work on we have a (ANTLR-based) parser for
>>>> the Thrift IDL language in order to generate code for a proprietary
>>>> serialization library.
>>>>
>>>> it struck me that perhaps we could use this parser the implement the
>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>> compiler itself could be built as a platform independent artifact --
>>>> which should make it a lot more elegant to write Maven plugins for
>>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>>> compiler binaries for all platforms and versions of the compiler.
>>>>
>>>> currently the parser lacks some minor features, but this could easily be
>>>> rectified. the real job is to add the code generation for various
>>>> languages.
>>>>
>>>> if anyone is interested in this, I am going to talk to some people
>>>> tomorrow to get formal approval for open sourcing it.
>>>>
>>>>
>>>>
>>>>
>>> I'm interested - in my experience, the C++ compiler has been one of the
>>> major barriers of Thrift adoption, since it's often a pain to get running
>>> on
>>> OSX, Windows, etc. A java code generator would be so much easier to get up
>>> and going, easier to add "plugin" support, and easier for a larger
>>> community
>>> of developers to contribute to.
>>>
>>> -Todd
>>>
>>>
>>
>>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 3:58 PM, Roger Meier <ro...@bufferoverflow.ch>wrote:
> Yes, that's an important issue!
>
> what do you think about a compiler that only needs C as described in
> https://issues.apache.org/jira/browse/THRIFT-506
>
> compile that one for different platforms and make it available for each
> version on the download web site?
>
> That gets rid of the boost dependency, but we're still forced to push out
built binaries for a bunch of platforms, which is kind of a pain compared to
Java.
However, porting over all our existing generators will take some time, for
sure.
-Todd
>
> Am 26.08.2010 23:56, schrieb Todd Lipcon:
>
> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com> wrote:
>>
>>
>>
>>> on the project I currently work on we have a (ANTLR-based) parser for
>>> the Thrift IDL language in order to generate code for a proprietary
>>> serialization library.
>>>
>>> it struck me that perhaps we could use this parser the implement the
>>> Thrift compiler in Java instead. this would mean that the thrift
>>> compiler itself could be built as a platform independent artifact --
>>> which should make it a lot more elegant to write Maven plugins for
>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>> compiler binaries for all platforms and versions of the compiler.
>>>
>>> currently the parser lacks some minor features, but this could easily be
>>> rectified. the real job is to add the code generation for various
>>> languages.
>>>
>>> if anyone is interested in this, I am going to talk to some people
>>> tomorrow to get formal approval for open sourcing it.
>>>
>>>
>>>
>>>
>> I'm interested - in my experience, the C++ compiler has been one of the
>> major barriers of Thrift adoption, since it's often a pain to get running
>> on
>> OSX, Windows, etc. A java code generator would be so much easier to get up
>> and going, easier to add "plugin" support, and easier for a larger
>> community
>> of developers to contribute to.
>>
>> -Todd
>>
>>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Mark Slee <ms...@facebook.com> writes:
>
> I'll provide the historical backstory here in case folks are
> interested. C++ was actually chosen *because* of the early commitment
> to open source. C++ was by far the best available option for ease of
> distribution and installation. Using Python would have required
> everyone to have the right Python extensions, like PLY, which are not
> part of the core language.
Well, Python does have a parser built in. That parses Python.
Granted, if you give programmers enough rope they will hang themselves,
but there are examples of (for instance) build systems at (ahem) certain
companies that use Python to define data structures that can later,
effectively be parsed using a parser that (at that time) had been 15
years in the making.
what I am getting at is that I think I would probably have tried to come
up with a way to describe structs, enums, services etc in terms of
Python data structures and avoid having to write a new parser at all.
(I guess DSLs are probably a slightly more recent fad among the
conference-going-book-writing hipster crowd :-)
> By contrast, lex/yacc are *completely* standard. You'll be
> hard-pressed to find a *NIX distro anywhere on Earth that doesn't have
> working versions out of the box, they've been around forever and their
> interfaces are super stable. Like the autotools, even if they're not
> the prettiest things to work with, everyone has them lying around, and
> they almost always just work.
well, almost. which is why you usually end up on the FAQ page when
trying to build on Mac. or on a system that has a different idea of
"recent".
> If our goal is ease-of-installation-and-use
> (optimizing for the Thrift user), then my vote is to work on removing
> boost as a dependency to build the compiler. If the goal is elegance
> and maintainability of the compiler code (optimizing for the Thrift
> developer, which theoretically benefits the user indirectly), then I
> agree that something like Java is probably better.
>
> Anyways, that's the story on why the Thrift compiler was written in
> C++.
I can understand where you are coming from. I spent about 15 years
primarily hacking C/C++ on UNIX before switching to Java as my primary
language around 2003 or so. Before 2002 I would probably have written a
tool like the thrift compiler in C -- and as far as possible try not to
depend on anything but an ANSI C compiler. in a moment of weakness I
might have done it in Perl (which would have been truly awful). 6
months ago I would probably have (ab)used Python to define a DSL and
avoid having to write an actual parser.
perspectives change over time.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> oh, and we have indeed built in on Windows on a few occasions, but it
> did require some fiddling around. the point was more that "it works for
> me" is not a very interesting data point. "I did *this* to make it
> work" is.
Maybe you can document your "fiddling around"?
--David
On 09/01/2010 02:49 AM, Bjorn Borud wrote:
> Michael Lum <mi...@openx.org> writes:
>>
>> The point of saying that it compiles fine for me is to make the
>> statement that it *is* possible to compile it on Windows, not to say
>> "hey, it works for me, what's wrong with you?"
>
> oh, and we have indeed built in on Windows on a few occasions, but it
> did require some fiddling around. the point was more that "it works for
> me" is not a very interesting data point. "I did *this* to make it
> work" is.
>
> (we had issues on OSX as well, but a bit of Googling and a quick copy of
> a file solved that problem. also not ideal, but more managable)
>
>> Since it is possible, then it can be documented, and then scripted,
>> and then we can patch it until it works for everyone.
>
> agreed.
>
> -Bjørn
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Michael Lum <mi...@openx.org> writes:
>
> The point of saying that it compiles fine for me is to make the
> statement that it *is* possible to compile it on Windows, not to say
> "hey, it works for me, what's wrong with you?"
oh, and we have indeed built in on Windows on a few occasions, but it
did require some fiddling around. the point was more that "it works for
me" is not a very interesting data point. "I did *this* to make it
work" is.
(we had issues on OSX as well, but a bit of Googling and a quick copy of
a file solved that problem. also not ideal, but more managable)
> Since it is possible, then it can be documented, and then scripted,
> and then we can patch it until it works for everyone.
agreed.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
Well, if we get the Windows build process documented and scripted, then
it should be trivial to publish the binaries for all Thrift versions
going forward. David already got the cross-compiler working.
The point of saying that it compiles fine for me is to make the
statement that it *is* possible to compile it on Windows, not to say
"hey, it works for me, what's wrong with you?" Since it is possible,
then it can be documented, and then scripted, and then we can patch it
until it works for everyone.
On 8/30/2010 5:40 AM, Bjorn Borud wrote:
> Michael Lum<mi...@openx.org> writes:
>
>> The compiler builds fine for me on MinGW + MSYS on Windows.
>
> years ago I saw a talk given by one of the PHP core developers that had
> an important slide. The slide said: "nobody cares if it works for you".
>
> yes, many people manage just fine to build the Thrift compiler on
> windows, but for most people it *is* fiddly and it *is* an unwanted
> complexity. ignoring this fact is not the way to stimulate the success
> of a technology. (after all, just look at how successful PHP has been
> as a project, and not even Rasmus Lerdorf would claim that PHP is
> "beautiful" or "optimal" -- but it does what it says on the tin with
> a minimum of fuss. which is a key selling point).
>
> (for me there is an additional level of complexity: I don't use Windows
> and on my project, the people who do use Windows neither should nor
> particularly want to maintain the Thrift compiler binary for Windows)
>
>> Autotools, lex, and yacc are only needed to build Thrift from source,
>> not to generate code from Thrift IDLs.
>
> this would not be an issue if statically linked binaries for all
> platforms were published for each (snapshot) version of thrift. which
> is not the case (I'm not sure you'd want to). so it means that you
> *do* have to deal with this in a typical project.
>
>> When I get some time, and if it would help, I could build a .MSI that
>> has thrift.exe and some .dlls. Windows users could then generate
>> Thrift code without the need for Cygwin, MinGW, or anything.
>
> of which version? for this exercise to be meaningful you would have to
> offer all the versions that are used in various releases of projects
> like Cassandra.
>
>> This is not an anti-Java position (we use Java extensively here), I'm
>> just saying that so far all the reasons given so far for switching the
>> compiler to Java have been build and packaging related, not with the
>> C++ language itself.
>
> exactly, but as it turns out, building and packaging is important.
> especially since Thrift is neither in the official Maven repositories,
> nor are binaries provided for all the odd snapshot releases that are
> currently used in the wild.
>
> -Bjørn
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Michael Lum <mi...@openx.org> writes:
> The compiler builds fine for me on MinGW + MSYS on Windows.
years ago I saw a talk given by one of the PHP core developers that had
an important slide. The slide said: "nobody cares if it works for you".
yes, many people manage just fine to build the Thrift compiler on
windows, but for most people it *is* fiddly and it *is* an unwanted
complexity. ignoring this fact is not the way to stimulate the success
of a technology. (after all, just look at how successful PHP has been
as a project, and not even Rasmus Lerdorf would claim that PHP is
"beautiful" or "optimal" -- but it does what it says on the tin with
a minimum of fuss. which is a key selling point).
(for me there is an additional level of complexity: I don't use Windows
and on my project, the people who do use Windows neither should nor
particularly want to maintain the Thrift compiler binary for Windows)
> Autotools, lex, and yacc are only needed to build Thrift from source,
> not to generate code from Thrift IDLs.
this would not be an issue if statically linked binaries for all
platforms were published for each (snapshot) version of thrift. which
is not the case (I'm not sure you'd want to). so it means that you
*do* have to deal with this in a typical project.
> When I get some time, and if it would help, I could build a .MSI that
> has thrift.exe and some .dlls. Windows users could then generate
> Thrift code without the need for Cygwin, MinGW, or anything.
of which version? for this exercise to be meaningful you would have to
offer all the versions that are used in various releases of projects
like Cassandra.
> This is not an anti-Java position (we use Java extensively here), I'm
> just saying that so far all the reasons given so far for switching the
> compiler to Java have been build and packaging related, not with the
> C++ language itself.
exactly, but as it turns out, building and packaging is important.
especially since Thrift is neither in the official Maven repositories,
nor are binaries provided for all the odd snapshot releases that are
currently used in the wild.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
I'm neck-deep in a time-sensitive project at the moment, but have
created THRIFT-874 for myself when time frees up.
On 8/27/2010 10:13 AM, David Reiss wrote:
>> The compiler builds fine for me on MinGW + MSYS on Windows.
> If you could write up some step-by-step instructions, that would be great.
> We've had *lots* of people complaining that they can't figure out how to
> build it.
>
>> When I get some time, and if it would
>> help, I could build a .MSI that has thrift.exe and some .dlls.
> That would be awesome.
>
> --David
>
> On 08/27/2010 09:24 AM, Michael Lum wrote:
>> The compiler builds fine for me on MinGW + MSYS on Windows. Autotools,
>> lex, and yacc are only needed to build Thrift from source, not to
>> generate code from Thrift IDLs. When I get some time, and if it would
>> help, I could build a .MSI that has thrift.exe and some .dlls. Windows
>> users could then generate Thrift code without the need for Cygwin,
>> MinGW, or anything.
>>
>> This is not an anti-Java position (we use Java extensively here), I'm
>> just saying that so far all the reasons given so far for switching the
>> compiler to Java have been build and packaging related, not with the C++
>> language itself.
>>
>> On 8/27/2010 7:39 AM, Rush Manbert wrote:
>>>
>>> On Aug 26, 2010, at 5:15 PM, Mark Slee wrote:
>>>
>>>>>> I have to admit that when I first looked at the Thrift compiler
>>>>>> I was a bit puzzled by the choice of language and the
>>>>>> dependence on Boost. then I realized that they probably didn't
>>>>>> have open sourcing it in mind when they made that decision.
>>>>
>>>> I'll provide the historical backstory here in case folks are
>>>> interested. C++ was actually chosen *because* of the early
>>>> commitment to open source. C++ was by far the best available option
>>>> for ease of distribution and installation. Using Python would have
>>>> required everyone to have the right Python extensions, like PLY,
>>>> which are not part of the core language. This work started in 2006
>>>> -- Python has continued to gain popularity over the past 4-5 years,
>>>> not all developer were so on board with it back then. Same issues
>>>> with Java -- the core language does not have a parser built in.
>>>> Also, there are still a fair amount of people interested in Thrift
>>>> that fall in the religiously-anti-Java camp (I am not one of them,
>>>> I like Java, but I accept that for better or worse a lot of
>>>> systems-y people dislike it). Not all Linux distributions have
>>>> consistent Python/Java interpreters setup as part of vanilla
>>>> installs (notably, the images we were using at Facebook did not
>>>> include a Java runtime, as we generally had no need for it). And if
>>>> you don't have root, dealing with things like site-packages and
>>>> paths is a big headache. So, using those languages would have meant
>>>> putting a big hurdle in front of many users considering trying out
>>>> Thrift.
>>>>
>>>> By contrast, lex/yacc are *completely* standard. You'll be
>>>> hard-pressed to find a *NIX distro anywhere on Earth that doesn't
>>>> have working versions out of the box, they've been around forever
>>>> and their interfaces are super stable. Like the autotools, even if
>>>> they're not the prettiest things to work with, everyone has them
>>>> lying around, and they almost always just work. Choosing C++ meant
>>>> that for 99% of users, getting the compiler up and running (step
>>>> one in evaluating Thrift) was as simple as (1) download (2)
>>>> configure (3) make.
>>>>
>>>
>>> Just one small comment. It seems that most on this list don't really
>>> care about Windows compatibility. (Not a criticism. That's just the
>>> picture that I get.) The argument can be made that if you want to use
>>> Thrift on Windows, you just install Cygwin and have at it.
>>>
>>> But many people can't do that for various reasons, and lex and yacc
>>> are not available, nor are the autotools. I have managed to wrestle
>>> the entire C++ runtime library into a shape where it can be used in
>>> Windows without requiring any GNU tools installed. (Of course, I had
>>> to use more boost libraries to make that happen. :-) But I couldn't
>>> do anything about the Thrift compiler. For us, and others who develop
>>> on both Windows and *nix platforms, we can sidestep the compiler
>>> issue by generating all the code on the *nix system. But it seems to
>>> me that this is a rather large barrier to adoption in the Windows
>>> world.
>>>
>>> For us, the Thrift compiler performance is absolutely not an issue,
>>> and I don't think it would become one if the compiler were written in
>>> Java. As long as it Just Worked and only required us to install a
>>> standard Java distro, I'd be happy.
>>>
>>> So I'm +1 on a Java version of the compiler.
>>>
>>> - Rush
>>>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> The compiler builds fine for me on MinGW + MSYS on Windows.
If you could write up some step-by-step instructions, that would be great.
We've had *lots* of people complaining that they can't figure out how to
build it.
> When I get some time, and if it would
> help, I could build a .MSI that has thrift.exe and some .dlls.
That would be awesome.
--David
On 08/27/2010 09:24 AM, Michael Lum wrote:
> The compiler builds fine for me on MinGW + MSYS on Windows. Autotools,
> lex, and yacc are only needed to build Thrift from source, not to
> generate code from Thrift IDLs. When I get some time, and if it would
> help, I could build a .MSI that has thrift.exe and some .dlls. Windows
> users could then generate Thrift code without the need for Cygwin,
> MinGW, or anything.
>
> This is not an anti-Java position (we use Java extensively here), I'm
> just saying that so far all the reasons given so far for switching the
> compiler to Java have been build and packaging related, not with the C++
> language itself.
>
> On 8/27/2010 7:39 AM, Rush Manbert wrote:
>>
>> On Aug 26, 2010, at 5:15 PM, Mark Slee wrote:
>>
>>>>> I have to admit that when I first looked at the Thrift compiler
>>>>> I was a bit puzzled by the choice of language and the
>>>>> dependence on Boost. then I realized that they probably didn't
>>>>> have open sourcing it in mind when they made that decision.
>>>
>>> I'll provide the historical backstory here in case folks are
>>> interested. C++ was actually chosen *because* of the early
>>> commitment to open source. C++ was by far the best available option
>>> for ease of distribution and installation. Using Python would have
>>> required everyone to have the right Python extensions, like PLY,
>>> which are not part of the core language. This work started in 2006
>>> -- Python has continued to gain popularity over the past 4-5 years,
>>> not all developer were so on board with it back then. Same issues
>>> with Java -- the core language does not have a parser built in.
>>> Also, there are still a fair amount of people interested in Thrift
>>> that fall in the religiously-anti-Java camp (I am not one of them,
>>> I like Java, but I accept that for better or worse a lot of
>>> systems-y people dislike it). Not all Linux distributions have
>>> consistent Python/Java interpreters setup as part of vanilla
>>> installs (notably, the images we were using at Facebook did not
>>> include a Java runtime, as we generally had no need for it). And if
>>> you don't have root, dealing with things like site-packages and
>>> paths is a big headache. So, using those languages would have meant
>>> putting a big hurdle in front of many users considering trying out
>>> Thrift.
>>>
>>> By contrast, lex/yacc are *completely* standard. You'll be
>>> hard-pressed to find a *NIX distro anywhere on Earth that doesn't
>>> have working versions out of the box, they've been around forever
>>> and their interfaces are super stable. Like the autotools, even if
>>> they're not the prettiest things to work with, everyone has them
>>> lying around, and they almost always just work. Choosing C++ meant
>>> that for 99% of users, getting the compiler up and running (step
>>> one in evaluating Thrift) was as simple as (1) download (2)
>>> configure (3) make.
>>>
>>
>> Just one small comment. It seems that most on this list don't really
>> care about Windows compatibility. (Not a criticism. That's just the
>> picture that I get.) The argument can be made that if you want to use
>> Thrift on Windows, you just install Cygwin and have at it.
>>
>> But many people can't do that for various reasons, and lex and yacc
>> are not available, nor are the autotools. I have managed to wrestle
>> the entire C++ runtime library into a shape where it can be used in
>> Windows without requiring any GNU tools installed. (Of course, I had
>> to use more boost libraries to make that happen. :-) But I couldn't
>> do anything about the Thrift compiler. For us, and others who develop
>> on both Windows and *nix platforms, we can sidestep the compiler
>> issue by generating all the code on the *nix system. But it seems to
>> me that this is a rather large barrier to adoption in the Windows
>> world.
>>
>> For us, the Thrift compiler performance is absolutely not an issue,
>> and I don't think it would become one if the compiler were written in
>> Java. As long as it Just Worked and only required us to install a
>> standard Java distro, I'd be happy.
>>
>> So I'm +1 on a Java version of the compiler.
>>
>> - Rush
>>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
The compiler builds fine for me on MinGW + MSYS on Windows. Autotools,
lex, and yacc are only needed to build Thrift from source, not to
generate code from Thrift IDLs. When I get some time, and if it would
help, I could build a .MSI that has thrift.exe and some .dlls. Windows
users could then generate Thrift code without the need for Cygwin,
MinGW, or anything.
This is not an anti-Java position (we use Java extensively here), I'm
just saying that so far all the reasons given so far for switching the
compiler to Java have been build and packaging related, not with the C++
language itself.
On 8/27/2010 7:39 AM, Rush Manbert wrote:
>
> On Aug 26, 2010, at 5:15 PM, Mark Slee wrote:
>
>>>> I have to admit that when I first looked at the Thrift compiler
>>>> I was a bit puzzled by the choice of language and the
>>>> dependence on Boost. then I realized that they probably didn't
>>>> have open sourcing it in mind when they made that decision.
>>
>> I'll provide the historical backstory here in case folks are
>> interested. C++ was actually chosen *because* of the early
>> commitment to open source. C++ was by far the best available option
>> for ease of distribution and installation. Using Python would have
>> required everyone to have the right Python extensions, like PLY,
>> which are not part of the core language. This work started in 2006
>> -- Python has continued to gain popularity over the past 4-5 years,
>> not all developer were so on board with it back then. Same issues
>> with Java -- the core language does not have a parser built in.
>> Also, there are still a fair amount of people interested in Thrift
>> that fall in the religiously-anti-Java camp (I am not one of them,
>> I like Java, but I accept that for better or worse a lot of
>> systems-y people dislike it). Not all Linux distributions have
>> consistent Python/Java interpreters setup as part of vanilla
>> installs (notably, the images we were using at Facebook did not
>> include a Java runtime, as we generally had no need for it). And if
>> you don't have root, dealing with things like site-packages and
>> paths is a big headache. So, using those languages would have meant
>> putting a big hurdle in front of many users considering trying out
>> Thrift.
>>
>> By contrast, lex/yacc are *completely* standard. You'll be
>> hard-pressed to find a *NIX distro anywhere on Earth that doesn't
>> have working versions out of the box, they've been around forever
>> and their interfaces are super stable. Like the autotools, even if
>> they're not the prettiest things to work with, everyone has them
>> lying around, and they almost always just work. Choosing C++ meant
>> that for 99% of users, getting the compiler up and running (step
>> one in evaluating Thrift) was as simple as (1) download (2)
>> configure (3) make.
>>
>
> Just one small comment. It seems that most on this list don't really
> care about Windows compatibility. (Not a criticism. That's just the
> picture that I get.) The argument can be made that if you want to use
> Thrift on Windows, you just install Cygwin and have at it.
>
> But many people can't do that for various reasons, and lex and yacc
> are not available, nor are the autotools. I have managed to wrestle
> the entire C++ runtime library into a shape where it can be used in
> Windows without requiring any GNU tools installed. (Of course, I had
> to use more boost libraries to make that happen. :-) But I couldn't
> do anything about the Thrift compiler. For us, and others who develop
> on both Windows and *nix platforms, we can sidestep the compiler
> issue by generating all the code on the *nix system. But it seems to
> me that this is a rather large barrier to adoption in the Windows
> world.
>
> For us, the Thrift compiler performance is absolutely not an issue,
> and I don't think it would become one if the compiler were written in
> Java. As long as it Just Worked and only required us to install a
> standard Java distro, I'd be happy.
>
> So I'm +1 on a Java version of the compiler.
>
> - Rush
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Rush Manbert <ru...@manbert.com> writes:
>
> But many people can't do that for various reasons, and lex and yacc
> are not available, nor are the autotools. I have managed to wrestle
> the entire C++ runtime library into a shape where it can be used in
> Windows without requiring any GNU tools installed. (Of course, I had
> to use more boost libraries to make that happen. :-) But I couldn't do
> anything about the Thrift compiler. For us, and others who develop on
> both Windows and *nix platforms, we can sidestep the compiler issue by
> generating all the code on the *nix system. But it seems to me that
> this is a rather large barrier to adoption in the Windows world.
you are describing some of the same problems we have had.
> For us, the Thrift compiler performance is absolutely not an issue,
> and I don't think it would become one if the compiler were written in
> Java. As long as it Just Worked and only required us to install a
> standard Java distro, I'd be happy.
I concur, the compiler performance is a non-issue.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Rush Manbert <ru...@manbert.com>.
On Aug 26, 2010, at 5:15 PM, Mark Slee wrote:
>>> I have to admit that when I first looked at the Thrift compiler I was a
>>> bit puzzled by the choice of language and the dependence on Boost. then
>>> I realized that they probably didn't have open sourcing it in mind when
>>> they made that decision.
>
> I'll provide the historical backstory here in case folks are interested. C++ was actually chosen *because* of the early commitment to open source. C++ was by far the best available option for ease of distribution and installation. Using Python would have required everyone to have the right Python extensions, like PLY, which are not part of the core language. This work started in 2006 -- Python has continued to gain popularity over the past 4-5 years, not all developer were so on board with it back then. Same issues with Java -- the core language does not have a parser built in. Also, there are still a fair amount of people interested in Thrift that fall in the religiously-anti-Java camp (I am not one of them, I like Java, but I accept that for better or worse a lot of systems-y people dislike it). Not all Linux distributions have consistent Python/Java interpreters setup as part of vanilla installs (notably, the images we were using at Facebook did not include a Java runtime, as we generally had no need for it). And if you don't have root, dealing with things like site-packages and paths is a big headache. So, using those languages would have meant putting a big hurdle in front of many users considering trying out Thrift.
>
> By contrast, lex/yacc are *completely* standard. You'll be hard-pressed to find a *NIX distro anywhere on Earth that doesn't have working versions out of the box, they've been around forever and their interfaces are super stable. Like the autotools, even if they're not the prettiest things to work with, everyone has them lying around, and they almost always just work. Choosing C++ meant that for 99% of users, getting the compiler up and running (step one in evaluating Thrift) was as simple as (1) download (2) configure (3) make.
>
Just one small comment. It seems that most on this list don't really care about Windows compatibility. (Not a criticism. That's just the picture that I get.) The argument can be made that if you want to use Thrift on Windows, you just install Cygwin and have at it.
But many people can't do that for various reasons, and lex and yacc are not available, nor are the autotools. I have managed to wrestle the entire C++ runtime library into a shape where it can be used in Windows without requiring any GNU tools installed. (Of course, I had to use more boost libraries to make that happen. :-) But I couldn't do anything about the Thrift compiler. For us, and others who develop on both Windows and *nix platforms, we can sidestep the compiler issue by generating all the code on the *nix system. But it seems to me that this is a rather large barrier to adoption in the Windows world.
For us, the Thrift compiler performance is absolutely not an issue, and I don't think it would become one if the compiler were written in Java. As long as it Just Worked and only required us to install a standard Java distro, I'd be happy.
So I'm +1 on a Java version of the compiler.
- Rush
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Mathias Herberts <ma...@gmail.com>.
Thanks for this insight Mark!
In case some of the list readers did not notice, you don't need to
actually build boost to build the Thrift compiler, unarchiving it and
ln -s . include; ln -s . lib and specifying --with-boost=... is
sufficient to make configure and the build process happy.
Mathias.
RE: Thrift compiler in Java (have parser, willing to hack)
Posted by Mark Slee <ms...@facebook.com>.
>> I have to admit that when I first looked at the Thrift compiler I was a
>> bit puzzled by the choice of language and the dependence on Boost. then
>> I realized that they probably didn't have open sourcing it in mind when
>> they made that decision.
I'll provide the historical backstory here in case folks are interested. C++ was actually chosen *because* of the early commitment to open source. C++ was by far the best available option for ease of distribution and installation. Using Python would have required everyone to have the right Python extensions, like PLY, which are not part of the core language. This work started in 2006 -- Python has continued to gain popularity over the past 4-5 years, not all developer were so on board with it back then. Same issues with Java -- the core language does not have a parser built in. Also, there are still a fair amount of people interested in Thrift that fall in the religiously-anti-Java camp (I am not one of them, I like Java, but I accept that for better or worse a lot of systems-y people dislike it). Not all Linux distributions have consistent Python/Java interpreters setup as part of vanilla installs (notably, the images we were using at Facebook did not include a Java runtime, as we generally had no need for it). And if you don't have root, dealing with things like site-packages and paths is a big headache. So, using those languages would have meant putting a big hurdle in front of many users considering trying out Thrift.
By contrast, lex/yacc are *completely* standard. You'll be hard-pressed to find a *NIX distro anywhere on Earth that doesn't have working versions out of the box, they've been around forever and their interfaces are super stable. Like the autotools, even if they're not the prettiest things to work with, everyone has them lying around, and they almost always just work. Choosing C++ meant that for 99% of users, getting the compiler up and running (step one in evaluating Thrift) was as simple as (1) download (2) configure (3) make.
Boost was *not* originally a compiler dependency, and I agree it can be a real pain to work with (perhaps comparably to maintaining site-packages for Python, or adding JARs to your environment classpath, etc.). If our goal is ease-of-installation-and-use (optimizing for the Thrift user), then my vote is to work on removing boost as a dependency to build the compiler. If the goal is elegance and maintainability of the compiler code (optimizing for the Thrift developer, which theoretically benefits the user indirectly), then I agree that something like Java is probably better.
Anyways, that's the story on why the Thrift compiler was written in C++.
Cheers,
mcslee
-----Original Message-----
From: Bjorn Borud [mailto:bborud@gmail.com]
Sent: Thursday, August 26, 2010 4:46 PM
To: thrift-dev@incubator.apache.org
Subject: Re: Thrift compiler in Java (have parser, willing to hack)
Roger Meier <ro...@bufferoverflow.ch> writes:
> Yes, that's an important issue!
>
> what do you think about a compiler that only needs C as described in
> https://issues.apache.org/jira/browse/THRIFT-506
I have to admit that when I first looked at the Thrift compiler I was a
bit puzzled by the choice of language and the dependence on Boost. then
I realized that they probably didn't have open sourcing it in mind when
they made that decision. (a lot of much more trivial code I have written
at various companies in the past have had far more pointless constraints
and dependencies due to, for instance, in-house standard libraries that
"everyone" must use).
choosing a language that produces platform specific binaries for a tool
that is, by nature, cross language and cross platform, is unfortunate.
even more so when we are talking about a moving target. (meaning, you
have to expect that people need to upgrade the compiler relatively often
until Thrift settles somewhat).
(I gave this some thought a few months ago and my conclusion then was
that Python would probably have been the best choice (mature enough,
understandable enough for "non-native speakers", present on almost all
platforms, no platform specific binaries to distribute,
"unoffensiveness" etc).
Note that I am no Python developer -- I just thought that for this type
of tool Python would probably be easier to sell to some majority of
developers across the language spectrum. And no, I didn't ask a
representative cross section of programmers for their opinion so it is
entirely within the realm of the possible that I am dead wrong :-)).
> compile that one for different platforms and make it available for
> each version on the download web site?
I've spent years of my life developing software in C/C++ that needs to
compile to lots and lots of OS/hw targets and I miss both the
development process and the release building about as much as I miss
having to hear Michael Bolton's music in public spaces.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Roger Meier <ro...@bufferoverflow.ch> writes:
> Yes, that's an important issue!
>
> what do you think about a compiler that only needs C as described in
> https://issues.apache.org/jira/browse/THRIFT-506
I have to admit that when I first looked at the Thrift compiler I was a
bit puzzled by the choice of language and the dependence on Boost. then
I realized that they probably didn't have open sourcing it in mind when
they made that decision. (a lot of much more trivial code I have written
at various companies in the past have had far more pointless constraints
and dependencies due to, for instance, in-house standard libraries that
"everyone" must use).
choosing a language that produces platform specific binaries for a tool
that is, by nature, cross language and cross platform, is unfortunate.
even more so when we are talking about a moving target. (meaning, you
have to expect that people need to upgrade the compiler relatively often
until Thrift settles somewhat).
(I gave this some thought a few months ago and my conclusion then was
that Python would probably have been the best choice (mature enough,
understandable enough for "non-native speakers", present on almost all
platforms, no platform specific binaries to distribute,
"unoffensiveness" etc).
Note that I am no Python developer -- I just thought that for this type
of tool Python would probably be easier to sell to some majority of
developers across the language spectrum. And no, I didn't ask a
representative cross section of programmers for their opinion so it is
entirely within the realm of the possible that I am dead wrong :-)).
> compile that one for different platforms and make it available for
> each version on the download web site?
I've spent years of my life developing software in C/C++ that needs to
compile to lots and lots of OS/hw targets and I miss both the
development process and the release building about as much as I miss
having to hear Michael Bolton's music in public spaces.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Yes, that's an important issue!
what do you think about a compiler that only needs C as described in
https://issues.apache.org/jira/browse/THRIFT-506
compile that one for different platforms and make it available for each
version on the download web site?
Am 26.08.2010 23:56, schrieb Todd Lipcon:
> On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud<bb...@gmail.com> wrote:
>
>
>> on the project I currently work on we have a (ANTLR-based) parser for
>> the Thrift IDL language in order to generate code for a proprietary
>> serialization library.
>>
>> it struck me that perhaps we could use this parser the implement the
>> Thrift compiler in Java instead. this would mean that the thrift
>> compiler itself could be built as a platform independent artifact --
>> which should make it a lot more elegant to write Maven plugins for
>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>> compiler binaries for all platforms and versions of the compiler.
>>
>> currently the parser lacks some minor features, but this could easily be
>> rectified. the real job is to add the code generation for various
>> languages.
>>
>> if anyone is interested in this, I am going to talk to some people
>> tomorrow to get formal approval for open sourcing it.
>>
>>
>>
> I'm interested - in my experience, the C++ compiler has been one of the
> major barriers of Thrift adoption, since it's often a pain to get running on
> OSX, Windows, etc. A java code generator would be so much easier to get up
> and going, easier to add "plugin" support, and easier for a larger community
> of developers to contribute to.
>
> -Todd
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Todd Lipcon <to...@cloudera.com> writes:
>
> I'm interested - in my experience, the C++ compiler has been one of the
> major barriers of Thrift adoption, since it's often a pain to get running on
> OSX, Windows, etc. A java code generator would be so much easier to get up
> and going, easier to add "plugin" support, and easier for a larger community
> of developers to contribute to.
cool. I'll clear it with the right people first thing tomorrow morning
and see if we can get this open sourced ASAP. (since we have a release
just around the corner we might not be able to devote a lot of attention
to the project right *now*, but I have suggested we make it a priority
as soon as we are able. at the very least we'll be able to get someone
to review code and manage the project).
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud <bb...@gmail.com> wrote:
>
> on the project I currently work on we have a (ANTLR-based) parser for
> the Thrift IDL language in order to generate code for a proprietary
> serialization library.
>
> it struck me that perhaps we could use this parser the implement the
> Thrift compiler in Java instead. this would mean that the thrift
> compiler itself could be built as a platform independent artifact --
> which should make it a lot more elegant to write Maven plugins for
> Thrift. it would also eliminate the need (for us) to maintain Thrift
> compiler binaries for all platforms and versions of the compiler.
>
> currently the parser lacks some minor features, but this could easily be
> rectified. the real job is to add the code generation for various
> languages.
>
> if anyone is interested in this, I am going to talk to some people
> tomorrow to get formal approval for open sourcing it.
>
>
I'm interested - in my experience, the C++ compiler has been one of the
major barriers of Thrift adoption, since it's often a pain to get running on
OSX, Windows, etc. A java code generator would be so much easier to get up
and going, easier to add "plugin" support, and easier for a larger community
of developers to contribute to.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Greg Stein <gs...@gmail.com> writes:
>>
>> I agree.
>
> I also agree.
the TDD crowd rejoices I am sure. :-)
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Aug 26, 2010 at 22:59, David Reiss <dr...@facebook.com> wrote:
>> If we plan to experiment with switching the compiler to a different
>> language, the acceptance test should be that it can take any thrift file and
>> generate hashwise identical code for all of the currently supported
>> languages.
>
> I agree.
I also agree.
I've done this in the past and will note that this also means the
"new" code needs to do some things very weirdly to get that
character-by-character equivalence. Even though it makes the new code
funky, it must be done to demonstrate that things match up. Then...
once equivalence is proved, you can go back and make incremental
changes to bring the "new" code into its True and Proper state.
It's only transient, but something that you Just Have To Do.
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by David Reiss <dr...@facebook.com>.
> If we plan to experiment with switching the compiler to a different
> language, the acceptance test should be that it can take any thrift file and
> generate hashwise identical code for all of the currently supported
> languages.
I agree.
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Rush Manbert <ru...@manbert.com>.
Well, you might look at Jira-591 and the really big patch that I submitted about a year ago now. A complete (at the time) Asio/Boost thread implementation of the Thrift C++ runtime that exists in parallel with, or as a replacement for, the POSIX/pthread implementation. Which version you use is selectable at configure time.
Full (perhaps amateurish) autoconf support for all of the C++-based tests. All C++ tests (including the stress test in many variations) are run by make check.
Bug fixes for everything I discovered along the way, including fixes to the concurrency manager and the concurrency tests.
Full Windows configuration support. Supports nmake builds, including "make check" just like *nix. Visual Studio projects for everything C++ (used by the nmake builds).
Pretty much ignored except by a few users who have put it into production.
It's not like I have time to maintain it, but it's disappointing that the general lack of interest and the fact that it's a lot of code sort of doomed it to patch purgatory. Now it's very out of date. If it had been on a branch in the repository it might have been easier to swallow, or at least play with.
- Rush
On Aug 27, 2010, at 1:28 PM, Michael Lum wrote:
> I guess I find it odd that the reaction to platform-specific build issues is to rewrite the project in another language instead of attempting to fix the issues. There are plenty of much larger C++ projects that build on multiple platforms just fine (Ogre3d and Mozilla come to mind).
>
> Are there examples of Windows patches that have been submitted and rejected? A patch that fixes something on one platform and breaks another isn't going to be acceptable, but with appropriate preprocessor directives, a cross-platform codebase can be created.
>
> "Little community interest" can turn into full support if someone will step up and work on Windows compatibility.
>
> On 8/27/2010 11:57 AM, Jeffrey DeCew wrote:
>> Thrift is a project focused on bridging barriers between languages and
>> environments. That's why it seems strange to me that there is so little
>> community support for a robust Windows-friendly solution as a core part of
>> Thrift (especially with regard to the C++ language support). I am well
>> aware that speaking out in favor of Windows support is the easiest way to
>> get ignored on an open source software mailing list, but it seems to me that
>> Thrift's goal should be to achieve the broadest adoption possible. From
>> what I've seen the Windows support (mostly meaning willingness to commit
>> windows-focused patches, but also just the availability of up-to-date
>> patches and responsiveness of the thrift-user list) has left a lot to be
>> desired.
>>
>> With respect to the C++ language support on Windows, I've noticed that there
>> are a bunch of different patches available that work with different versions
>> of thrift and different compiler environments, each one maintained (or not)
>> by a different person. These are often left as patches because they are not
>> general solutions that can be integrated without breaking other
>> environments, but they are written that way in part because their developers
>> see little community interest in adopting windows-helping patches into the
>> trunk. (I know my coworker created a QT-based TSocket and just excluded
>> half of the C++ runtime library to get his code to compile. He and I chose
>> that route over attempting to 'patch' the project because it was clear to us
>> that C++ patches for windows support were not being adopted.)
>>
>> While a Java compiler may be far less convenient than C++ with
>> lexx/yacc/autotools for offices where everyone uses Linux, it seems that the
>> complexities of making the C++ compiler usable and accessible across all
>> platforms has proven to be a big problem. If adoption is important, we need
>> to remove as many hurdles as possible, and I believe moving to a Java
>> compiler is the best bet for accomplishing that.
>>
>> --
>> Jeff DeCew
>>
>>
>> On Fri, Aug 27, 2010 at 10:50 AM, Bruce Lowekamp
>> <br...@skype.net>wrote:
>>
>>> +1
>>>
>>> On Aug 27, 2010, at 10:23 AM, Anthony Molinaro wrote:
>>>
>>>> First off, I think we should stick with the current compiler for a while.
>>>> I think its quicker to package up the compiler for a few platforms, and
>>>> look at some of the patches that have been made available for building
>>>> thrift with visual studio.
>>>>
>>>> If some folks want to start a branch and work on a java compiler that's
>>>> great, go for it, I'd say either a git fork of the thrift git mirror or
>>>> if you have commit access put it in a branch of svn (or even in
>>> compiler/java,
>>>> just leave it disabled by default). When its as complete as the current
>>>> compiler a decision can be made which to use.
>>>>
>>>> However, something to think about is right now there are 49 open issues
>>>> for java and 41 for C++, and 252 open issues on the jira. Before we
>>>> add to those issues with a completely new compiler, maybe we should get
>>>> the current list down to a manageable level?
>>>>
>>>> And I'd like to see more tests tied into make check for more languages.
>>>> I don't think we even have basic functionality. I figured out last night
>>>> while working on a minor issue with the erlang framed transport that
>>>> the test/erl/Makefile doesn't even compile the test files and the test
>>>> files are never run. I'll be working on getting some eunit tests
>>>> tied into the make check call.
>>>>
>>>> So I guess it summary, while I like the passion this conversation has
>>>> created in the community (longest thread in the past year as far as I
>>>> can tell), I think the energy has to be put into getting the current
>>>> system as solid as possible before working too much on a new system.
>>>> The project seems to be at a tipping point now, where it is being
>>>> prodded to work through its issues and hopefully graduate out of
>>>> incubator status soon. I think that a new compiler would lead to
>>>> a division of energy and a major setback in this area. After the
>>>> project has graduated and hit '1.0' then I'd think this could be
>>>> revisited.
>>>>
>>>> Just my 2 cents,
>>>>
>>>> -Anthony
>>>>
>>>> On Fri, Aug 27, 2010 at 07:40:25AM -0700, Bryan Duxbury wrote:
>>>>> An alternative would be to have unit tests that cover the basic
>>>>> functionality of each of the languages' generated code and make sure
>>> those
>>>>> keep passing. (I think we should already have these tests, but I don't
>>>>> expect anyone to jump up to make them.)
>>>>>
>>>>> On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon<to...@cloudera.com> wrote:
>>>>>
>>>>>> On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein<gs...@gmail.com> wrote:
>>>>>>
>>>>>>> ASN.1, huh? I clicked that link. Couldn't find an overview link.
>>>>>>> "Papers" and "presentations" and "consortium members" and all that.
>>>>>>>
>>>>>>> Wikipedia has a more reasonable introduction. All those consortia
>>>>>>> members should learn from that, rather than worrying about their
>>>>>>> "international standard". :-P
>>>>>>>
>>>>>>> That said, ASN.1 is about both description and on-wire formats. If
>>>>>>> Thrift used ASN as the descriptive language, then people might think
>>>>>>> it also uses the same wire format. And ASN.1 might not have things
>>>>>>> like field tag numbers, needed by Thrift.
>>>>>>>
>>>>>>> I don't think we're talking about a new IDL here. Just the technology
>>>>>>> to convert those into compilable runtimes.
>>>>>>>
>>>>>>>
>>>>>> Yes, any incompatible change in IDL would be hugely awful.
>>>>>>
>>>>>> If we plan to experiment with switching the compiler to a different
>>>>>> language, the acceptance test should be that it can take any thrift
>>> file
>>>>>> and
>>>>>> generate hashwise identical code for all of the currently supported
>>>>>> languages.
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Cheers,
>>>>>>> -g
>>>>>>>
>>>>>>> On Thu, Aug 26, 2010 at 21:22, Roger Meier<ro...@bufferoverflow.ch>
>>>>>>> wrote:
>>>>>>>> talking about interface descriptions and languages to build to
>>>>>> compiler.
>>>>>>>>
>>>>>>>> What about ASN.1 (http://www.asn1.org/) as a discription language or
>>>>>> pre
>>>>>>>> format?
>>>>>>>>
>>>>>>>> Does somebody have a ASN.1 to Thrift compiler ?
>>>>>>>>
>>>>>>>> Of course there might be some gaps, but what about that?
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 26.08.2010 22:54, schrieb Bjorn Borud:
>>>>>>>>>
>>>>>>>>> on the project I currently work on we have a (ANTLR-based) parser
>>> for
>>>>>>>>> the Thrift IDL language in order to generate code for a proprietary
>>>>>>>>> serialization library.
>>>>>>>>>
>>>>>>>>> it struck me that perhaps we could use this parser the implement the
>>>>>>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>>>>>>> compiler itself could be built as a platform independent artifact --
>>>>>>>>> which should make it a lot more elegant to write Maven plugins for
>>>>>>>>> Thrift. it would also eliminate the need (for us) to maintain
>>> Thrift
>>>>>>>>> compiler binaries for all platforms and versions of the compiler.
>>>>>>>>>
>>>>>>>>> currently the parser lacks some minor features, but this could
>>> easily
>>>>>> be
>>>>>>>>> rectified. the real job is to add the code generation for various
>>>>>>>>> languages.
>>>>>>>>>
>>>>>>>>> if anyone is interested in this, I am going to talk to some people
>>>>>>>>> tomorrow to get formal approval for open sourcing it.
>>>>>>>>>
>>>>>>>>> any thoughts?
>>>>>>>>>
>>>>>>>>> -Bjørn
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>
>>>> --
>>>> ------------------------------------------------------------------------
>>>> Anthony Molinaro<an...@alumni.caltech.edu>
>>>
>>>
>>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
On Fri, Aug 27, 2010 at 1:28 PM, Michael Lum <mi...@openx.org> wrote:
>
> "Little community interest" can turn into full support if someone will step
> up and work on Windows compatibility.
>
>
+1 to that.
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
Not that I think its broken, but I think it hasn't been "fixed" already
because the developers doing the most work on Thrift are on a handful of
platforms, most commonly Snow Leopard and various flavors of Linux.
Using MacPorts, I don't need to "fiddle" with anything.
Having more developers on different platforms, and doing automated
builds on various platforms that report results and regressions could
solve this "problem".
On 8/30/2010 6:31 AM, Bjorn Borud wrote:
> Michael Lum<mi...@openx.org> writes:
>
>> I guess I find it odd that the reaction to platform-specific build
>> issues is to rewrite the project in another language instead of
>> attempting to fix the issues. There are plenty of much larger C++
>> projects that build on multiple platforms just fine (Ogre3d and
>> Mozilla come to mind).
>
> I've worked on lots of C/C++ projects that are cross platform. there is
> nothing remotely "easy" about doing this for non-trivial projects, and
> Mozilla is decidedly non-trivial, so I don't think it is a relevant
> example here.
>
> nobody is arguing that it wouldn't be possible to write a trivially
> portable implementation of the Thrift compiler in C or C++, but it would
> appear that this is not what we *have*. and that presents a problem.
>
> if we had a trivially portable implementation that just works
> everywhere, how come people only manage to reliably build it on just a
> small handful of operating systems? why, then, are there so many people
> who need to fiddle around to build it on their macs and on their windows
> machines? if this was trivially fixable, why isn't it fixed already?
>
> the good thing about writing it in Python or Java would be that there
> would be no need to maintain binaries for X flavors of Y operating
> systems for Z versions of the compiler.
>
> yes, it is entirely possible to screw up any implementation in any
> language by lazily drowning it in awkward dependencies and packaging.
> this is nothing unique for C++, Java, Python, PHP, Perl or whichever
> language you like/dislike.
>
> -Bjørn
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Michael Lum <mi...@openx.org> writes:
> I guess I find it odd that the reaction to platform-specific build
> issues is to rewrite the project in another language instead of
> attempting to fix the issues. There are plenty of much larger C++
> projects that build on multiple platforms just fine (Ogre3d and
> Mozilla come to mind).
I've worked on lots of C/C++ projects that are cross platform. there is
nothing remotely "easy" about doing this for non-trivial projects, and
Mozilla is decidedly non-trivial, so I don't think it is a relevant
example here.
nobody is arguing that it wouldn't be possible to write a trivially
portable implementation of the Thrift compiler in C or C++, but it would
appear that this is not what we *have*. and that presents a problem.
if we had a trivially portable implementation that just works
everywhere, how come people only manage to reliably build it on just a
small handful of operating systems? why, then, are there so many people
who need to fiddle around to build it on their macs and on their windows
machines? if this was trivially fixable, why isn't it fixed already?
the good thing about writing it in Python or Java would be that there
would be no need to maintain binaries for X flavors of Y operating
systems for Z versions of the compiler.
yes, it is entirely possible to screw up any implementation in any
language by lazily drowning it in awkward dependencies and packaging.
this is nothing unique for C++, Java, Python, PHP, Perl or whichever
language you like/dislike.
-Bjørn
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Michael Lum <mi...@openx.org>.
I guess I find it odd that the reaction to platform-specific build
issues is to rewrite the project in another language instead of
attempting to fix the issues. There are plenty of much larger C++
projects that build on multiple platforms just fine (Ogre3d and Mozilla
come to mind).
Are there examples of Windows patches that have been submitted and
rejected? A patch that fixes something on one platform and breaks
another isn't going to be acceptable, but with appropriate preprocessor
directives, a cross-platform codebase can be created.
"Little community interest" can turn into full support if someone will
step up and work on Windows compatibility.
On 8/27/2010 11:57 AM, Jeffrey DeCew wrote:
> Thrift is a project focused on bridging barriers between languages and
> environments. That's why it seems strange to me that there is so little
> community support for a robust Windows-friendly solution as a core part of
> Thrift (especially with regard to the C++ language support). I am well
> aware that speaking out in favor of Windows support is the easiest way to
> get ignored on an open source software mailing list, but it seems to me that
> Thrift's goal should be to achieve the broadest adoption possible. From
> what I've seen the Windows support (mostly meaning willingness to commit
> windows-focused patches, but also just the availability of up-to-date
> patches and responsiveness of the thrift-user list) has left a lot to be
> desired.
>
> With respect to the C++ language support on Windows, I've noticed that there
> are a bunch of different patches available that work with different versions
> of thrift and different compiler environments, each one maintained (or not)
> by a different person. These are often left as patches because they are not
> general solutions that can be integrated without breaking other
> environments, but they are written that way in part because their developers
> see little community interest in adopting windows-helping patches into the
> trunk. (I know my coworker created a QT-based TSocket and just excluded
> half of the C++ runtime library to get his code to compile. He and I chose
> that route over attempting to 'patch' the project because it was clear to us
> that C++ patches for windows support were not being adopted.)
>
> While a Java compiler may be far less convenient than C++ with
> lexx/yacc/autotools for offices where everyone uses Linux, it seems that the
> complexities of making the C++ compiler usable and accessible across all
> platforms has proven to be a big problem. If adoption is important, we need
> to remove as many hurdles as possible, and I believe moving to a Java
> compiler is the best bet for accomplishing that.
>
> --
> Jeff DeCew
>
>
> On Fri, Aug 27, 2010 at 10:50 AM, Bruce Lowekamp
> <br...@skype.net>wrote:
>
>> +1
>>
>> On Aug 27, 2010, at 10:23 AM, Anthony Molinaro wrote:
>>
>>> First off, I think we should stick with the current compiler for a while.
>>> I think its quicker to package up the compiler for a few platforms, and
>>> look at some of the patches that have been made available for building
>>> thrift with visual studio.
>>>
>>> If some folks want to start a branch and work on a java compiler that's
>>> great, go for it, I'd say either a git fork of the thrift git mirror or
>>> if you have commit access put it in a branch of svn (or even in
>> compiler/java,
>>> just leave it disabled by default). When its as complete as the current
>>> compiler a decision can be made which to use.
>>>
>>> However, something to think about is right now there are 49 open issues
>>> for java and 41 for C++, and 252 open issues on the jira. Before we
>>> add to those issues with a completely new compiler, maybe we should get
>>> the current list down to a manageable level?
>>>
>>> And I'd like to see more tests tied into make check for more languages.
>>> I don't think we even have basic functionality. I figured out last night
>>> while working on a minor issue with the erlang framed transport that
>>> the test/erl/Makefile doesn't even compile the test files and the test
>>> files are never run. I'll be working on getting some eunit tests
>>> tied into the make check call.
>>>
>>> So I guess it summary, while I like the passion this conversation has
>>> created in the community (longest thread in the past year as far as I
>>> can tell), I think the energy has to be put into getting the current
>>> system as solid as possible before working too much on a new system.
>>> The project seems to be at a tipping point now, where it is being
>>> prodded to work through its issues and hopefully graduate out of
>>> incubator status soon. I think that a new compiler would lead to
>>> a division of energy and a major setback in this area. After the
>>> project has graduated and hit '1.0' then I'd think this could be
>>> revisited.
>>>
>>> Just my 2 cents,
>>>
>>> -Anthony
>>>
>>> On Fri, Aug 27, 2010 at 07:40:25AM -0700, Bryan Duxbury wrote:
>>>> An alternative would be to have unit tests that cover the basic
>>>> functionality of each of the languages' generated code and make sure
>> those
>>>> keep passing. (I think we should already have these tests, but I don't
>>>> expect anyone to jump up to make them.)
>>>>
>>>> On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon<to...@cloudera.com> wrote:
>>>>
>>>>> On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein<gs...@gmail.com> wrote:
>>>>>
>>>>>> ASN.1, huh? I clicked that link. Couldn't find an overview link.
>>>>>> "Papers" and "presentations" and "consortium members" and all that.
>>>>>>
>>>>>> Wikipedia has a more reasonable introduction. All those consortia
>>>>>> members should learn from that, rather than worrying about their
>>>>>> "international standard". :-P
>>>>>>
>>>>>> That said, ASN.1 is about both description and on-wire formats. If
>>>>>> Thrift used ASN as the descriptive language, then people might think
>>>>>> it also uses the same wire format. And ASN.1 might not have things
>>>>>> like field tag numbers, needed by Thrift.
>>>>>>
>>>>>> I don't think we're talking about a new IDL here. Just the technology
>>>>>> to convert those into compilable runtimes.
>>>>>>
>>>>>>
>>>>> Yes, any incompatible change in IDL would be hugely awful.
>>>>>
>>>>> If we plan to experiment with switching the compiler to a different
>>>>> language, the acceptance test should be that it can take any thrift
>> file
>>>>> and
>>>>> generate hashwise identical code for all of the currently supported
>>>>> languages.
>>>>>
>>>>> -Todd
>>>>>
>>>>>
>>>>>
>>>>>> Cheers,
>>>>>> -g
>>>>>>
>>>>>> On Thu, Aug 26, 2010 at 21:22, Roger Meier<ro...@bufferoverflow.ch>
>>>>>> wrote:
>>>>>>> talking about interface descriptions and languages to build to
>>>>> compiler.
>>>>>>>
>>>>>>> What about ASN.1 (http://www.asn1.org/) as a discription language or
>>>>> pre
>>>>>>> format?
>>>>>>>
>>>>>>> Does somebody have a ASN.1 to Thrift compiler ?
>>>>>>>
>>>>>>> Of course there might be some gaps, but what about that?
>>>>>>>
>>>>>>>
>>>>>>> Am 26.08.2010 22:54, schrieb Bjorn Borud:
>>>>>>>>
>>>>>>>> on the project I currently work on we have a (ANTLR-based) parser
>> for
>>>>>>>> the Thrift IDL language in order to generate code for a proprietary
>>>>>>>> serialization library.
>>>>>>>>
>>>>>>>> it struck me that perhaps we could use this parser the implement the
>>>>>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>>>>>> compiler itself could be built as a platform independent artifact --
>>>>>>>> which should make it a lot more elegant to write Maven plugins for
>>>>>>>> Thrift. it would also eliminate the need (for us) to maintain
>> Thrift
>>>>>>>> compiler binaries for all platforms and versions of the compiler.
>>>>>>>>
>>>>>>>> currently the parser lacks some minor features, but this could
>> easily
>>>>> be
>>>>>>>> rectified. the real job is to add the code generation for various
>>>>>>>> languages.
>>>>>>>>
>>>>>>>> if anyone is interested in this, I am going to talk to some people
>>>>>>>> tomorrow to get formal approval for open sourcing it.
>>>>>>>>
>>>>>>>> any thoughts?
>>>>>>>>
>>>>>>>> -Bjørn
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>
>>> --
>>> ------------------------------------------------------------------------
>>> Anthony Molinaro<an...@alumni.caltech.edu>
>>
>>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Jeffrey DeCew <Je...@DeCew.org>.
Thrift is a project focused on bridging barriers between languages and
environments. That's why it seems strange to me that there is so little
community support for a robust Windows-friendly solution as a core part of
Thrift (especially with regard to the C++ language support). I am well
aware that speaking out in favor of Windows support is the easiest way to
get ignored on an open source software mailing list, but it seems to me that
Thrift's goal should be to achieve the broadest adoption possible. From
what I've seen the Windows support (mostly meaning willingness to commit
windows-focused patches, but also just the availability of up-to-date
patches and responsiveness of the thrift-user list) has left a lot to be
desired.
With respect to the C++ language support on Windows, I've noticed that there
are a bunch of different patches available that work with different versions
of thrift and different compiler environments, each one maintained (or not)
by a different person. These are often left as patches because they are not
general solutions that can be integrated without breaking other
environments, but they are written that way in part because their developers
see little community interest in adopting windows-helping patches into the
trunk. (I know my coworker created a QT-based TSocket and just excluded
half of the C++ runtime library to get his code to compile. He and I chose
that route over attempting to 'patch' the project because it was clear to us
that C++ patches for windows support were not being adopted.)
While a Java compiler may be far less convenient than C++ with
lexx/yacc/autotools for offices where everyone uses Linux, it seems that the
complexities of making the C++ compiler usable and accessible across all
platforms has proven to be a big problem. If adoption is important, we need
to remove as many hurdles as possible, and I believe moving to a Java
compiler is the best bet for accomplishing that.
--
Jeff DeCew
On Fri, Aug 27, 2010 at 10:50 AM, Bruce Lowekamp
<br...@skype.net>wrote:
> +1
>
> On Aug 27, 2010, at 10:23 AM, Anthony Molinaro wrote:
>
> > First off, I think we should stick with the current compiler for a while.
> > I think its quicker to package up the compiler for a few platforms, and
> > look at some of the patches that have been made available for building
> > thrift with visual studio.
> >
> > If some folks want to start a branch and work on a java compiler that's
> > great, go for it, I'd say either a git fork of the thrift git mirror or
> > if you have commit access put it in a branch of svn (or even in
> compiler/java,
> > just leave it disabled by default). When its as complete as the current
> > compiler a decision can be made which to use.
> >
> > However, something to think about is right now there are 49 open issues
> > for java and 41 for C++, and 252 open issues on the jira. Before we
> > add to those issues with a completely new compiler, maybe we should get
> > the current list down to a manageable level?
> >
> > And I'd like to see more tests tied into make check for more languages.
> > I don't think we even have basic functionality. I figured out last night
> > while working on a minor issue with the erlang framed transport that
> > the test/erl/Makefile doesn't even compile the test files and the test
> > files are never run. I'll be working on getting some eunit tests
> > tied into the make check call.
> >
> > So I guess it summary, while I like the passion this conversation has
> > created in the community (longest thread in the past year as far as I
> > can tell), I think the energy has to be put into getting the current
> > system as solid as possible before working too much on a new system.
> > The project seems to be at a tipping point now, where it is being
> > prodded to work through its issues and hopefully graduate out of
> > incubator status soon. I think that a new compiler would lead to
> > a division of energy and a major setback in this area. After the
> > project has graduated and hit '1.0' then I'd think this could be
> > revisited.
> >
> > Just my 2 cents,
> >
> > -Anthony
> >
> > On Fri, Aug 27, 2010 at 07:40:25AM -0700, Bryan Duxbury wrote:
> >> An alternative would be to have unit tests that cover the basic
> >> functionality of each of the languages' generated code and make sure
> those
> >> keep passing. (I think we should already have these tests, but I don't
> >> expect anyone to jump up to make them.)
> >>
> >> On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >>
> >>> On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein <gs...@gmail.com> wrote:
> >>>
> >>>> ASN.1, huh? I clicked that link. Couldn't find an overview link.
> >>>> "Papers" and "presentations" and "consortium members" and all that.
> >>>>
> >>>> Wikipedia has a more reasonable introduction. All those consortia
> >>>> members should learn from that, rather than worrying about their
> >>>> "international standard". :-P
> >>>>
> >>>> That said, ASN.1 is about both description and on-wire formats. If
> >>>> Thrift used ASN as the descriptive language, then people might think
> >>>> it also uses the same wire format. And ASN.1 might not have things
> >>>> like field tag numbers, needed by Thrift.
> >>>>
> >>>> I don't think we're talking about a new IDL here. Just the technology
> >>>> to convert those into compilable runtimes.
> >>>>
> >>>>
> >>> Yes, any incompatible change in IDL would be hugely awful.
> >>>
> >>> If we plan to experiment with switching the compiler to a different
> >>> language, the acceptance test should be that it can take any thrift
> file
> >>> and
> >>> generate hashwise identical code for all of the currently supported
> >>> languages.
> >>>
> >>> -Todd
> >>>
> >>>
> >>>
> >>>> Cheers,
> >>>> -g
> >>>>
> >>>> On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch>
> >>>> wrote:
> >>>>> talking about interface descriptions and languages to build to
> >>> compiler.
> >>>>>
> >>>>> What about ASN.1 (http://www.asn1.org/) as a discription language or
> >>> pre
> >>>>> format?
> >>>>>
> >>>>> Does somebody have a ASN.1 to Thrift compiler ?
> >>>>>
> >>>>> Of course there might be some gaps, but what about that?
> >>>>>
> >>>>>
> >>>>> Am 26.08.2010 22:54, schrieb Bjorn Borud:
> >>>>>>
> >>>>>> on the project I currently work on we have a (ANTLR-based) parser
> for
> >>>>>> the Thrift IDL language in order to generate code for a proprietary
> >>>>>> serialization library.
> >>>>>>
> >>>>>> it struck me that perhaps we could use this parser the implement the
> >>>>>> Thrift compiler in Java instead. this would mean that the thrift
> >>>>>> compiler itself could be built as a platform independent artifact --
> >>>>>> which should make it a lot more elegant to write Maven plugins for
> >>>>>> Thrift. it would also eliminate the need (for us) to maintain
> Thrift
> >>>>>> compiler binaries for all platforms and versions of the compiler.
> >>>>>>
> >>>>>> currently the parser lacks some minor features, but this could
> easily
> >>> be
> >>>>>> rectified. the real job is to add the code generation for various
> >>>>>> languages.
> >>>>>>
> >>>>>> if anyone is interested in this, I am going to talk to some people
> >>>>>> tomorrow to get formal approval for open sourcing it.
> >>>>>>
> >>>>>> any thoughts?
> >>>>>>
> >>>>>> -Bjørn
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>>
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro <an...@alumni.caltech.edu>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bruce Lowekamp <br...@skype.net>.
+1
On Aug 27, 2010, at 10:23 AM, Anthony Molinaro wrote:
> First off, I think we should stick with the current compiler for a while.
> I think its quicker to package up the compiler for a few platforms, and
> look at some of the patches that have been made available for building
> thrift with visual studio.
>
> If some folks want to start a branch and work on a java compiler that's
> great, go for it, I'd say either a git fork of the thrift git mirror or
> if you have commit access put it in a branch of svn (or even in compiler/java,
> just leave it disabled by default). When its as complete as the current
> compiler a decision can be made which to use.
>
> However, something to think about is right now there are 49 open issues
> for java and 41 for C++, and 252 open issues on the jira. Before we
> add to those issues with a completely new compiler, maybe we should get
> the current list down to a manageable level?
>
> And I'd like to see more tests tied into make check for more languages.
> I don't think we even have basic functionality. I figured out last night
> while working on a minor issue with the erlang framed transport that
> the test/erl/Makefile doesn't even compile the test files and the test
> files are never run. I'll be working on getting some eunit tests
> tied into the make check call.
>
> So I guess it summary, while I like the passion this conversation has
> created in the community (longest thread in the past year as far as I
> can tell), I think the energy has to be put into getting the current
> system as solid as possible before working too much on a new system.
> The project seems to be at a tipping point now, where it is being
> prodded to work through its issues and hopefully graduate out of
> incubator status soon. I think that a new compiler would lead to
> a division of energy and a major setback in this area. After the
> project has graduated and hit '1.0' then I'd think this could be
> revisited.
>
> Just my 2 cents,
>
> -Anthony
>
> On Fri, Aug 27, 2010 at 07:40:25AM -0700, Bryan Duxbury wrote:
>> An alternative would be to have unit tests that cover the basic
>> functionality of each of the languages' generated code and make sure those
>> keep passing. (I think we should already have these tests, but I don't
>> expect anyone to jump up to make them.)
>>
>> On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>
>>> On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein <gs...@gmail.com> wrote:
>>>
>>>> ASN.1, huh? I clicked that link. Couldn't find an overview link.
>>>> "Papers" and "presentations" and "consortium members" and all that.
>>>>
>>>> Wikipedia has a more reasonable introduction. All those consortia
>>>> members should learn from that, rather than worrying about their
>>>> "international standard". :-P
>>>>
>>>> That said, ASN.1 is about both description and on-wire formats. If
>>>> Thrift used ASN as the descriptive language, then people might think
>>>> it also uses the same wire format. And ASN.1 might not have things
>>>> like field tag numbers, needed by Thrift.
>>>>
>>>> I don't think we're talking about a new IDL here. Just the technology
>>>> to convert those into compilable runtimes.
>>>>
>>>>
>>> Yes, any incompatible change in IDL would be hugely awful.
>>>
>>> If we plan to experiment with switching the compiler to a different
>>> language, the acceptance test should be that it can take any thrift file
>>> and
>>> generate hashwise identical code for all of the currently supported
>>> languages.
>>>
>>> -Todd
>>>
>>>
>>>
>>>> Cheers,
>>>> -g
>>>>
>>>> On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch>
>>>> wrote:
>>>>> talking about interface descriptions and languages to build to
>>> compiler.
>>>>>
>>>>> What about ASN.1 (http://www.asn1.org/) as a discription language or
>>> pre
>>>>> format?
>>>>>
>>>>> Does somebody have a ASN.1 to Thrift compiler ?
>>>>>
>>>>> Of course there might be some gaps, but what about that?
>>>>>
>>>>>
>>>>> Am 26.08.2010 22:54, schrieb Bjorn Borud:
>>>>>>
>>>>>> on the project I currently work on we have a (ANTLR-based) parser for
>>>>>> the Thrift IDL language in order to generate code for a proprietary
>>>>>> serialization library.
>>>>>>
>>>>>> it struck me that perhaps we could use this parser the implement the
>>>>>> Thrift compiler in Java instead. this would mean that the thrift
>>>>>> compiler itself could be built as a platform independent artifact --
>>>>>> which should make it a lot more elegant to write Maven plugins for
>>>>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>>>>> compiler binaries for all platforms and versions of the compiler.
>>>>>>
>>>>>> currently the parser lacks some minor features, but this could easily
>>> be
>>>>>> rectified. the real job is to add the code generation for various
>>>>>> languages.
>>>>>>
>>>>>> if anyone is interested in this, I am going to talk to some people
>>>>>> tomorrow to get formal approval for open sourcing it.
>>>>>>
>>>>>> any thoughts?
>>>>>>
>>>>>> -Bjørn
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
On Fri, Aug 27, 2010 at 10:23 AM, Anthony Molinaro <
anthonym@alumni.caltech.edu> wrote:
> However, something to think about is right now there are 49 open issues
> for java and 41 for C++, and 252 open issues on the jira. Before we
> add to those issues with a completely new compiler, maybe we should get
> the current list down to a manageable level?
>
I like the idea, but I'm not sure that's feasible. My expectation is that
we'll always have a certain number of tickets open at all times for unsolved
bugs and speculative new features. I would love for people to pitch in and
start sending in patches, but I don't think the number of issues should be a
limiting factor for anything.
>
> And I'd like to see more tests tied into make check for more languages.
> I don't think we even have basic functionality. I figured out last night
> while working on a minor issue with the erlang framed transport that
> the test/erl/Makefile doesn't even compile the test files and the test
> files are never run. I'll be working on getting some eunit tests
> tied into the make check call.
>
Sounds like you should be creating some tickets...
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Upayavira <uv...@odoko.co.uk>.
On Fri, 2010-08-27 at 10:23 -0700, Anthony Molinaro wrote:
> First off, I think we should stick with the current compiler for a while.
> I think its quicker to package up the compiler for a few platforms, and
> look at some of the patches that have been made available for building
> thrift with visual studio.
>
> If some folks want to start a branch and work on a java compiler that's
> great, go for it, I'd say either a git fork of the thrift git mirror or
> if you have commit access put it in a branch of svn (or even in compiler/java,
> just leave it disabled by default). When its as complete as the current
> compiler a decision can be made which to use.
>
> However, something to think about is right now there are 49 open issues
> for java and 41 for C++, and 252 open issues on the jira. Before we
> add to those issues with a completely new compiler, maybe we should get
> the current list down to a manageable level?
>
> And I'd like to see more tests tied into make check for more languages.
> I don't think we even have basic functionality. I figured out last night
> while working on a minor issue with the erlang framed transport that
> the test/erl/Makefile doesn't even compile the test files and the test
> files are never run. I'll be working on getting some eunit tests
> tied into the make check call.
>
> So I guess it summary, while I like the passion this conversation has
> created in the community (longest thread in the past year as far as I
> can tell), I think the energy has to be put into getting the current
> system as solid as possible before working too much on a new system.
> The project seems to be at a tipping point now, where it is being
> prodded to work through its issues and hopefully graduate out of
> incubator status soon. I think that a new compiler would lead to
> a division of energy and a major setback in this area. After the
> project has graduated and hit '1.0' then I'd think this could be
> revisited.
I don't think this is quite that straight-forward. While traditional
software teams are resource limited, open source teams aren't
necessarily. If two different sets of folks want to get on with a java
and fixing up the C++ compilers, I'd say go for it. We can worry about
one 'winning' at some point later, and it likely won't be entirely
technical merit that will pick the winner (if it is decided that a
winner is needed).
Upayavira
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Anthony Molinaro <an...@alumni.caltech.edu>.
First off, I think we should stick with the current compiler for a while.
I think its quicker to package up the compiler for a few platforms, and
look at some of the patches that have been made available for building
thrift with visual studio.
If some folks want to start a branch and work on a java compiler that's
great, go for it, I'd say either a git fork of the thrift git mirror or
if you have commit access put it in a branch of svn (or even in compiler/java,
just leave it disabled by default). When its as complete as the current
compiler a decision can be made which to use.
However, something to think about is right now there are 49 open issues
for java and 41 for C++, and 252 open issues on the jira. Before we
add to those issues with a completely new compiler, maybe we should get
the current list down to a manageable level?
And I'd like to see more tests tied into make check for more languages.
I don't think we even have basic functionality. I figured out last night
while working on a minor issue with the erlang framed transport that
the test/erl/Makefile doesn't even compile the test files and the test
files are never run. I'll be working on getting some eunit tests
tied into the make check call.
So I guess it summary, while I like the passion this conversation has
created in the community (longest thread in the past year as far as I
can tell), I think the energy has to be put into getting the current
system as solid as possible before working too much on a new system.
The project seems to be at a tipping point now, where it is being
prodded to work through its issues and hopefully graduate out of
incubator status soon. I think that a new compiler would lead to
a division of energy and a major setback in this area. After the
project has graduated and hit '1.0' then I'd think this could be
revisited.
Just my 2 cents,
-Anthony
On Fri, Aug 27, 2010 at 07:40:25AM -0700, Bryan Duxbury wrote:
> An alternative would be to have unit tests that cover the basic
> functionality of each of the languages' generated code and make sure those
> keep passing. (I think we should already have these tests, but I don't
> expect anyone to jump up to make them.)
>
> On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein <gs...@gmail.com> wrote:
> >
> > > ASN.1, huh? I clicked that link. Couldn't find an overview link.
> > > "Papers" and "presentations" and "consortium members" and all that.
> > >
> > > Wikipedia has a more reasonable introduction. All those consortia
> > > members should learn from that, rather than worrying about their
> > > "international standard". :-P
> > >
> > > That said, ASN.1 is about both description and on-wire formats. If
> > > Thrift used ASN as the descriptive language, then people might think
> > > it also uses the same wire format. And ASN.1 might not have things
> > > like field tag numbers, needed by Thrift.
> > >
> > > I don't think we're talking about a new IDL here. Just the technology
> > > to convert those into compilable runtimes.
> > >
> > >
> > Yes, any incompatible change in IDL would be hugely awful.
> >
> > If we plan to experiment with switching the compiler to a different
> > language, the acceptance test should be that it can take any thrift file
> > and
> > generate hashwise identical code for all of the currently supported
> > languages.
> >
> > -Todd
> >
> >
> >
> > > Cheers,
> > > -g
> > >
> > > On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch>
> > > wrote:
> > > > talking about interface descriptions and languages to build to
> > compiler.
> > > >
> > > > What about ASN.1 (http://www.asn1.org/) as a discription language or
> > pre
> > > > format?
> > > >
> > > > Does somebody have a ASN.1 to Thrift compiler ?
> > > >
> > > > Of course there might be some gaps, but what about that?
> > > >
> > > >
> > > > Am 26.08.2010 22:54, schrieb Bjorn Borud:
> > > >>
> > > >> on the project I currently work on we have a (ANTLR-based) parser for
> > > >> the Thrift IDL language in order to generate code for a proprietary
> > > >> serialization library.
> > > >>
> > > >> it struck me that perhaps we could use this parser the implement the
> > > >> Thrift compiler in Java instead. this would mean that the thrift
> > > >> compiler itself could be built as a platform independent artifact --
> > > >> which should make it a lot more elegant to write Maven plugins for
> > > >> Thrift. it would also eliminate the need (for us) to maintain Thrift
> > > >> compiler binaries for all platforms and versions of the compiler.
> > > >>
> > > >> currently the parser lacks some minor features, but this could easily
> > be
> > > >> rectified. the real job is to add the code generation for various
> > > >> languages.
> > > >>
> > > >> if anyone is interested in this, I am going to talk to some people
> > > >> tomorrow to get formal approval for open sourcing it.
> > > >>
> > > >> any thoughts?
> > > >>
> > > >> -Bjørn
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
An alternative would be to have unit tests that cover the basic
functionality of each of the languages' generated code and make sure those
keep passing. (I think we should already have these tests, but I don't
expect anyone to jump up to make them.)
On Thu, Aug 26, 2010 at 6:41 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein <gs...@gmail.com> wrote:
>
> > ASN.1, huh? I clicked that link. Couldn't find an overview link.
> > "Papers" and "presentations" and "consortium members" and all that.
> >
> > Wikipedia has a more reasonable introduction. All those consortia
> > members should learn from that, rather than worrying about their
> > "international standard". :-P
> >
> > That said, ASN.1 is about both description and on-wire formats. If
> > Thrift used ASN as the descriptive language, then people might think
> > it also uses the same wire format. And ASN.1 might not have things
> > like field tag numbers, needed by Thrift.
> >
> > I don't think we're talking about a new IDL here. Just the technology
> > to convert those into compilable runtimes.
> >
> >
> Yes, any incompatible change in IDL would be hugely awful.
>
> If we plan to experiment with switching the compiler to a different
> language, the acceptance test should be that it can take any thrift file
> and
> generate hashwise identical code for all of the currently supported
> languages.
>
> -Todd
>
>
>
> > Cheers,
> > -g
> >
> > On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch>
> > wrote:
> > > talking about interface descriptions and languages to build to
> compiler.
> > >
> > > What about ASN.1 (http://www.asn1.org/) as a discription language or
> pre
> > > format?
> > >
> > > Does somebody have a ASN.1 to Thrift compiler ?
> > >
> > > Of course there might be some gaps, but what about that?
> > >
> > >
> > > Am 26.08.2010 22:54, schrieb Bjorn Borud:
> > >>
> > >> on the project I currently work on we have a (ANTLR-based) parser for
> > >> the Thrift IDL language in order to generate code for a proprietary
> > >> serialization library.
> > >>
> > >> it struck me that perhaps we could use this parser the implement the
> > >> Thrift compiler in Java instead. this would mean that the thrift
> > >> compiler itself could be built as a platform independent artifact --
> > >> which should make it a lot more elegant to write Maven plugins for
> > >> Thrift. it would also eliminate the need (for us) to maintain Thrift
> > >> compiler binaries for all platforms and versions of the compiler.
> > >>
> > >> currently the parser lacks some minor features, but this could easily
> be
> > >> rectified. the real job is to add the code generation for various
> > >> languages.
> > >>
> > >> if anyone is interested in this, I am going to talk to some people
> > >> tomorrow to get formal approval for open sourcing it.
> > >>
> > >> any thoughts?
> > >>
> > >> -Bjørn
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Aug 26, 2010 at 6:33 PM, Greg Stein <gs...@gmail.com> wrote:
> ASN.1, huh? I clicked that link. Couldn't find an overview link.
> "Papers" and "presentations" and "consortium members" and all that.
>
> Wikipedia has a more reasonable introduction. All those consortia
> members should learn from that, rather than worrying about their
> "international standard". :-P
>
> That said, ASN.1 is about both description and on-wire formats. If
> Thrift used ASN as the descriptive language, then people might think
> it also uses the same wire format. And ASN.1 might not have things
> like field tag numbers, needed by Thrift.
>
> I don't think we're talking about a new IDL here. Just the technology
> to convert those into compilable runtimes.
>
>
Yes, any incompatible change in IDL would be hugely awful.
If we plan to experiment with switching the compiler to a different
language, the acceptance test should be that it can take any thrift file and
generate hashwise identical code for all of the currently supported
languages.
-Todd
> Cheers,
> -g
>
> On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch>
> wrote:
> > talking about interface descriptions and languages to build to compiler.
> >
> > What about ASN.1 (http://www.asn1.org/) as a discription language or pre
> > format?
> >
> > Does somebody have a ASN.1 to Thrift compiler ?
> >
> > Of course there might be some gaps, but what about that?
> >
> >
> > Am 26.08.2010 22:54, schrieb Bjorn Borud:
> >>
> >> on the project I currently work on we have a (ANTLR-based) parser for
> >> the Thrift IDL language in order to generate code for a proprietary
> >> serialization library.
> >>
> >> it struck me that perhaps we could use this parser the implement the
> >> Thrift compiler in Java instead. this would mean that the thrift
> >> compiler itself could be built as a platform independent artifact --
> >> which should make it a lot more elegant to write Maven plugins for
> >> Thrift. it would also eliminate the need (for us) to maintain Thrift
> >> compiler binaries for all platforms and versions of the compiler.
> >>
> >> currently the parser lacks some minor features, but this could easily be
> >> rectified. the real job is to add the code generation for various
> >> languages.
> >>
> >> if anyone is interested in this, I am going to talk to some people
> >> tomorrow to get formal approval for open sourcing it.
> >>
> >> any thoughts?
> >>
> >> -Bjørn
> >>
> >>
> >>
> >>
> >
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Roger Meier <ro...@bufferoverflow.ch>.
Fully agree with your statements. I'm not the consortium lover.
I prefer democracy where all have the same vote weight.
one thing i had in mind was converting some asn.1 we have at work into
thirft idl
or
a similar thing might be protocol buffers idl to thrift converter.
I know, sounds crazy...but sometimes things start there!
Am 27.08.2010 03:33, schrieb Greg Stein:
> ASN.1, huh? I clicked that link. Couldn't find an overview link.
> "Papers" and "presentations" and "consortium members" and all that.
>
> Wikipedia has a more reasonable introduction. All those consortia
> members should learn from that, rather than worrying about their
> "international standard". :-P
>
> That said, ASN.1 is about both description and on-wire formats. If
> Thrift used ASN as the descriptive language, then people might think
> it also uses the same wire format. And ASN.1 might not have things
> like field tag numbers, needed by Thrift.
>
> I don't think we're talking about a new IDL here. Just the technology
> to convert those into compilable runtimes.
>
> Cheers,
> -g
>
> On Thu, Aug 26, 2010 at 21:22, Roger Meier<ro...@bufferoverflow.ch> wrote:
>
>> talking about interface descriptions and languages to build to compiler.
>>
>> What about ASN.1 (http://www.asn1.org/) as a discription language or pre
>> format?
>>
>> Does somebody have a ASN.1 to Thrift compiler ?
>>
>> Of course there might be some gaps, but what about that?
>>
>>
>> Am 26.08.2010 22:54, schrieb Bjorn Borud:
>>
>>> on the project I currently work on we have a (ANTLR-based) parser for
>>> the Thrift IDL language in order to generate code for a proprietary
>>> serialization library.
>>>
>>> it struck me that perhaps we could use this parser the implement the
>>> Thrift compiler in Java instead. this would mean that the thrift
>>> compiler itself could be built as a platform independent artifact --
>>> which should make it a lot more elegant to write Maven plugins for
>>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>>> compiler binaries for all platforms and versions of the compiler.
>>>
>>> currently the parser lacks some minor features, but this could easily be
>>> rectified. the real job is to add the code generation for various
>>> languages.
>>>
>>> if anyone is interested in this, I am going to talk to some people
>>> tomorrow to get formal approval for open sourcing it.
>>>
>>> any thoughts?
>>>
>>> -Bjørn
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
ASN.1, huh? I clicked that link. Couldn't find an overview link.
"Papers" and "presentations" and "consortium members" and all that.
Wikipedia has a more reasonable introduction. All those consortia
members should learn from that, rather than worrying about their
"international standard". :-P
That said, ASN.1 is about both description and on-wire formats. If
Thrift used ASN as the descriptive language, then people might think
it also uses the same wire format. And ASN.1 might not have things
like field tag numbers, needed by Thrift.
I don't think we're talking about a new IDL here. Just the technology
to convert those into compilable runtimes.
Cheers,
-g
On Thu, Aug 26, 2010 at 21:22, Roger Meier <ro...@bufferoverflow.ch> wrote:
> talking about interface descriptions and languages to build to compiler.
>
> What about ASN.1 (http://www.asn1.org/) as a discription language or pre
> format?
>
> Does somebody have a ASN.1 to Thrift compiler ?
>
> Of course there might be some gaps, but what about that?
>
>
> Am 26.08.2010 22:54, schrieb Bjorn Borud:
>>
>> on the project I currently work on we have a (ANTLR-based) parser for
>> the Thrift IDL language in order to generate code for a proprietary
>> serialization library.
>>
>> it struck me that perhaps we could use this parser the implement the
>> Thrift compiler in Java instead. this would mean that the thrift
>> compiler itself could be built as a platform independent artifact --
>> which should make it a lot more elegant to write Maven plugins for
>> Thrift. it would also eliminate the need (for us) to maintain Thrift
>> compiler binaries for all platforms and versions of the compiler.
>>
>> currently the parser lacks some minor features, but this could easily be
>> rectified. the real job is to add the code generation for various
>> languages.
>>
>> if anyone is interested in this, I am going to talk to some people
>> tomorrow to get formal approval for open sourcing it.
>>
>> any thoughts?
>>
>> -Bjørn
>>
>>
>>
>>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Roger Meier <ro...@bufferoverflow.ch>.
talking about interface descriptions and languages to build to compiler.
What about ASN.1 (http://www.asn1.org/) as a discription language or pre
format?
Does somebody have a ASN.1 to Thrift compiler ?
Of course there might be some gaps, but what about that?
Am 26.08.2010 22:54, schrieb Bjorn Borud:
> on the project I currently work on we have a (ANTLR-based) parser for
> the Thrift IDL language in order to generate code for a proprietary
> serialization library.
>
> it struck me that perhaps we could use this parser the implement the
> Thrift compiler in Java instead. this would mean that the thrift
> compiler itself could be built as a platform independent artifact --
> which should make it a lot more elegant to write Maven plugins for
> Thrift. it would also eliminate the need (for us) to maintain Thrift
> compiler binaries for all platforms and versions of the compiler.
>
> currently the parser lacks some minor features, but this could easily be
> rectified. the real job is to add the code generation for various
> languages.
>
> if anyone is interested in this, I am going to talk to some people
> tomorrow to get formal approval for open sourcing it.
>
> any thoughts?
>
> -Bjørn
>
>
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Ryan King <ry...@twitter.com>.
On Thu, Aug 26, 2010 at 4:19 PM, Bryan Duxbury <br...@rapleaf.com> wrote:
> I am also very interested in this. I think it would be a lot easier to
> develop a clean, well-factored compiler in Java, and we'd get the easier
> cross-platform usage.
>
> If you get us over the initial parser hump, I'll gladly contribute time to
> porting the ruby and java code generators.
I'd likewise be willing to help on the ruby side (and I've done stuff
with ruby and antlr before).
-ryan
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
I am also very interested in this. I think it would be a lot easier to
develop a clean, well-factored compiler in Java, and we'd get the easier
cross-platform usage.
If you get us over the initial parser hump, I'll gladly contribute time to
porting the ruby and java code generators.
Make a ticket?
On Thu, Aug 26, 2010 at 1:54 PM, Bjorn Borud <bb...@gmail.com> wrote:
>
> on the project I currently work on we have a (ANTLR-based) parser for
> the Thrift IDL language in order to generate code for a proprietary
> serialization library.
>
> it struck me that perhaps we could use this parser the implement the
> Thrift compiler in Java instead. this would mean that the thrift
> compiler itself could be built as a platform independent artifact --
> which should make it a lot more elegant to write Maven plugins for
> Thrift. it would also eliminate the need (for us) to maintain Thrift
> compiler binaries for all platforms and versions of the compiler.
>
> currently the parser lacks some minor features, but this could easily be
> rectified. the real job is to add the code generation for various
> languages.
>
> if anyone is interested in this, I am going to talk to some people
> tomorrow to get formal approval for open sourcing it.
>
> any thoughts?
>
> -Bjørn
>
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bjorn Borud <bb...@gmail.com>.
Chad Walters <Ch...@microsoft.com> writes:
>
> Picking any one language for the compiler will always displease some
> part of the community. If you move to Java exclusively, C++ shops may
> complain, etc. If I had to pick one, I'd pick something a bit
> lighter-weight than Java. Probably Perl fits the bill -- not because I
> like the language much but because it can serve as a lowest-common
> denominator language and the compiler honestly doesn't need to be that
> complex.
when pondering which lowest-common-denominator-language to pick I came
up with Python. at this point I think Python is about as common as
Perl in the wild, and although not my favorite language, it is cleaner
than Perl.
I've written a *lot* of Perl code in the past and my problem with Perl
is that people tend to favor unnecessarily convoluted, read-only idioms.
Or in short: Perl is fine as long as you only have to deal with your own
code -- it is a pain in the neck if you have to deal with the code of
someone else. (This is not made any better by the fact that very
visible Perl luminaries write books that favor "tricks" over more
verbose but readable code. Nor the fact that the basic credo of Perl is
"there is more than one way to do it").
> Alternatively, turn all the language implementations into a set of
> template files and come up with a lightweight template generation
> specification that can be implemented simply in any language.
I can see why this would be tempting, but I doubt if it is worth the
effort. I suspect more effort would be spent on maintaining this
templating system than on more important things.
-Bjørn
RE: Thrift compiler in Java (have parser, willing to hack)
Posted by Chad Walters <Ch...@microsoft.com>.
My $0.02 (for whatever it is worth as a non-contributor at this point):
I agree that dual maintenance is likely to be a total pain.
Picking any one language for the compiler will always displease some part of the community. If you move to Java exclusively, C++ shops may complain, etc. If I had to pick one, I'd pick something a bit lighter-weight than Java. Probably Perl fits the bill -- not because I like the language much but because it can serve as a lowest-common denominator language and the compiler honestly doesn't need to be that complex.
Alternatively, turn all the language implementations into a set of template files and come up with a lightweight template generation specification that can be implemented simply in any language. Then if you don't like any of the language choices, you can go ahead and implement it in the language of your choice. Obviously this is a lot more work and requires some real brain power to be applied but it might actually be a nice separate open-source project with more general applicability if done right.
Chad
-----Original Message-----
From: Bryan Duxbury [mailto:bryan@rapleaf.com]
Sent: Friday, August 27, 2010 7:38 AM
To: thrift-dev@incubator.apache.org
Subject: Re: Thrift compiler in Java (have parser, willing to hack)
I agree that our end goal would be to have a single compiler, though clearly there would be a long period of us maintaining both.
On Fri, Aug 27, 2010 at 4:43 AM, Greg Stein <gs...@gmail.com> wrote:
> On Fri, Aug 27, 2010 at 05:21, Emmanuel Bourg <eb...@apache.org> wrote:
> > Le 26/08/2010 22:54, Bjorn Borud a écrit :
> >
> >> if anyone is interested in this, I am going to talk to some people
> >> tomorrow to get formal approval for open sourcing it.
> >>
> >> any thoughts?
> >
> > Having a Java base parser/compiler would be really nice!
> >
> > When I wrote my "light" compiler for Java [1] I first thought about
> > rewriting the compiler in Java from scratch. Eventually I chose to
> > hack
> the
> > existing C++ code, but from the perspective of a Java developer
> > working
> on
> > Windows that was really a pain. Getting the Thrift source to compile
> isn't
> > really trivial, and using the compiler is rather impractical (I
> > switch
> back
> > and forth between Windows and a Debian box).
> >
> > There is no reason to replace the existing compiler in C++, a Java
> compiler
> > can coexist. Choice is good.
>
> Dual maintenance is awful. It is a serious drag on the community to
> try and support *two* compilers. I would be *very* against trying to
> have coexisting compilers. We need to stick to one and make it
> successful.
>
> Cheers,
> -g
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Bryan Duxbury <br...@rapleaf.com>.
I agree that our end goal would be to have a single compiler, though clearly
there would be a long period of us maintaining both.
On Fri, Aug 27, 2010 at 4:43 AM, Greg Stein <gs...@gmail.com> wrote:
> On Fri, Aug 27, 2010 at 05:21, Emmanuel Bourg <eb...@apache.org> wrote:
> > Le 26/08/2010 22:54, Bjorn Borud a écrit :
> >
> >> if anyone is interested in this, I am going to talk to some people
> >> tomorrow to get formal approval for open sourcing it.
> >>
> >> any thoughts?
> >
> > Having a Java base parser/compiler would be really nice!
> >
> > When I wrote my "light" compiler for Java [1] I first thought about
> > rewriting the compiler in Java from scratch. Eventually I chose to hack
> the
> > existing C++ code, but from the perspective of a Java developer working
> on
> > Windows that was really a pain. Getting the Thrift source to compile
> isn't
> > really trivial, and using the compiler is rather impractical (I switch
> back
> > and forth between Windows and a Debian box).
> >
> > There is no reason to replace the existing compiler in C++, a Java
> compiler
> > can coexist. Choice is good.
>
> Dual maintenance is awful. It is a serious drag on the community to
> try and support *two* compilers. I would be *very* against trying to
> have coexisting compilers. We need to stick to one and make it
> successful.
>
> Cheers,
> -g
>
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 27/08/2010 13:43, Greg Stein a écrit :
> Dual maintenance is awful. It is a serious drag on the community to
> try and support *two* compilers. I would be *very* against trying to
> have coexisting compilers. We need to stick to one and make it
> successful.
I understand you may not like it, but that will most likely happen
anyway, and the question is to know where this will happen.
Let say Bjorn releases its compiler on Github and some folks like myself
join to help. Wouldn't we become part of the Thrift community
implicitly? That would just be sad to not share the same home and work
together at Apache.
Emmanuel Bourg
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Aug 27, 2010 at 05:21, Emmanuel Bourg <eb...@apache.org> wrote:
> Le 26/08/2010 22:54, Bjorn Borud a écrit :
>
>> if anyone is interested in this, I am going to talk to some people
>> tomorrow to get formal approval for open sourcing it.
>>
>> any thoughts?
>
> Having a Java base parser/compiler would be really nice!
>
> When I wrote my "light" compiler for Java [1] I first thought about
> rewriting the compiler in Java from scratch. Eventually I chose to hack the
> existing C++ code, but from the perspective of a Java developer working on
> Windows that was really a pain. Getting the Thrift source to compile isn't
> really trivial, and using the compiler is rather impractical (I switch back
> and forth between Windows and a Debian box).
>
> There is no reason to replace the existing compiler in C++, a Java compiler
> can coexist. Choice is good.
Dual maintenance is awful. It is a serious drag on the community to
try and support *two* compilers. I would be *very* against trying to
have coexisting compilers. We need to stick to one and make it
successful.
Cheers,
-g
Re: Thrift compiler in Java (have parser, willing to hack)
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 26/08/2010 22:54, Bjorn Borud a écrit :
> if anyone is interested in this, I am going to talk to some people
> tomorrow to get formal approval for open sourcing it.
>
> any thoughts?
Having a Java base parser/compiler would be really nice!
When I wrote my "light" compiler for Java [1] I first thought about
rewriting the compiler in Java from scratch. Eventually I chose to hack
the existing C++ code, but from the perspective of a Java developer
working on Windows that was really a pain. Getting the Thrift source to
compile isn't really trivial, and using the compiler is rather
impractical (I switch back and forth between Windows and a Debian box).
There is no reason to replace the existing compiler in C++, a Java
compiler can coexist. Choice is good.
Emmanuel Bourg
[1] http://wiki.github.com/ebourg/thrift/