You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Damjan Jovanovic <da...@apache.org> on 2016/02/05 19:32:35 UTC
Thoughts on a new build system for AOO
Hi
With the recent buildbot saga and Patricia's Windows building nightmare, a
number of serious issues were highlighted which got me thinking about how
our build system could be improved.
The problems with building AOO were researched and documented years ago (
https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good
analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis
which is worth a read.
Their solution was gbuild: a build system using GNU make's new "eval"
feature to rewrite the makefile from within, which would generate one giant
dependency graph of every file in AOO and produce reliable, stable,
deterministic, and maximally parallelized builds, with minimal rebuilding
on changes. The problem is, the full benefits of gbuild can only be seen
once every module is converted to gbuild, and with only about 10% having
been converted so far, it doesn't even try to build a complete project-wide
dependency graph, but rather gets invoked per-module just like dmake.
Converting dmake modules to gbuild is very difficult: so far I only managed
to convert main/formula, but main/animations was mysteriously segfaulting
AOO...
Besides, LO has completed converting every module to gbuild, guinea pigging
gbuild perfectly for us ;-). How has it gone for them? One of our
committers, jani, evaluated LO's gbuild in April 2013 in a thread here
called "Make x Dmake x Build", where he said how it's "not particulary fast
either", and called mkdir 431 times on a directory that already existed.
And worst of all, gbuild uses its own language with horrible syntax (eg.
LISP-like syntax, spaces around commas not allowed, blank lines sometimes
mandatory and at other times forbidden), is almost completely undocumented,
and very difficult to debug - a maintenance nightmare.
So what other alternatives are there?
There was a comment in
https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements
that got my attention:
"One might consider using Python as a tool for _all_ non-standard build
tasks. Python is easy to distribute and install on all platforms. However,
it is also a very fat dependency and thus it is questionable, if it would
be worth the effort just to get rid of cygwin (or another Unix environment
on windows)."
We already build and use Python, and there is a good build system written
in Python: SCons (http://www.scons.org). It's modern yet well established,
having started in the year 2000, and is used by a number of open source and
commercial projects. It's MIT licensed so we can include it as a
dependency. The build scripts are simple Python scripts and can use the
Python language and libraries, though you don't need to know (much) Python
to use it. It's well documented. It's debuggable and has options to print
dependencies and explain actions taken. It supports all the operating
systems and architectures we support (even unofficially), including Cygwin,
*BSD, Solaris, OS/2. It's "local" package is designed for easy embedding
into a pre-existing project so it runs straight after extraction with no
need to install it system-wide, and is only 445 kB.
It has many features we need or would benefit from:
* It's able to parse included headers from C/C++ files and use them as
dependencies.
* It can generate Visual Studio project files.
* Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex.
* Builds zip and tar archives.
* Just like gbuild, it has a global dependency view and can fully
parallelize building files across module boundaries while maintaining
stability.
* Using Python APIs instead of *nix CLI tools might eventually result in us
being able to ditch Cygwin entirely, not to mention better performance due
to less subprocess spawning.
* Unlike classical make tools which only use mtimes, it determines what to
rebuild using several configurable options, including MD5 checksums of file
contents, meaning changes which don't result in different binaries (eg.
formatting or comment changes) will only recompile changed files and stop
there, because it knows the object files are the same as before, so no
linking or downstream rebuilding needs to be performed.
In other words SCons does most of what gbuild is supposed to be good for,
and more. I see it outperforming LO's pure gbuild, while being more
readable, maintainable, debuggable, and a sight for sore eyes after the
triple whammy of dmake, gbuild and build.pl we've endured for so long.
If people are happy, I'd suggest a migration plan where first scons is
added as a dependency, then the new version of serf is built with it (see
i126312), then gbuild modules are ported to scons until gbuild is
eliminated, then dmake modules are ported to scons until dmake is
eliminated, and finally scons replaces build.pl.
Thoughts?
Damjan
RE: Thoughts on a new build system for AOO
Posted by "Dennis E. Hamilton" <de...@acm.org>.
I think Marcus raises important points about capacity and how we could bootstrap through this.
Some musings:
There are also some related activities that need to follow in some manner, such as changing deployment for Windows and probably for OSX too, both for similar reasons. Getting into the respective stores with authentic binaries is to be yearned for.
With regard to Pedro's comments about Python, there is no reason to use IronPython, which is a .NET system, similarly with how Jython operate in and for Java.
However, native (Windows) Python is now quite well integrated into Microsoft Visual Studio and usable from Microsoft Visual Code (and used in its implementation). Also, so long as we are talking about command-line running and not any need for GUI libraries, using a stock Python would preserve great cross-platform consistency.
Some folks kvetch that Python 3.x is still not widely used for this kind of purpose. I don't think that is something to worry about.
I would love for cygWin to disappear on Windows, most of all. The clash of system models is just too brittle and inscrutable.
Finally: This is not a tiny effort. There is exquisite need to work confined changes, confirmation, and QA for a pro-longed, sustainable period. I think it is critical that if the effort is abandoned or goes dormant for any reason, the project survives in no worse shape. There is a serious "do no harm" imperative.
Having said that, and respecting the quality of the first-level analysis that Damjan has provided, I am eager to know more on how this could be evolved toward.
- Dennis
> -----Original Message-----
> From: Marcus [mailto:marcus.mail@wtnet.de]
> Sent: Friday, February 5, 2016 10:55
> To: dev@openoffice.apache.org
> Subject: Re: Thoughts on a new build system for AOO
>
> Am 02/05/2016 07:32 PM, schrieb Damjan Jovanovic:
[ ... ]
> > If people are happy, I'd suggest a migration plan where first scons is
> > added as a dependency, then the new version of serf is built with it
> (see
> > i126312), then gbuild modules are ported to scons until gbuild is
> > eliminated, then dmake modules are ported to scons until dmake is
> > eliminated, and finally scons replaces build.pl.
> >
> > Thoughts?
>
> thanks for your analysis and suggestion.
>
> I'm not a developer in the traditional way. So, I don't know anything
> about build systems and I've to believe your points - especially about
> the dis-/advantages for gbuild and scons.
>
> To migrate the gbuild-using modules first is clever as it would be a
> nightmare to have a growing mix of dmake, gbuild and scons together.
>
> Eliminating the cygwin stuff on Windows could be another big advantage
> for developers on Windows.
>
> Last but not least the famous question about participation. If you would
> volunteer to start/do the migration then it would be great. But in
> parallel it could also become a life time task. ;-P.
>
> But, yes, using scons sounds a good idea and doing a test with some
> modules that are working with gbuild could be a good start.
>
> Marcus
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Thoughts on a new build system for AOO
Posted by Marcus <ma...@wtnet.de>.
Am 02/05/2016 07:32 PM, schrieb Damjan Jovanovic:
> Hi
>
> With the recent buildbot saga and Patricia's Windows building nightmare, a
> number of serious issues were highlighted which got me thinking about how
> our build system could be improved.
>
> The problems with building AOO were researched and documented years ago (
> https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good
> analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis
> which is worth a read.
>
> Their solution was gbuild: a build system using GNU make's new "eval"
> feature to rewrite the makefile from within, which would generate one giant
> dependency graph of every file in AOO and produce reliable, stable,
> deterministic, and maximally parallelized builds, with minimal rebuilding
> on changes. The problem is, the full benefits of gbuild can only be seen
> once every module is converted to gbuild, and with only about 10% having
> been converted so far, it doesn't even try to build a complete project-wide
> dependency graph, but rather gets invoked per-module just like dmake.
> Converting dmake modules to gbuild is very difficult: so far I only managed
> to convert main/formula, but main/animations was mysteriously segfaulting
> AOO...
>
> Besides, LO has completed converting every module to gbuild, guinea pigging
> gbuild perfectly for us ;-). How has it gone for them? One of our
> committers, jani, evaluated LO's gbuild in April 2013 in a thread here
> called "Make x Dmake x Build", where he said how it's "not particulary fast
> either", and called mkdir 431 times on a directory that already existed.
> And worst of all, gbuild uses its own language with horrible syntax (eg.
> LISP-like syntax, spaces around commas not allowed, blank lines sometimes
> mandatory and at other times forbidden), is almost completely undocumented,
> and very difficult to debug - a maintenance nightmare.
>
> So what other alternatives are there?
>
> There was a comment in
> https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements
> that got my attention:
>
> "One might consider using Python as a tool for _all_ non-standard build
> tasks. Python is easy to distribute and install on all platforms. However,
> it is also a very fat dependency and thus it is questionable, if it would
> be worth the effort just to get rid of cygwin (or another Unix environment
> on windows)."
>
> We already build and use Python, and there is a good build system written
> in Python: SCons (http://www.scons.org). It's modern yet well established,
> having started in the year 2000, and is used by a number of open source and
> commercial projects. It's MIT licensed so we can include it as a
> dependency. The build scripts are simple Python scripts and can use the
> Python language and libraries, though you don't need to know (much) Python
> to use it. It's well documented. It's debuggable and has options to print
> dependencies and explain actions taken. It supports all the operating
> systems and architectures we support (even unofficially), including Cygwin,
> *BSD, Solaris, OS/2. It's "local" package is designed for easy embedding
> into a pre-existing project so it runs straight after extraction with no
> need to install it system-wide, and is only 445 kB.
>
> It has many features we need or would benefit from:
> * It's able to parse included headers from C/C++ files and use them as
> dependencies.
> * It can generate Visual Studio project files.
> * Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex.
> * Builds zip and tar archives.
> * Just like gbuild, it has a global dependency view and can fully
> parallelize building files across module boundaries while maintaining
> stability.
> * Using Python APIs instead of *nix CLI tools might eventually result in us
> being able to ditch Cygwin entirely, not to mention better performance due
> to less subprocess spawning.
> * Unlike classical make tools which only use mtimes, it determines what to
> rebuild using several configurable options, including MD5 checksums of file
> contents, meaning changes which don't result in different binaries (eg.
> formatting or comment changes) will only recompile changed files and stop
> there, because it knows the object files are the same as before, so no
> linking or downstream rebuilding needs to be performed.
>
> In other words SCons does most of what gbuild is supposed to be good for,
> and more. I see it outperforming LO's pure gbuild, while being more
> readable, maintainable, debuggable, and a sight for sore eyes after the
> triple whammy of dmake, gbuild and build.pl we've endured for so long.
>
> If people are happy, I'd suggest a migration plan where first scons is
> added as a dependency, then the new version of serf is built with it (see
> i126312), then gbuild modules are ported to scons until gbuild is
> eliminated, then dmake modules are ported to scons until dmake is
> eliminated, and finally scons replaces build.pl.
>
> Thoughts?
thanks for your analysis and suggestion.
I'm not a developer in the traditional way. So, I don't know anything
about build systems and I've to believe your points - especially about
the dis-/advantages for gbuild and scons.
To migrate the gbuild-using modules first is clever as it would be a
nightmare to have a growing mix of dmake, gbuild and scons together.
Eliminating the cygwin stuff on Windows could be another big advantage
for developers on Windows.
Last but not least the famous question about participation. If you would
volunteer to start/do the migration then it would be great. But in
parallel it could also become a life time task. ;-P.
But, yes, using scons sounds a good idea and doing a test with some
modules that are working with gbuild could be a good start.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Thoughts on a new build system for AOO
Posted by Kay Schenk <ka...@gmail.com>.
On 02/05/2016 10:32 AM, Damjan Jovanovic wrote:
> Hi
>
> With the recent buildbot saga and Patricia's Windows building nightmare, a
> number of serious issues were highlighted which got me thinking about how
> our build system could be improved.
>
> The problems with building AOO were researched and documented years ago (
> https://wiki.openoffice.org/wiki/Build_Environment_Effort) and a very good
> analysis was done in https://wiki.openoffice.org/wiki/Build_System_Analysis
> which is worth a read.
>
> Their solution was gbuild: a build system using GNU make's new "eval"
> feature to rewrite the makefile from within, which would generate one giant
> dependency graph of every file in AOO and produce reliable, stable,
> deterministic, and maximally parallelized builds, with minimal rebuilding
> on changes. The problem is, the full benefits of gbuild can only be seen
> once every module is converted to gbuild, and with only about 10% having
> been converted so far, it doesn't even try to build a complete project-wide
> dependency graph, but rather gets invoked per-module just like dmake.
> Converting dmake modules to gbuild is very difficult: so far I only managed
> to convert main/formula, but main/animations was mysteriously segfaulting
> AOO...
>
> Besides, LO has completed converting every module to gbuild, guinea pigging
> gbuild perfectly for us ;-). How has it gone for them? One of our
> committers, jani, evaluated LO's gbuild in April 2013 in a thread here
> called "Make x Dmake x Build", where he said how it's "not particulary fast
> either", and called mkdir 431 times on a directory that already existed.
> And worst of all, gbuild uses its own language with horrible syntax (eg.
> LISP-like syntax, spaces around commas not allowed, blank lines sometimes
> mandatory and at other times forbidden), is almost completely undocumented,
> and very difficult to debug - a maintenance nightmare.
>
> So what other alternatives are there?
>
> There was a comment in
> https://wiki.openoffice.org/wiki/Build_Environment_Effort/New_Build_System_Requirements
> that got my attention:
>
> "One might consider using Python as a tool for _all_ non-standard build
> tasks. Python is easy to distribute and install on all platforms. However,
> it is also a very fat dependency and thus it is questionable, if it would
> be worth the effort just to get rid of cygwin (or another Unix environment
> on windows)."
>
> We already build and use Python, and there is a good build system written
> in Python: SCons (http://www.scons.org). It's modern yet well established,
> having started in the year 2000, and is used by a number of open source and
> commercial projects. It's MIT licensed so we can include it as a
> dependency. The build scripts are simple Python scripts and can use the
> Python language and libraries, though you don't need to know (much) Python
> to use it. It's well documented. It's debuggable and has options to print
> dependencies and explain actions taken. It supports all the operating
> systems and architectures we support (even unofficially), including Cygwin,
> *BSD, Solaris, OS/2. It's "local" package is designed for easy embedding
> into a pre-existing project so it runs straight after extraction with no
> need to install it system-wide, and is only 445 kB.
>
> It has many features we need or would benefit from:
> * It's able to parse included headers from C/C++ files and use them as
> dependencies.
> * It can generate Visual Studio project files.
> * Supports many languages including C, C++, D, Java, Fortran, Yacc, Lex.
> * Builds zip and tar archives.
> * Just like gbuild, it has a global dependency view and can fully
> parallelize building files across module boundaries while maintaining
> stability.
> * Using Python APIs instead of *nix CLI tools might eventually result in us
> being able to ditch Cygwin entirely, not to mention better performance due
> to less subprocess spawning.
> * Unlike classical make tools which only use mtimes, it determines what to
> rebuild using several configurable options, including MD5 checksums of file
> contents, meaning changes which don't result in different binaries (eg.
> formatting or comment changes) will only recompile changed files and stop
> there, because it knows the object files are the same as before, so no
> linking or downstream rebuilding needs to be performed.
>
> In other words SCons does most of what gbuild is supposed to be good for,
> and more. I see it outperforming LO's pure gbuild, while being more
> readable, maintainable, debuggable, and a sight for sore eyes after the
> triple whammy of dmake, gbuild and build.pl we've endured for so long.
>
> If people are happy, I'd suggest a migration plan where first scons is
> added as a dependency, then the new version of serf is built with it (see
> i126312), then gbuild modules are ported to scons until gbuild is
> eliminated, then dmake modules are ported to scons until dmake is
> eliminated, and finally scons replaces build.pl.
>
> Thoughts?
>
> Damjan
>
I am not a build architecture guru, but I will share my thoughts
anyway. As an open source project relying on volunteers, I think we
need to consider sustainability of whatever we use for building or
anything else. So this means considering what potential volunteers
have learned to use in building applications.
Our current build system is at its heart, dmake, with build.pl on
top of that to deal with integrating the non-C++ aspects into
OpenOffice and provide a means of parallel build capabilities,
unless I'm very mistaken about this. Right now, the Windows parts
are using cmake as well. What we are calling "gbuild" is using GNU
make to build. If I'm not mistaken this is a name internal to
OpenOffice and LibreOffice. And, yes, we have been slow in this
migration.
In addition to the wiki pages already referenced, we have a short
little wiki page which gives a quick overview of our current build
system:
https://wiki.openoffice.org/wiki/Build_System#Drawbacks
As far as I can discover, VisualStudio can be setup to use GNU make
for building, (see for example:
www.bordoon.com/cplusplus/VisualStudioIntegration.pdf) so I don't
see using "gbuild" as a show stopper for GUI IDEs.
My thoughts at this point are to TRY to stick with GNU Make
(gbuild), and see if we can't remove code sections which no longer
really need to be build with build.pl. I hate the use of IDLs for
header generation and would like to see us "staticize" as much as
possible.
I don't know what it would take to move our build into SCONs, or how
well it would scale for our codebase. We'd obviously need some
trials on this.
--
--------------------------------------------
MzK
"Though no one can go back and make a brand new start,
anyone can start from now and make a brand new ending."
-- Carl Bard
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org