gbuild porting guide

[Damjan Jovanovic <da...@apache.org>]

Hi

Summary notes from the trenches, for porting to gbuild. This is meant
to be read in addition to
https://wiki.openoffice.org/wiki/Build_Environment_Effort/Module_Migration

You are not only welcome but encouraged to help, and I am happy to
review patches and answer questions, but please let's first coordinate
porting on the dev mailing list so we don't duplicate work.

----------------------------
build.pl and dmake
----------------------------
1. We still build with build.pl, which means execution always starts
in main/<module>/prj/build.lst. That file should have only 2 lines,
the first exactly as it is already, and the second running "nmake" in
that module's prj directory. See any gbuild module for an example -
you can find them by running "ls */prj/makefile.mk" from main/.

2. Using that second line in build.pl, dmake runs on
main/<module>/prj/makefile.mk, which is exactly the same in all gbuild
modules, and be copied from any. All that this file does is get dmake
to run GNU make, which we will examine just now.

3. After everything is done - and only if done successfully - build.pl
will run "deliver" which runs the commands in main/<module>/prj/d.lst.
This file will be obsolete with gbuild, and need to be an empty, 0
byte long file, in all gbuild modules for now. What it does in dmake
modules, needs to be done by GNU make in its makefiles instead, when
converted to gbuild.

Your first step isn't writing build scripts, it's intelligence
gathering. Thoroughly read build.lst and see what directories it goes
into and what action it runs in them  (the "usr1" action seems to be a
no-op, "nmake" runs dmake, the "get" action is unclear to me but also
seems to be a no-op). Thoroughly read d.lst and understand what is
being copied to solver and from where. Thoroughly read all of dmake's
makefile.mks in that module. Read the .map files to see which symbols
binary deliverables are exporting. Understand all the dmake variables
used. Build a complete picture of what is being done. Ideally draw a
picture of all files and actions.

Dmake's makefile.mk files are full of gotchas, obscure settings, and
strange behaviour. It seems like in AOO building with dmake is only
done on 1 directory of files at a time, and when files from multiple
directories need to be compiled into one deliverable, each directory
is built into its own static library, and a separate directory links
these static libraries together into the final deliverable :-D. We can
do better. See my main/package patch(es?) for an example that
eliminated these intermediate static libraries.

-----------------
GNU make
-----------------
1. In step 2 above, GNU make is invoked from dmake, and it runs the
main/<module>/Makefile which is the same in all gbuild modules.

2. That sets up gbuild and runs main/<module>/Module_<module>.mk which
you need to write. Modules can (and usually do) contain multiple
deliverables. By convention, each deliverable is in a separate file,
named <DeliverableType>_<DeliverableName>.mk (eg.
Library_fileaccess.mk, Package_inc.mk), and these are included into
the module's Module_<module>.mk using gb_Module_add_targets and the
like.

Inclusion of these deliverable makefiles can be conditional, happening
only on a certain operating system or only if certain dependencies are
present. This is accomplished in the Module_<module>.mk, using GNU
make's if conditions and only calling gb_Module_add_targets if those
conditions are met (note that in general the *add* functions like
*_add_targets can be called multiple times, but the *set* ones usually
can't). See module/x11_extensions/Module_x11_extensions.mk for a
simple example of how to only run actions on *nix.

3. Deliverables are the sole reason modules exist, and all the actual
action happens inside the makefiles for those deliverables. So what
types of deliverables do we have, how are they used, and what do they
do?

The easiest to understand is the "Package" deliverable, which simply
copies files to the solver. The easiest modules to understand are
those that do nothing but package files, such as main/MathMLDTD and
main/x11_extensions. The Package action pretty much does what the old
main/<module>/prj/d.lst file did. Note that gbuild will automatically
package other types of deliverables such as libraries and executables,
and only files that gbuild doesn't have specific deliverable actions
for have to be packaged explicitly. These are typically:
* Header files, for modules that provide them for use by other
modules, are typically in a Package_inc.mk. Header files don't have to
be packaged for the module to be able to use them to compile, they
only have to be packaged so other modules can use them, which isn't
always necessary.
* DTD files are typically in Package_dtd.mk
* XML files are typically in Package_xml.mk
* Any other custom files you wish to copy.

The "Library" deliverable compiles a shared library, on Window it's a
DLL. There's also a "StaticLibrary" for static libraries.

It is mandatory to also register every library in main/Repository.mk.
Libraries come in several "groups" or "layers", listed in
main/Repository.mk, and the group affects the way the library is named
and its RPATH. Look at the openoffice/program directory of a *nix AOO
installation. The lib<name>.so are usually in the OOOLIBS group, while
the <name>.uno.so are usually in the UNOLIBS_OOO group, and the
lib<name>gcc3.so are usually in the RTLIBS group. Note that this is
just a naming convention. UNO libraries can be in the OOOLIBS group,
and some are. Also main/RepositoryFixes.mk hacks some names...

UNO libraries have a .component file which you specify using
gb_Library_set_componentfile. Dmake did it in a much uglier way, with
a long make rule. Note that for such libraries, you have to update
main/postprocess/packcomponents/makefile.mk and change the component
name in the my_components variable from eg. binaryurp to
component/binaryurp/source/binaryurp (this comes from the path within
the module). For some components, this has to be done in the
undocumented main/urp/source/makefile.mk too.

gb_*_add_exception_objects is for C++ files that are to be compiled
with exception handling enabled, and gb_*_add_noexception_objects is
for those to be compiled with exception handling disabled. How do you
know which to use? ENABLE_EXCEPTIONS in makefile.mk, which is off by
default. Other supported languages for libraries and executables
include C and Objective C which have their own actions, while Java has
a separate "Jar" deliverable type which is currently only used in
main/wizards and relatively easy to understand compared to the C++
"Library". Listing a source filename that doesn't exist currently
results in an infinite loop when building, so be careful!

When using add_linked_libs, you have the use gbuild's naming (eg. sal)
instead of dmake's (eg. $(SALLIB)). It's not always that obvious.

Precompiled headers must either be empty, or 1 per deliverable,
instead of 1 per module. It is a build-breaking error - on Windows
only (since other platforms don't use precompiled headers) - to use
the same non-empty precompiled header in 2 or more deliverables.

Gbuild modules have to link with "stl" on Windows, or you get
mysterious link errors.

Gbuild changed from dmake's exporting all symbols by default and
having to hide unwanted ones using a .map file, to hiding all symbols
by default and having to export the wanted ones with
SAL_DLLPUBLIC_EXPORT or the like. Yes, you are editing C++ code for
this, not just makefiles. If you forget to export, the build will
usually break on Windows but succeed on *nix and presumably fail at
runtime.

"Executable" is for applications. These also have groups and also need
to be registered in main/Repository.mk.

"GoogleTest" is for C/C++ unit tests that run during the build. It's
very similar to Executable, because it builds executables that run the
tests. (After porting our unit tests from cppunit to Google Test, I
discovered it's possible to do it through libraries too lol.)
"JunitTest" is for "subsequent checks" that can be run after AOO is
built with "make subsequentcheck" in the module directory.

"AllLangResTarget" is for translations. These come from dmake's .src
files. See main/formula, main/uui and others for examples.

When a module is ported to gbuild, you must add it to
main/Module_ooo.mk, I think this lets it participate in global actions
like "make clean".

As you can see, AOO uses an advanced, complex build system with many
programming languages, much custom infrastructure, custom compilers,
custom source documentation generation, custom translations, etc. The
gbuild makefiles in solenv/gbuild are currently around 8000 lines and
still incomplete. Using another build system instead of gbuild would
first require porting all this build logic to that build system. And
as should be clear this far, the dmake->gbuild conversion involves
multiple changes: changes to symbol visibility, elimination of
build.pl, elimination of dmake, elimination of deliver, changes to
syntax, changes to the naming of dependencies, changes to include
paths, etc. Any gbuild->other migration will be a lot easier by
comparison!

Syntax is a bit of a pain. You can't put spaces around commas, have to
end non-final lines in a definition with a backslash, etc. On the
whole though, gbuild makefiles for modules are fewer in number,
clearer and more readable than dmake's, and only found in the module's
root directory instead of hidden in subdirectories. The high level
declarative syntax is also a pleasure to work with compared to
dmake's.

When in doubt, read other modules' makefiles, not the almost
indecipherable gbuild code.

---------------
The future
---------------
52 of 182 modules (29%) are currently using gbuild, up from 40 (22%)
before my recent conversion burst began.

I am hacking and porting as many modules to gbuild as possible, as
soon as possible. The most important thing right now is to build up
experience and document as much of gbuild as possible (and sadly
dmake/build.pl/deliver, since understanding their files comes first),
to prepare for tackling the large modules and making changes to the
gbuild makefiles themselves.

The UNO modules seem easier to convert and many are really small. I am
currently converting the Java modules, and javaunohelper should be the
next to get committed.

I think another group needs to be added for libraries like
libstore.so.3 whose naming format isn't provided by any library group.
Yacc support also needs adding to gbuild for several modules that use
it to parse languages (IDL compilers, main/connectivity for parsing
SQL queries, etc.), and other gbuild infrastructure probably needs
developing further.

It seems Sun was trying to implement a "split build" before all the
modules were ported, where modules in first part are built with
build.pl, and those in the second part are built purely with a single
GNU make instance that can build multiple modules concurrently at a
file level, with maximum parallelization to give the highest
performance ;-), and would no longer need the prj/ subdirectories.
It's not clear to me how they were planning to do this or on exactly
which modules (the "late" modules would have been the first to go into
this gbuild-only part, of which Calc is the last of the "big 6" that
aren't in gbuild yet), but I can't wait to start telling dmake and
build.pl goodbye!

Happy hacking
Damjan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org

Re: gbuild porting guide

[Peter Kovacs <le...@gmail.com>]

Hi Damjan,

This is really awesome.
I will support you the best I can with intelligence.
I am a bit lazy, so I wrote a programme that indexes all files, I want to
go through all build configuration files, c++ and java file and see if I
can consolidate all this information into Wikipages to start developers
documentation on on our confluence wiki. My plan / hope is I can provide a
Dependency tree on filebases until end of this year. Next step will be
extended the documentation by classes and Namespaces and last I would like
to add function and methods.
I am sorry that I will not be faster. A lot to do currently.

All the best
Peter

Marcus <ma...@wtnet.de> schrieb am Sa., 10. Dez. 2016, 10:35:

> Am 12/09/2016 06:37 PM, schrieb Damjan Jovanovic:
> >
> > [...]
> >
> > ---------------
> > The future
> > ---------------
> > 52 of 182 modules (29%) are currently using gbuild, up from 40 (22%)
> > before my recent conversion burst began.
> >
> > I am hacking and porting as many modules to gbuild as possible, as
> > soon as possible. The most important thing right now is to build up
> > experience and document as much of gbuild as possible (and sadly
> > dmake/build.pl/deliver, since understanding their files comes first),
> > to prepare for tackling the large modules and making changes to the
> > gbuild makefiles themselves.
> >
> > The UNO modules seem easier to convert and many are really small. I am
> > currently converting the Java modules, and javaunohelper should be the
> > next to get committed.
> >
> > I think another group needs to be added for libraries like
> > libstore.so.3 whose naming format isn't provided by any library group.
> > Yacc support also needs adding to gbuild for several modules that use
> > it to parse languages (IDL compilers, main/connectivity for parsing
> > SQL queries, etc.), and other gbuild infrastructure probably needs
> > developing further.
> >
> > It seems Sun was trying to implement a "split build" before all the
> > modules were ported, where modules in first part are built with
> > build.pl, and those in the second part are built purely with a single
> > GNU make instance that can build multiple modules concurrently at a
> > file level, with maximum parallelization to give the highest
> > performance ;-), and would no longer need the prj/ subdirectories.
> > It's not clear to me how they were planning to do this or on exactly
> > which modules (the "late" modules would have been the first to go into
> > this gbuild-only part, of which Calc is the last of the "big 6" that
> > aren't in gbuild yet), but I can't wait to start telling dmake and
> > build.pl goodbye!
>
> unfortunately, I cannot provide you with much help. The only thing I can
> do is bulding the beast and report back problems.
>
> But besides this I like your work. It's great to see a successful way
> towards a single and improved build system.
>
> In the meantime we could paint a "Goodbye" sign for the special day. :-)
>
> Again, thanks for your effort.
>
> Marcus
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
> --

Disclaimer: Diese Nachricht stammt aus einem Google Account. Ihre Antwort
wird in der Google Cloud Gespeichert und durch Google Algorythmen zwecks
werbeanaöysen gescannt. Es ist derzeit nicht auszuschließen das ihre
Nachricht auch durch einen NSA Mitarbeiter geprüft wird. Durch
kommunikation mit diesen Account stimmen Sie zu das ihre Mail, ihre
Kontaktdaten und die Termine die Sie mit mir vereinbaren online zu Google
konditionen in der Googlecloud gespeichert wird. Sollten sie dies nicht
wünschen kontaktieren sie mich bitte Umgehend um z.B. alternativen zu
verhandeln.

Re: gbuild porting guide

[Marcus <ma...@wtnet.de>]

Am 12/09/2016 06:37 PM, schrieb Damjan Jovanovic:
>
> [...]
>
> ---------------
> The future
> ---------------
> 52 of 182 modules (29%) are currently using gbuild, up from 40 (22%)
> before my recent conversion burst began.
>
> I am hacking and porting as many modules to gbuild as possible, as
> soon as possible. The most important thing right now is to build up
> experience and document as much of gbuild as possible (and sadly
> dmake/build.pl/deliver, since understanding their files comes first),
> to prepare for tackling the large modules and making changes to the
> gbuild makefiles themselves.
>
> The UNO modules seem easier to convert and many are really small. I am
> currently converting the Java modules, and javaunohelper should be the
> next to get committed.
>
> I think another group needs to be added for libraries like
> libstore.so.3 whose naming format isn't provided by any library group.
> Yacc support also needs adding to gbuild for several modules that use
> it to parse languages (IDL compilers, main/connectivity for parsing
> SQL queries, etc.), and other gbuild infrastructure probably needs
> developing further.
>
> It seems Sun was trying to implement a "split build" before all the
> modules were ported, where modules in first part are built with
> build.pl, and those in the second part are built purely with a single
> GNU make instance that can build multiple modules concurrently at a
> file level, with maximum parallelization to give the highest
> performance ;-), and would no longer need the prj/ subdirectories.
> It's not clear to me how they were planning to do this or on exactly
> which modules (the "late" modules would have been the first to go into
> this gbuild-only part, of which Calc is the last of the "big 6" that
> aren't in gbuild yet), but I can't wait to start telling dmake and
> build.pl goodbye!

unfortunately, I cannot provide you with much help. The only thing I can 
do is bulding the beast and report back problems.

But besides this I like your work. It's great to see a successful way 
towards a single and improved build system.

In the meantime we could paint a "Goodbye" sign for the special day. :-)

Again, thanks for your effort.

Marcus


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org