You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Andre Fischer <aw...@gmail.com> on 2014/01/22 14:32:16 UTC
Building sw/ with ninja
Not quite a week ago I wrote about an idea to use XML files to store the
declarative part of our makefiles: dependencies of libraries on source
files, which resources are to be created and so on. In the meantime I
have found the time to do make (conduct?) an experiment. I am now able
to build module sw from the XML files with the help of the ninja build
'system' [3]. Most of the work of converting the XML files into one
single build.ninja file was done on one weekend. You can see the source
code at [1] ([2] contains everything zipped together).
The results are promising. It runs faster and the build.ninja generator
looks more maintainable than our solenv/gbuild/... makefiles. But I am
certainly biased.
Before I give you some numbers, I should say that I have collected the
numbers totally unscientifically and it may be necessary to add some
missing steps to the ninja build. To the best of my knowledge all C++
files are compiled, libraries linked, resource files built, XML files
copied. Only the single sw.component file somehow escaped.
I ran my experiments on ani7 2.2GHz, 8GB notebook.
Complete build of a clean module:
gbuild about 9m30s (make -sr -j8)
ninja about 7m15s (ninja)
Cleaning up
gbuild about 40s (make clean)
ninja less then 1s (ninja -t clean)
rebuild after touching one single header (sw/inc/section.hxx)
gbuild about 1m10s (make -sr -j8)
ninja about 50s (ninja)
Building an already built module (nothing to do): depends very much on
whether the disk cache is warm or cold. Best times:
gbuild more than 3s (make -sr -j8)
ninja about 0.4s (ninja)
Why is ninja faster than make/gbuild?
- Make runs each recipe in its own shell (bash), ninja executes its
command directly.
- Ninja understands the header dependencies created by gxx/clang and
msvc and stores them in a compact format that can be read in very fast
on startup.
- I avoided some steps of build that are unnecessary in ninja
= Ninja creates directories for the targets it makes. Gbuild creates
them explicitly.
= GBuild first creates empty dependency files and later, in a second
step, fills them with the actual dependency information created by one
of the C/C++ compilers.
But, for me, these numbers are just a welcome side effect. More
important to me is maintainability.
Ninja follows a very different approach from (GNU) make. Its lack of
even simplest control structures such as if/then/else or foreach,
requires the generation of the main makefile (by default that is called
build.ninja) by program or script. This leads to my current approach:
- Use XML to represent the static data (C++ files, libraries, resource
files, XML files).
- Use a Perl script to translate the XML files into the build.ninja file.
The best tool for each job (XML: data representation, Perl: data
processing). Instead of Perl we could use any language that is part of
our current build requirements (Java, C/C++, Python (we would have to
compile that first, though)). Look at the Perl files in [1] or [2]
(build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see
which you can understand better.
I think this could be one way to set up a better maintainable build
system that is even slightly faster then what we currently have.
Best regards,
Andre
[1] http://people.apache.org/~af/build/
[2] http://people.apache.org/build.zip
[3] http://martine.github.io/ninja/manual.html
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Andre Fischer <aw...@gmail.com>.
On 23.01.2014 19:23, Kay Schenk wrote:
> On Wed, Jan 22, 2014 at 5:32 AM, Andre Fischer <aw...@gmail.com> wrote:
>
>> Not quite a week ago I wrote about an idea to use XML files to store the
>> declarative part of our makefiles: dependencies of libraries on source
>> files, which resources are to be created and so on. In the meantime I have
>> found the time to do make (conduct?) an experiment. I am now able to build
>> module sw from the XML files with the help of the ninja build 'system' [3].
>> Most of the work of converting the XML files into one single build.ninja
>> file was done on one weekend. You can see the source code at [1] ([2]
>> contains everything zipped together).
>>
> I think link [2] needs to be:
>
> [2] http://people.apache.org/~af/build.zip<http://people.apache.org/build.zip>
>
> instead of --
>
> http://people.apache.org/build.zip <http://people.apache.org/build.zip>
Yes, thank you.
>
> I am looking forward to checking this out.
>
Great, but please be aware that this is just an experiment. At the
moment it only works on Windows.
-Andre
>
>
>
>
>
>> The results are promising. It runs faster and the build.ninja generator
>> looks more maintainable than our solenv/gbuild/... makefiles. But I am
>> certainly biased.
>> Before I give you some numbers, I should say that I have collected the
>> numbers totally unscientifically and it may be necessary to add some
>> missing steps to the ninja build. To the best of my knowledge all C++
>> files are compiled, libraries linked, resource files built, XML files
>> copied. Only the single sw.component file somehow escaped.
>>
>> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>>
>> Complete build of a clean module:
>> gbuild about 9m30s (make -sr -j8)
>> ninja about 7m15s (ninja)
>>
>> Cleaning up
>> gbuild about 40s (make clean)
>> ninja less then 1s (ninja -t clean)
>>
>> rebuild after touching one single header (sw/inc/section.hxx)
>> gbuild about 1m10s (make -sr -j8)
>> ninja about 50s (ninja)
>>
>> Building an already built module (nothing to do): depends very much on
>> whether the disk cache is warm or cold. Best times:
>> gbuild more than 3s (make -sr -j8)
>> ninja about 0.4s (ninja)
>>
>>
>> Why is ninja faster than make/gbuild?
>> - Make runs each recipe in its own shell (bash), ninja executes its
>> command directly.
>> - Ninja understands the header dependencies created by gxx/clang and msvc
>> and stores them in a compact format that can be read in very fast on
>> startup.
>> - I avoided some steps of build that are unnecessary in ninja
>> = Ninja creates directories for the targets it makes. Gbuild creates
>> them explicitly.
>> = GBuild first creates empty dependency files and later, in a second
>> step, fills them with the actual dependency information created by one of
>> the C/C++ compilers.
>>
>>
>> But, for me, these numbers are just a welcome side effect. More important
>> to me is maintainability.
>> Ninja follows a very different approach from (GNU) make. Its lack of even
>> simplest control structures such as if/then/else or foreach, requires the
>> generation of the main makefile (by default that is called build.ninja) by
>> program or script. This leads to my current approach:
>> - Use XML to represent the static data (C++ files, libraries, resource
>> files, XML files).
>> - Use a Perl script to translate the XML files into the build.ninja file.
>> The best tool for each job (XML: data representation, Perl: data
>> processing). Instead of Perl we could use any language that is part of our
>> current build requirements (Java, C/C++, Python (we would have to compile
>> that first, though)). Look at the Perl files in [1] or [2]
>> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see
>> which you can understand better.
>>
>>
>> I think this could be one way to set up a better maintainable build system
>> that is even slightly faster then what we currently have.
>>
>> Best regards,
>> Andre
>>
>>
>> [1] http://people.apache.org/~af/build/
>> [2] http://people.apache.org/build.zip
>> [3] http://martine.github.io/ninja/manual.html
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Jan 22, 2014 at 5:32 AM, Andre Fischer <aw...@gmail.com> wrote:
> Not quite a week ago I wrote about an idea to use XML files to store the
> declarative part of our makefiles: dependencies of libraries on source
> files, which resources are to be created and so on. In the meantime I have
> found the time to do make (conduct?) an experiment. I am now able to build
> module sw from the XML files with the help of the ninja build 'system' [3].
> Most of the work of converting the XML files into one single build.ninja
> file was done on one weekend. You can see the source code at [1] ([2]
> contains everything zipped together).
>
I think link [2] needs to be:
[2] http://people.apache.org/~af/build.zip<http://people.apache.org/build.zip>
instead of --
http://people.apache.org/build.zip <http://people.apache.org/build.zip>
I am looking forward to checking this out.
>
> The results are promising. It runs faster and the build.ninja generator
> looks more maintainable than our solenv/gbuild/... makefiles. But I am
> certainly biased.
> Before I give you some numbers, I should say that I have collected the
> numbers totally unscientifically and it may be necessary to add some
> missing steps to the ninja build. To the best of my knowledge all C++
> files are compiled, libraries linked, resource files built, XML files
> copied. Only the single sw.component file somehow escaped.
>
> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>
> Complete build of a clean module:
> gbuild about 9m30s (make -sr -j8)
> ninja about 7m15s (ninja)
>
> Cleaning up
> gbuild about 40s (make clean)
> ninja less then 1s (ninja -t clean)
>
> rebuild after touching one single header (sw/inc/section.hxx)
> gbuild about 1m10s (make -sr -j8)
> ninja about 50s (ninja)
>
> Building an already built module (nothing to do): depends very much on
> whether the disk cache is warm or cold. Best times:
> gbuild more than 3s (make -sr -j8)
> ninja about 0.4s (ninja)
>
>
> Why is ninja faster than make/gbuild?
> - Make runs each recipe in its own shell (bash), ninja executes its
> command directly.
> - Ninja understands the header dependencies created by gxx/clang and msvc
> and stores them in a compact format that can be read in very fast on
> startup.
> - I avoided some steps of build that are unnecessary in ninja
> = Ninja creates directories for the targets it makes. Gbuild creates
> them explicitly.
> = GBuild first creates empty dependency files and later, in a second
> step, fills them with the actual dependency information created by one of
> the C/C++ compilers.
>
>
> But, for me, these numbers are just a welcome side effect. More important
> to me is maintainability.
> Ninja follows a very different approach from (GNU) make. Its lack of even
> simplest control structures such as if/then/else or foreach, requires the
> generation of the main makefile (by default that is called build.ninja) by
> program or script. This leads to my current approach:
> - Use XML to represent the static data (C++ files, libraries, resource
> files, XML files).
> - Use a Perl script to translate the XML files into the build.ninja file.
> The best tool for each job (XML: data representation, Perl: data
> processing). Instead of Perl we could use any language that is part of our
> current build requirements (Java, C/C++, Python (we would have to compile
> that first, though)). Look at the Perl files in [1] or [2]
> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see
> which you can understand better.
>
>
> I think this could be one way to set up a better maintainable build system
> that is even slightly faster then what we currently have.
>
> Best regards,
> Andre
>
>
> [1] http://people.apache.org/~af/build/
> [2] http://people.apache.org/build.zip
> [3] http://martine.github.io/ninja/manual.html
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"Cats do not have to be shown how to have a good time,
for they are unfailing ingenious in that respect."
-- James Mason
Re: Building sw/ with ninja
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 1/22/14 2:45 PM, Rob Weir wrote:
> On Wed, Jan 22, 2014 at 8:32 AM, Andre Fischer <aw...@gmail.com> wrote:
>> Not quite a week ago I wrote about an idea to use XML files to store the
>> declarative part of our makefiles: dependencies of libraries on source
>> files, which resources are to be created and so on. In the meantime I have
>> found the time to do make (conduct?) an experiment. I am now able to build
>> module sw from the XML files with the help of the ninja build 'system' [3].
>> Most of the work of converting the XML files into one single build.ninja
>> file was done on one weekend. You can see the source code at [1] ([2]
>> contains everything zipped together).
>>
>> The results are promising. It runs faster and the build.ninja generator
>> looks more maintainable than our solenv/gbuild/... makefiles. But I am
>> certainly biased.
>> Before I give you some numbers, I should say that I have collected the
>> numbers totally unscientifically and it may be necessary to add some missing
>> steps to the ninja build. To the best of my knowledge all C++ files are
>> compiled, libraries linked, resource files built, XML files copied. Only
>> the single sw.component file somehow escaped.
>>
>> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>>
>> Complete build of a clean module:
>> gbuild about 9m30s (make -sr -j8)
>> ninja about 7m15s (ninja)
>>
>> Cleaning up
>> gbuild about 40s (make clean)
>> ninja less then 1s (ninja -t clean)
>>
>> rebuild after touching one single header (sw/inc/section.hxx)
>> gbuild about 1m10s (make -sr -j8)
>> ninja about 50s (ninja)
>>
>> Building an already built module (nothing to do): depends very much on
>> whether the disk cache is warm or cold. Best times:
>> gbuild more than 3s (make -sr -j8)
>> ninja about 0.4s (ninja)
>>
>>
>> Why is ninja faster than make/gbuild?
>> - Make runs each recipe in its own shell (bash), ninja executes its command
>> directly.
>> - Ninja understands the header dependencies created by gxx/clang and msvc
>> and stores them in a compact format that can be read in very fast on
>> startup.
>> - I avoided some steps of build that are unnecessary in ninja
>> = Ninja creates directories for the targets it makes. Gbuild creates them
>> explicitly.
>> = GBuild first creates empty dependency files and later, in a second step,
>> fills them with the actual dependency information created by one of the
>> C/C++ compilers.
>>
>>
>> But, for me, these numbers are just a welcome side effect. More important
>> to me is maintainability.
>> Ninja follows a very different approach from (GNU) make. Its lack of even
>> simplest control structures such as if/then/else or foreach, requires the
>> generation of the main makefile (by default that is called build.ninja) by
>> program or script. This leads to my current approach:
>> - Use XML to represent the static data (C++ files, libraries, resource
>> files, XML files).
>> - Use a Perl script to translate the XML files into the build.ninja file.
>> The best tool for each job (XML: data representation, Perl: data
>> processing). Instead of Perl we could use any language that is part of our
>> current build requirements (Java, C/C++, Python (we would have to compile
>> that first, though)). Look at the Perl files in [1] or [2]
>> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see which
>> you can understand better.
>>
>>
>> I think this could be one way to set up a better maintainable build system
>> that is even slightly faster then what we currently have.
>>
>
> Do you get a sense for how well-maintained Ninja is? Are there many
> contributors? Many users? Are we confident it will be around in 5
> years? I worry (but only a little) of another DMake.
Ninjas are known that they can survive ;-) But it is indeed a valid
question. The fact that it is used by Google to build Chromium is at
least promising.
Juergen
>
> -Rob
>
>
>> Best regards,
>> Andre
>>
>>
>> [1] http://people.apache.org/~af/build/
>> [2] http://people.apache.org/build.zip
>> [3] http://martine.github.io/ninja/manual.html
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Andre Fischer <aw...@gmail.com>.
On 22.01.2014 17:28, Armin Le Grand wrote:
> Hi Andre,
>
> On 22.01.2014 07:06, Andre Fischer wrote:
>> On 22.01.2014 14:58, Andre Fischer wrote:
> ----8<-----8<-----
>>> I only know that it was developed by/for the chrome project [4] and
>>> that cmake has support for ninja as back end.
>>>
>>>> Are we confident it will be around in 5
>>>> years? I worry (but only a little) of another DMake.
>>>
>>> Then I probably should not tell you that I may make a similar
>>> experiment with tup as backend.
>>
>
> This is wonderful news, the build system is one of the 'blocking'
> factors for further development. Thank you very much for doing this
> experiments and driving this forward.
>
>> There are two different and independent ideas:
>>
>> - Use an easy to read data format for expressing module data (C++
>> files, libraries, etc.) that is independent from the actual build tool.
>>
>> - Use ninja as a back end.
>>
>> The first part is much easier to accomplish and enables us to make
>> experiments regarding different back ends.
>> Our current gbuild system would be a natural choice for the first
>> back end. Pure make might be the second. And maybe ninja would be
>> the third.
>
> I think that seperation is a very good approach. Backends could be the
> numbered ones, but also script-created stuff for eclipse and msdev
> (probably?).
Jan is working on a similar approach for msdev. Eclipse has its own
idea how building a project works. I can not say if that can,
eventually, mapped to our build system.
But I am currently working on a small Eclipse addon that provides a few
buttons (or menu entries, etc.) that start a build of the current
module, directory or file. That might be enough for the time being.
>
> We will have to make sure then that - when different ways exist to
> build the office - that we all still build the same. We already have
> issues when comparing stuff e.g. when looking into bugfixes on
> different systems, we do not really want this on the same system for
> different ways of building the office.
Good point. As final outcome we should have only one build system, the
ability to have different back ends is primarily interesting for
evaluating different replacements of the current system. At best there
could be a secondary build system for the integration into IDEs. But
for building releases and reporting bugs we should use only one build
system.
>
> This also means - if I get you right - that the creation of the needed
> build info from the xml description is an integral part of the build
> system, probably the first step. Thus, changing it (e.g. adding a new
> file to the AOO build tree) will (has to) always be done in the xml
> description. Then to rebuild the needed stuff - dependent of the build
> method, e.g. ninja - the build system needs to have a dependency that
> leads to recreation of the needed build data itself, then to the build
> of AOO.
Think of the XML files as a replacement with a different syntax for the
makefiles like Library_sw.mk. The only difference is that make can not
includes these files directly but has to translate them first into
makefiles. This could be done with a simple make rule like
Library_sw.mk : Library_sw.xml
xml2mk $< $@
and then include Library_sw.mk
For ninja this is already working in my experiment. Change one of the
xml files and build.ninja is rebuilt, included, and the updated build
rules executed.
>
> When (just dreaming) applying this to something like eclipe or msdev
> this would mean that the created data for these 'helpers' would need
> to be somehow automatically recreated/reloaded on the fly, would
> somehing like that be possibe...?
It depends on how much work one wants to invest. It would be possible
to write an addon for editing the xml files of the build system. It is
also possible, but more difficult, to add some hook that adds a new C++
file to the corresponding XML file when that C++ file is created via an
Eclipse wizard.
>
> I also see making the build faster as a nice side effect, better
> readability/maintainability is the biggest plus.
> And reliable global dependencies would be a big plus, too, of course...
>
> Let's drive this forward, Im ready to help converting modules when
> it's principally working and the transition step is defined!
Thank you, that is great to hear.
-Andre
>
> Sincerely,
> Armin
>
>>
>> -Andre
>>
>>>
>>> -Andre
>>>
>>>
>>> [4] http://www.aosabook.org/en/posa/ninja.html
>>> [5] http://gittup.org/tup/
>>>
>>>
> ----8<-----8<-----
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Armin Le Grand <Ar...@me.com>.
Hi Andre,
On 22.01.2014 07:06, Andre Fischer wrote:
> On 22.01.2014 14:58, Andre Fischer wrote:
----8<-----8<-----
>> I only know that it was developed by/for the chrome project [4] and
>> that cmake has support for ninja as back end.
>>
>>> Are we confident it will be around in 5
>>> years? I worry (but only a little) of another DMake.
>>
>> Then I probably should not tell you that I may make a similar
>> experiment with tup as backend.
>
This is wonderful news, the build system is one of the 'blocking'
factors for further development. Thank you very much for doing this
experiments and driving this forward.
> There are two different and independent ideas:
>
> - Use an easy to read data format for expressing module data (C++
> files, libraries, etc.) that is independent from the actual build tool.
>
> - Use ninja as a back end.
>
> The first part is much easier to accomplish and enables us to make
> experiments regarding different back ends.
> Our current gbuild system would be a natural choice for the first back
> end. Pure make might be the second. And maybe ninja would be the third.
I think that seperation is a very good approach. Backends could be the
numbered ones, but also script-created stuff for eclipse and msdev
(probably?).
We will have to make sure then that - when different ways exist to build
the office - that we all still build the same. We already have issues
when comparing stuff e.g. when looking into bugfixes on different
systems, we do not really want this on the same system for different
ways of building the office.
This also means - if I get you right - that the creation of the needed
build info from the xml description is an integral part of the build
system, probably the first step. Thus, changing it (e.g. adding a new
file to the AOO build tree) will (has to) always be done in the xml
description. Then to rebuild the needed stuff - dependent of the build
method, e.g. ninja - the build system needs to have a dependency that
leads to recreation of the needed build data itself, then to the build
of AOO.
When (just dreaming) applying this to something like eclipe or msdev
this would mean that the created data for these 'helpers' would need to
be somehow automatically recreated/reloaded on the fly, would somehing
like that be possibe...?
I also see making the build faster as a nice side effect, better
readability/maintainability is the biggest plus.
And reliable global dependencies would be a big plus, too, of course...
Let's drive this forward, Im ready to help converting modules when it's
principally working and the transition step is defined!
Sincerely,
Armin
>
> -Andre
>
>>
>> -Andre
>>
>>
>> [4] http://www.aosabook.org/en/posa/ninja.html
>> [5] http://gittup.org/tup/
>>
>>
----8<-----8<-----
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Rob Weir <ro...@apache.org>.
On Wed, Jan 22, 2014 at 9:06 AM, Andre Fischer <aw...@gmail.com> wrote:
> On 22.01.2014 14:58, Andre Fischer wrote:
>>
>> On 22.01.2014 14:45, Rob Weir wrote:
>>>
>>> On Wed, Jan 22, 2014 at 8:32 AM, Andre Fischer <aw...@gmail.com> wrote:
>>>>
>>>> Not quite a week ago I wrote about an idea to use XML files to store the
>>>> declarative part of our makefiles: dependencies of libraries on source
>>>> files, which resources are to be created and so on. In the meantime I
>>>> have
>>>> found the time to do make (conduct?) an experiment. I am now able to
>>>> build
>>>> module sw from the XML files with the help of the ninja build 'system'
>>>> [3].
>>>> Most of the work of converting the XML files into one single build.ninja
>>>> file was done on one weekend. You can see the source code at [1] ([2]
>>>> contains everything zipped together).
>>>>
>>>> The results are promising. It runs faster and the build.ninja generator
>>>> looks more maintainable than our solenv/gbuild/... makefiles. But I am
>>>> certainly biased.
>>>> Before I give you some numbers, I should say that I have collected the
>>>> numbers totally unscientifically and it may be necessary to add some
>>>> missing
>>>> steps to the ninja build. To the best of my knowledge all C++ files are
>>>> compiled, libraries linked, resource files built, XML files copied.
>>>> Only
>>>> the single sw.component file somehow escaped.
>>>>
>>>> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>>>>
>>>> Complete build of a clean module:
>>>> gbuild about 9m30s (make -sr -j8)
>>>> ninja about 7m15s (ninja)
>>>>
>>>> Cleaning up
>>>> gbuild about 40s (make clean)
>>>> ninja less then 1s (ninja -t clean)
>>>>
>>>> rebuild after touching one single header (sw/inc/section.hxx)
>>>> gbuild about 1m10s (make -sr -j8)
>>>> ninja about 50s (ninja)
>>>>
>>>> Building an already built module (nothing to do): depends very much on
>>>> whether the disk cache is warm or cold. Best times:
>>>> gbuild more than 3s (make -sr -j8)
>>>> ninja about 0.4s (ninja)
>>>>
>>>>
>>>> Why is ninja faster than make/gbuild?
>>>> - Make runs each recipe in its own shell (bash), ninja executes its
>>>> command
>>>> directly.
>>>> - Ninja understands the header dependencies created by gxx/clang and
>>>> msvc
>>>> and stores them in a compact format that can be read in very fast on
>>>> startup.
>>>> - I avoided some steps of build that are unnecessary in ninja
>>>> = Ninja creates directories for the targets it makes. Gbuild creates
>>>> them
>>>> explicitly.
>>>> = GBuild first creates empty dependency files and later, in a second
>>>> step,
>>>> fills them with the actual dependency information created by one of the
>>>> C/C++ compilers.
>>>>
>>>>
>>>> But, for me, these numbers are just a welcome side effect. More
>>>> important
>>>> to me is maintainability.
>>>> Ninja follows a very different approach from (GNU) make. Its lack of
>>>> even
>>>> simplest control structures such as if/then/else or foreach, requires
>>>> the
>>>> generation of the main makefile (by default that is called build.ninja)
>>>> by
>>>> program or script. This leads to my current approach:
>>>> - Use XML to represent the static data (C++ files, libraries, resource
>>>> files, XML files).
>>>> - Use a Perl script to translate the XML files into the build.ninja
>>>> file.
>>>> The best tool for each job (XML: data representation, Perl: data
>>>> processing). Instead of Perl we could use any language that is part of
>>>> our
>>>> current build requirements (Java, C/C++, Python (we would have to
>>>> compile
>>>> that first, though)). Look at the Perl files in [1] or [2]
>>>> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see
>>>> which
>>>> you can understand better.
>>>>
>>>>
>>>> I think this could be one way to set up a better maintainable build
>>>> system
>>>> that is even slightly faster then what we currently have.
>>>>
>>> Do you get a sense for how well-maintained Ninja is? Are there many
>>> contributors? Many users?
>>
>>
>> I only know that it was developed by/for the chrome project [4] and that
>> cmake has support for ninja as back end.
>>
>>> Are we confident it will be around in 5
>>> years? I worry (but only a little) of another DMake.
>>
>>
>> Then I probably should not tell you that I may make a similar experiment
>> with tup as backend.
>
>
> There are two different and independent ideas:
>
> - Use an easy to read data format for expressing module data (C++ files,
> libraries, etc.) that is independent from the actual build tool.
>
> - Use ninja as a back end.
>
> The first part is much easier to accomplish and enables us to make
> experiments regarding different back ends.
That's a fair point. Good design is always in style.
-Rob
> Our current gbuild system would be a natural choice for the first back end.
> Pure make might be the second. And maybe ninja would be the third.
>
> -Andre
>
>
>>
>> -Andre
>>
>>
>> [4] http://www.aosabook.org/en/posa/ninja.html
>> [5] http://gittup.org/tup/
>>
>>>
>>> -Rob
>>>
>>>
>>>> Best regards,
>>>> Andre
>>>>
>>>>
>>>> [1] http://people.apache.org/~af/build/
>>>> [2] http://people.apache.org/build.zip
>>>> [3] http://martine.github.io/ninja/manual.html
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Andre Fischer <aw...@gmail.com>.
On 22.01.2014 14:58, Andre Fischer wrote:
> On 22.01.2014 14:45, Rob Weir wrote:
>> On Wed, Jan 22, 2014 at 8:32 AM, Andre Fischer <aw...@gmail.com>
>> wrote:
>>> Not quite a week ago I wrote about an idea to use XML files to store
>>> the
>>> declarative part of our makefiles: dependencies of libraries on source
>>> files, which resources are to be created and so on. In the meantime
>>> I have
>>> found the time to do make (conduct?) an experiment. I am now able
>>> to build
>>> module sw from the XML files with the help of the ninja build
>>> 'system' [3].
>>> Most of the work of converting the XML files into one single
>>> build.ninja
>>> file was done on one weekend. You can see the source code at [1] ([2]
>>> contains everything zipped together).
>>>
>>> The results are promising. It runs faster and the build.ninja
>>> generator
>>> looks more maintainable than our solenv/gbuild/... makefiles. But I am
>>> certainly biased.
>>> Before I give you some numbers, I should say that I have collected the
>>> numbers totally unscientifically and it may be necessary to add some
>>> missing
>>> steps to the ninja build. To the best of my knowledge all C++ files
>>> are
>>> compiled, libraries linked, resource files built, XML files copied.
>>> Only
>>> the single sw.component file somehow escaped.
>>>
>>> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>>>
>>> Complete build of a clean module:
>>> gbuild about 9m30s (make -sr -j8)
>>> ninja about 7m15s (ninja)
>>>
>>> Cleaning up
>>> gbuild about 40s (make clean)
>>> ninja less then 1s (ninja -t clean)
>>>
>>> rebuild after touching one single header (sw/inc/section.hxx)
>>> gbuild about 1m10s (make -sr -j8)
>>> ninja about 50s (ninja)
>>>
>>> Building an already built module (nothing to do): depends very much on
>>> whether the disk cache is warm or cold. Best times:
>>> gbuild more than 3s (make -sr -j8)
>>> ninja about 0.4s (ninja)
>>>
>>>
>>> Why is ninja faster than make/gbuild?
>>> - Make runs each recipe in its own shell (bash), ninja executes its
>>> command
>>> directly.
>>> - Ninja understands the header dependencies created by gxx/clang and
>>> msvc
>>> and stores them in a compact format that can be read in very fast on
>>> startup.
>>> - I avoided some steps of build that are unnecessary in ninja
>>> = Ninja creates directories for the targets it makes. Gbuild
>>> creates them
>>> explicitly.
>>> = GBuild first creates empty dependency files and later, in a
>>> second step,
>>> fills them with the actual dependency information created by one of the
>>> C/C++ compilers.
>>>
>>>
>>> But, for me, these numbers are just a welcome side effect. More
>>> important
>>> to me is maintainability.
>>> Ninja follows a very different approach from (GNU) make. Its lack
>>> of even
>>> simplest control structures such as if/then/else or foreach,
>>> requires the
>>> generation of the main makefile (by default that is called
>>> build.ninja) by
>>> program or script. This leads to my current approach:
>>> - Use XML to represent the static data (C++ files, libraries, resource
>>> files, XML files).
>>> - Use a Perl script to translate the XML files into the build.ninja
>>> file.
>>> The best tool for each job (XML: data representation, Perl: data
>>> processing). Instead of Perl we could use any language that is part
>>> of our
>>> current build requirements (Java, C/C++, Python (we would have to
>>> compile
>>> that first, though)). Look at the Perl files in [1] or [2]
>>> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and
>>> see which
>>> you can understand better.
>>>
>>>
>>> I think this could be one way to set up a better maintainable build
>>> system
>>> that is even slightly faster then what we currently have.
>>>
>> Do you get a sense for how well-maintained Ninja is? Are there many
>> contributors? Many users?
>
> I only know that it was developed by/for the chrome project [4] and
> that cmake has support for ninja as back end.
>
>> Are we confident it will be around in 5
>> years? I worry (but only a little) of another DMake.
>
> Then I probably should not tell you that I may make a similar
> experiment with tup as backend.
There are two different and independent ideas:
- Use an easy to read data format for expressing module data (C++ files,
libraries, etc.) that is independent from the actual build tool.
- Use ninja as a back end.
The first part is much easier to accomplish and enables us to make
experiments regarding different back ends.
Our current gbuild system would be a natural choice for the first back
end. Pure make might be the second. And maybe ninja would be the third.
-Andre
>
> -Andre
>
>
> [4] http://www.aosabook.org/en/posa/ninja.html
> [5] http://gittup.org/tup/
>
>>
>> -Rob
>>
>>
>>> Best regards,
>>> Andre
>>>
>>>
>>> [1] http://people.apache.org/~af/build/
>>> [2] http://people.apache.org/build.zip
>>> [3] http://martine.github.io/ninja/manual.html
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Andre Fischer <aw...@gmail.com>.
On 22.01.2014 14:45, Rob Weir wrote:
> On Wed, Jan 22, 2014 at 8:32 AM, Andre Fischer <aw...@gmail.com> wrote:
>> Not quite a week ago I wrote about an idea to use XML files to store the
>> declarative part of our makefiles: dependencies of libraries on source
>> files, which resources are to be created and so on. In the meantime I have
>> found the time to do make (conduct?) an experiment. I am now able to build
>> module sw from the XML files with the help of the ninja build 'system' [3].
>> Most of the work of converting the XML files into one single build.ninja
>> file was done on one weekend. You can see the source code at [1] ([2]
>> contains everything zipped together).
>>
>> The results are promising. It runs faster and the build.ninja generator
>> looks more maintainable than our solenv/gbuild/... makefiles. But I am
>> certainly biased.
>> Before I give you some numbers, I should say that I have collected the
>> numbers totally unscientifically and it may be necessary to add some missing
>> steps to the ninja build. To the best of my knowledge all C++ files are
>> compiled, libraries linked, resource files built, XML files copied. Only
>> the single sw.component file somehow escaped.
>>
>> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>>
>> Complete build of a clean module:
>> gbuild about 9m30s (make -sr -j8)
>> ninja about 7m15s (ninja)
>>
>> Cleaning up
>> gbuild about 40s (make clean)
>> ninja less then 1s (ninja -t clean)
>>
>> rebuild after touching one single header (sw/inc/section.hxx)
>> gbuild about 1m10s (make -sr -j8)
>> ninja about 50s (ninja)
>>
>> Building an already built module (nothing to do): depends very much on
>> whether the disk cache is warm or cold. Best times:
>> gbuild more than 3s (make -sr -j8)
>> ninja about 0.4s (ninja)
>>
>>
>> Why is ninja faster than make/gbuild?
>> - Make runs each recipe in its own shell (bash), ninja executes its command
>> directly.
>> - Ninja understands the header dependencies created by gxx/clang and msvc
>> and stores them in a compact format that can be read in very fast on
>> startup.
>> - I avoided some steps of build that are unnecessary in ninja
>> = Ninja creates directories for the targets it makes. Gbuild creates them
>> explicitly.
>> = GBuild first creates empty dependency files and later, in a second step,
>> fills them with the actual dependency information created by one of the
>> C/C++ compilers.
>>
>>
>> But, for me, these numbers are just a welcome side effect. More important
>> to me is maintainability.
>> Ninja follows a very different approach from (GNU) make. Its lack of even
>> simplest control structures such as if/then/else or foreach, requires the
>> generation of the main makefile (by default that is called build.ninja) by
>> program or script. This leads to my current approach:
>> - Use XML to represent the static data (C++ files, libraries, resource
>> files, XML files).
>> - Use a Perl script to translate the XML files into the build.ninja file.
>> The best tool for each job (XML: data representation, Perl: data
>> processing). Instead of Perl we could use any language that is part of our
>> current build requirements (Java, C/C++, Python (we would have to compile
>> that first, though)). Look at the Perl files in [1] or [2]
>> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see which
>> you can understand better.
>>
>>
>> I think this could be one way to set up a better maintainable build system
>> that is even slightly faster then what we currently have.
>>
> Do you get a sense for how well-maintained Ninja is? Are there many
> contributors? Many users?
I only know that it was developed by/for the chrome project [4] and that
cmake has support for ninja as back end.
> Are we confident it will be around in 5
> years? I worry (but only a little) of another DMake.
Then I probably should not tell you that I may make a similar experiment
with tup as backend.
-Andre
[4] http://www.aosabook.org/en/posa/ninja.html
[5] http://gittup.org/tup/
>
> -Rob
>
>
>> Best regards,
>> Andre
>>
>>
>> [1] http://people.apache.org/~af/build/
>> [2] http://people.apache.org/build.zip
>> [3] http://martine.github.io/ninja/manual.html
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Building sw/ with ninja
Posted by Rob Weir <ro...@apache.org>.
On Wed, Jan 22, 2014 at 8:32 AM, Andre Fischer <aw...@gmail.com> wrote:
> Not quite a week ago I wrote about an idea to use XML files to store the
> declarative part of our makefiles: dependencies of libraries on source
> files, which resources are to be created and so on. In the meantime I have
> found the time to do make (conduct?) an experiment. I am now able to build
> module sw from the XML files with the help of the ninja build 'system' [3].
> Most of the work of converting the XML files into one single build.ninja
> file was done on one weekend. You can see the source code at [1] ([2]
> contains everything zipped together).
>
> The results are promising. It runs faster and the build.ninja generator
> looks more maintainable than our solenv/gbuild/... makefiles. But I am
> certainly biased.
> Before I give you some numbers, I should say that I have collected the
> numbers totally unscientifically and it may be necessary to add some missing
> steps to the ninja build. To the best of my knowledge all C++ files are
> compiled, libraries linked, resource files built, XML files copied. Only
> the single sw.component file somehow escaped.
>
> I ran my experiments on ani7 2.2GHz, 8GB notebook.
>
> Complete build of a clean module:
> gbuild about 9m30s (make -sr -j8)
> ninja about 7m15s (ninja)
>
> Cleaning up
> gbuild about 40s (make clean)
> ninja less then 1s (ninja -t clean)
>
> rebuild after touching one single header (sw/inc/section.hxx)
> gbuild about 1m10s (make -sr -j8)
> ninja about 50s (ninja)
>
> Building an already built module (nothing to do): depends very much on
> whether the disk cache is warm or cold. Best times:
> gbuild more than 3s (make -sr -j8)
> ninja about 0.4s (ninja)
>
>
> Why is ninja faster than make/gbuild?
> - Make runs each recipe in its own shell (bash), ninja executes its command
> directly.
> - Ninja understands the header dependencies created by gxx/clang and msvc
> and stores them in a compact format that can be read in very fast on
> startup.
> - I avoided some steps of build that are unnecessary in ninja
> = Ninja creates directories for the targets it makes. Gbuild creates them
> explicitly.
> = GBuild first creates empty dependency files and later, in a second step,
> fills them with the actual dependency information created by one of the
> C/C++ compilers.
>
>
> But, for me, these numbers are just a welcome side effect. More important
> to me is maintainability.
> Ninja follows a very different approach from (GNU) make. Its lack of even
> simplest control structures such as if/then/else or foreach, requires the
> generation of the main makefile (by default that is called build.ninja) by
> program or script. This leads to my current approach:
> - Use XML to represent the static data (C++ files, libraries, resource
> files, XML files).
> - Use a Perl script to translate the XML files into the build.ninja file.
> The best tool for each job (XML: data representation, Perl: data
> processing). Instead of Perl we could use any language that is part of our
> current build requirements (Java, C/C++, Python (we would have to compile
> that first, though)). Look at the Perl files in [1] or [2]
> (build/source/ninja/*pm) and compare them to solenv/gbuild/*mk and see which
> you can understand better.
>
>
> I think this could be one way to set up a better maintainable build system
> that is even slightly faster then what we currently have.
>
Do you get a sense for how well-maintained Ninja is? Are there many
contributors? Many users? Are we confident it will be around in 5
years? I worry (but only a little) of another DMake.
-Rob
> Best regards,
> Andre
>
>
> [1] http://people.apache.org/~af/build/
> [2] http://people.apache.org/build.zip
> [3] http://martine.github.io/ninja/manual.html
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org