Proposal: Precompiled headers for Windows use in cmake

[Jeff Coffler <Je...@microsoft.com.INVALID>]

I had previously send an E-Mail to the Mesos DEV list on Tue 2/14/2017 11:29 AM. I've attached that message here for reference (hopefully the attachment will make it across).

Work has proceeded, and I'm working out some commits to apply to the changes for Windows (cmake only).

While I was asked to work out timing for Linux platforms, I was unable to do so as the Linux code doesn't currently build with precompiled headers due to overloading of the std namespace (Set/set, as I recall). We could hack sources to build with precompiled headers, but this would be throw-away work and thus doesn't seem all that useful an engineering expenditure. But without the permanent changes needed there to avoid namespace pollution, or hack/throw away changes, it's not possible to get timing tests for PCH use on Linux at this time.

For Windows, here are the results of timing tests that I've done:

Build times on Windows (no PCH):
  1. Full build time (everything including 3rd party products): 24:49.47 (1489.47s total)
  2. Time to rebuild all of Mesos itself: 21:8.63 (1268.63s total)
  3. Time for a build with no changes: 0:30.47 (30.47s total)
  4. Modified file 3rdparty/stout/include/stout/os/os.hpp (added a comment)
  5. Time for an incremental build: 36:55.55 (2215.55s total)
     (Very odd that an incremental takes longer than a full build!)
  6. Time for an incremental build (just mesos-agent): 0:14:24.73 (864.73s total)
  7. Time for an incremental build, mesos-agent only: 0:16:42.57 (1002.57s total)
     (Includes speedup to linker flags)


Build times on Windows (with PCH):
  1. Full build time (everything including 3rd party products): 19:54.49 (1194.49s total)
  2. Time to rebuild all of Mesos itself: 0:15:49.58 (949.58s total)
  3. Time for a build with no changes: 0:0:21.72 (21.72s total)
  4. Modified file 3rdparty/stout/include/stout/os/os.hpp (added a comment)
  5. Time for an incremental build: 0:34:40.64 (2080.64s total)
  6. Time for an incremental build (just mesos-agent): 0:11:53.57 (713.57s total)
  7. Time for an incremental build, mesos-agent only: 0:10:43.10 (643.10s total)
     (Includes speedup to linker flags)

To be clear here, more work can be done. We pretty much just optimized building mesos-agent. There are other targets that would benefit from PCH as well. However, unless/until we start doing major portions of work in those areas, it may not make sense. (For example, we're unlikely to do major pieces of redesign to most of our 3rd party components, nor do we have the ability to change those build processes anyway.)

The above timing tests include the time to actually build the PCH. In the full build case, the reduction is close to 25%. In the incremental build case (#7), the reduction is closer to 33%. In both cases, we believe that this change is worthwhile.

Any questions/concerns? If not, I'll work on finalizing the commits and getting them tested in the next few days.

/Jeff

Re: Proposal: Precompiled headers for Windows use in cmake

[Joris Van Remoortere <jo...@mesosphere.io>]

Thanks Jeff. This looks great.

It's good to know that as more people start contributing on the Windows
platform their productivity will be improving.

Hopefully someone from the community can follow up and keep exploring the
linux side of this as well. Could you please file the relevant JIRAs if you
don't plan on following up on this yourself in the near future?

—
*Joris Van Remoortere*
Mesosphere

On Mon, Mar 27, 2017 at 4:03 PM, Jeff Coffler <
Jeff.Coffler@microsoft.com.invalid> wrote:

> I had previously send an E-Mail to the Mesos DEV list on Tue 2/14/2017
> 11:29 AM. I’ve attached that message here for reference (hopefully the
> attachment will make it across).
>
>
>
> Work has proceeded, and I’m working out some commits to apply to the
> changes for Windows (cmake only).
>
>
>
> While I was asked to work out timing for Linux platforms, I was unable to
> do so as the Linux code doesn’t currently build with precompiled headers
> due to overloading of the std namespace (Set/set, as I recall). We could
> hack sources to build with precompiled headers, but this would be
> throw-away work and thus doesn’t seem all that useful an engineering
> expenditure. But without the permanent changes needed there to avoid
> namespace pollution, or hack/throw away changes, it’s not possible to get
> timing tests for PCH use on Linux at this time.
>
>
>
> For Windows, here are the results of timing tests that I’ve done:
>
>
>
> Build times on Windows (no PCH):
>
>   1. Full build time (everything including 3rd party products): 24:49.47
> (1489.47s total)
>
>   2. Time to rebuild all of Mesos itself: 21:8.63 (1268.63s total)
>
>   3. Time for a build with no changes: 0:30.47 (30.47s total)
>
>   4. Modified file 3rdparty/stout/include/stout/os/os.hpp (added a
> comment)
>
>   5. Time for an incremental build: 36:55.55 (2215.55s total)
>
>      (Very odd that an incremental takes longer than a full build!)
>
>   6. Time for an incremental build (just mesos-agent): 0:14:24.73 (864.73s
> total)
>
>   7. Time for an incremental build, mesos-agent only: 0:16:42.57 (1002.57s
> total)
>
>      (Includes speedup to linker flags)
>
>
>
>
>
> Build times on Windows (with PCH):
>
>   1. Full build time (everything including 3rd party products): 19:54.49
> (1194.49s total)
>
>   2. Time to rebuild all of Mesos itself: 0:15:49.58 (949.58s total)
>
>   3. Time for a build with no changes: 0:0:21.72 (21.72s total)
>
>   4. Modified file 3rdparty/stout/include/stout/os/os.hpp (added a
> comment)
>
>   5. Time for an incremental build: 0:34:40.64 (2080.64s total)
>
>   6. Time for an incremental build (just mesos-agent): 0:11:53.57 (713.57s
> total)
>
>   7. Time for an incremental build, mesos-agent only: 0:10:43.10 (643.10s
> total)
>
>      (Includes speedup to linker flags)
>
>
>
> To be clear here, more work can be done. We pretty much just optimized
> building mesos-agent. There are other targets that would benefit from PCH
> as well. However, unless/until we start doing major portions of work in
> those areas, it may not make sense. (For example, we’re unlikely to do
> major pieces of redesign to most of our 3rd party components, nor do we
> have the ability to change those build processes anyway.)
>
>
>
> The above timing tests include the time to actually build the PCH. In the
> full build case, the reduction is close to 25%. In the incremental build
> case (#7), the reduction is closer to 33%. In both cases, we believe that
> this change is worthwhile.
>
>
>
> Any questions/concerns? If not, I’ll work on finalizing the commits and
> getting them tested in the next few days.
>
>
>
> /Jeff
>
>
> ---------- Forwarded message ----------
> From: Jeff Coffler <Je...@microsoft.com.invalid>
> To: "dev@mesos.apache.org" <de...@mesos.apache.org>
> Cc:
> Bcc:
> Date: Tue, 14 Feb 2017 19:28:50 +0000
> Subject: Proposal for Mesos Build Improvements
> Proposal For Build Improvements
>
> The Mesos build process is in dire need of some build infrastructure
> improvements. These improvements will improve speed and ease of work in
> particular components, and dramatically improve overall build time,
> especially in the Windows environment, but likely in the Linux environment
> as well.
>
>
> Background:
>
> It is currently recommended to use the ccache project with the Mesos build
> process. This makes the Linux build process more tolerable in terms of
> speed, but unfortunately such software is not available on Windows.
> Ultimately, though, the caching software is covering up two fundamental
> flaws in the overall build process:
>
> 1. Lack of use of libraries
> 2. Lack of precompiled headers
>
> By not allowing use of libraries, the overall build process is often much
> longer, particularly when a lot of work is being done in a particular
> component. If work is being done in a particular component, only that
> library need be rebuilt (and then the overall image relinked). Currently,
> since there is no such modularization, all source files must be considered
> at build time. Interestingly enough, there is such modularization in the
> source code layout; that modularization just isn't utilized at the compiler
> level.
>
> Precompiled headers exist on both Windows and Linux. For Linux, you can
> refer to https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgcc.gnu.org%2Fonlinedocs%2Fgcc%2FPrecompiled-Headers.html&
> data=02%7C01%7CJeff.Coffler%40microsoft.com%7C063d6dbc04304614f82f08d4550f
> c22d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%
> 7C636226973566412987&sdata=cacmJfK7ofxwx%2B663qCWVw3QzDdg61Eqps7ZV1Trxd
> 0%3D&reserved=0. Straight from the GNU CC documentation: "The time the
> compiler takes to process these header files over and over again can
> account for nearly all of the time required to build the project."
>
> In my prior use of precompiled headers, each C or C++ file generally took
> about 4 seconds to compile. After switching to precompiled headers, the
> precompiled header creation took about 4 seconds, but each C/C++ file now
> took about 200 milliseconds to compile. The overall build speed was thus
> dramatically reduced.
>
>
> Scope of Changes:
>
> These changes are only being proposed for the CMake system. Going forward,
> the CMake system is the easiest way to maintain some level of portability
> between the Linux and Windows platforms.
>
>
> Details for Modularization:
>
> For the modularization, the intent is to simply make each source directory
> of files, if functionally separate, to be compiled into an archive (.a)
> file. These archive files will then be linked together to form the actual
> executables. These changes will primarily be in the CMake system, and
> should have limited effect on any actual source code.
>
> At a later date, if it makes sense, we can look at building shared library
> (.so) files. However, this only makes the most sense if the code is truly
> shared between different executable files. If that's not the case, then it
> likely makes sense just to stick with .a files. Regardless, generation of
> .so files is out of scope for this change.
>
>
> Details for Precompiled Header Changes:
>
> Precompiled headers will make use of stout (a very large header-only
> library) essentially "free" from a compile-time overhead point of view.
> Basically, precompiled headers will take a list of header files (including
> very long header files, like "windows.h"), and generate the compiler memory
> structures for their representation.
>
> During precompiled header generation, these memory structures are flushed
> to disk. Then, when components are built, the memory structures are
> reloaded from disk, which is dramatically faster than actually parsing the
> tens of thousands of lines of header files and building the memory
> structures.
>
> For precompiled headers to be useful, a relatively "consistent" set of
> headers must be included by all of the C/C++ files. So, for example,
> consider the following C file:
>
> #if defined(windows)
> #include <windows.h>
> #endif
>
> #include <header-a>
> #include <header-b>
> #include <header-c>
>
> < - Remainder of module - >
>
> To make a precompiled header for this module, all of the #include files
> would be included in a new file, mesos_common.h. The C file would then be
> changed as follows:
>
> #include "mesos_common.h"
>
> < - Remainder of module - >
>
> Structurally, the code is identical, and need not be built with
> precompiled headers. However, use of precompiled headers will make file
> compilation dramatically faster.
>
> Note that other include files can be included after the precompiled header
> if appropriate. For example, the following is valid:
>
> #include "mesos_common.h"
> #inclue <header-d>
>
> < - Remainder of module - >
>
> For efficiency purposes, if a header file is included by 50% or more of
> the source files, it should be included in the precompiled header. If a
> header is included in fewer than 50% of the source files, then it can be
> separately included (and thus would not benefit from precompiled headers).
> Note that this is a guideline; even if a header is used by less than 50% of
> source files, if it's very large, we still may decide to throw it in the
> precompiled header.
>
> Note that, for use of precompiled headers, there will be a great deal of
> code churn (almost exclusively in the #include list of source files). This
> will mean that there will be a lot of code merges, but ultimately no "code
> logic" changes. If merges are not done in a timely fashion, this can easily
> result in needless hand merging of changes. Due to these issues, we will
> need a dedicated sheppard that will integrate the patches quickly. This
> kind of work is easily invalidated when the include list is changed by
> another developer, necessitating us to redo the patch. [Note that Joseph
> has stepped up to the plate for this, thanks Joseph!]
>
>
> This is the end of my proposal, feedback would be appreciated.
>
>