[native-toolchain-CR] Add ccache support

["Hector Acosta (Code Review)" <ge...@cloudera.org>]

Hello Philip Zeyliger, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12574

to look at the new patch set (#2).

Change subject: Add ccache support
......................................................................

Add ccache support

Here we add ccache support, which speeds up building the entire
toolchain. This almost works out of the box, except for the following
complications:
 * We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to
   'content', this is because by default ccache includes the modification
   time of the compiler, which would prevent us to reuse the cache
   between runs.
 * We specifically set CC and CXX variables, the initial approach of
   setting them to 'ccache $CC' breaks boost (and possibly others)
   because they try to run `ccache $CC`. A second approach that i tried
   was relying on PATH ordering to make ccache execute the right thing.
   When using the system compiler, we want:
     PATH=$CCACHE_PATH:/usr/bin:...
     CC=$CCACHE_PATH/gcc
   After bootstrapping we want:
     PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH
     CC=$CCACHE_PATH/gcc
   I think complex modification of PATH is too britle and would cause
   ccache to execute the system compiler when $CCACHE_PATH == /usr/bin.
   Instead we create a wrapper around ccache that executes CC and place
   it at the beginning of the path, we set CC to this wrapper script.

I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"`
on the following configs:
disabled ccache:           3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k
empty ccache:              3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k
empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k
full ccache:               2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k
full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k

Here's ccache -s after the full ccache build (statistics were cleared
before strting the build).
+ ccache -s
cache directory                     build_docker/ccache
primary config                      build_docker/ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
cache hit (direct)                138821
cache hit (preprocessed)           17437
cache miss                           225
cache hit rate                     99.86 %
called for link                    15551
called for preprocessing            9567
multiple source files                 42
compile failed                      7705
preprocessor error                  2647
bad compiler arguments              6047
unsupported source language           42
autoconf compile/link              18651
unsupported compiler option          342
unsupported code directive            24
no input file                       4323
cleanups performed                     0
files in cache                    289759
cache size                          30.5 GB
max cache size                      50.0 GB

Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB.

Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
---
M .gitignore
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/debian7.df
M docker/debian8.df
M docker/redhat6.df
M docker/redhat7.df
M docker/sles12.df
M docker/ubuntu1204.df
M docker/ubuntu1404.df
M docker/ubuntu1604.df
M docker/ubuntu1804.df
M functions.sh
M in-docker.py
M init-compiler.sh
M init.sh
16 files changed, 154 insertions(+), 113 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/74/12574/2
-- 
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>