You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "hector.acosta@cloudera.com (Code Review)" <ge...@cloudera.org> on 2019/02/25 17:05:42 UTC
[native-toolchain-CR] Add ccache support
hector.acosta@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12574
Change subject: Add ccache support
......................................................................
Add ccache support
Here we add ccache support, which speeds up building the entire
toolchain. This almost works out of the box, except for the following
complications:
* We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to
'content', this is because by default ccache includes the modification
time of the compiler, which would prevent us to reuse the cache
between runs.
* We specifically set CC and CXX variables, the initial approach of
setting them to 'ccache $CC' breaks boost (and possibly others)
because they try to run `ccache $CC`. A second approach that i tried
was relying on PATH ordering to make ccache execute the right thing.
When using the system compiler, we want:
PATH=$CCACHE_PATH:/usr/bin:...
CC=$CCACHE_PATH/gcc
After bootstrapping we want:
PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH
CC=$CCACHE_PATH/gcc
I think complex modification of PATH is too britle and would cause
ccache to execute the system compiler when $CCACHE_PATH == /usr/bin.
Instead we create a wrapper around ccache that executes CC and place
it at the beginning of the path, we set CC to this wrapper script.
I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"`
on the following configs:
disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k
empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k
empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k
full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k
full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k
Here's ccache -s after the full ccache build (statistics were cleared
before strting the build).
+ ccache -s
cache directory build_docker/ccache
primary config build_docker/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 138821
cache hit (preprocessed) 17437
cache miss 225
cache hit rate 99.86 %
called for link 15551
called for preprocessing 9567
multiple source files 42
compile failed 7705
preprocessor error 2647
bad compiler arguments 6047
unsupported source language 42
autoconf compile/link 18651
unsupported compiler option 342
unsupported code directive 24
no input file 4323
cleanups performed 0
files in cache 289759
cache size 30.5 GB
max cache size 50.0 GB
Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB.
Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
---
M .gitignore
M docker/debian7.df
M docker/debian8.df
M docker/redhat6.df
M docker/redhat7.df
M docker/sles12.df
M docker/ubuntu1204.df
M docker/ubuntu1404.df
M docker/ubuntu1604.df
M docker/ubuntu1804.df
M functions.sh
M in-docker.py
M init-compiler.sh
M init.sh
14 files changed, 165 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/74/12574/1
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 1
Gerrit-Owner: hector.acosta@cloudera.com <he...@cloudera.com>
[native-toolchain-CR] Add ccache support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Add ccache support
Here we add ccache support, which speeds up building the entire
toolchain. This almost works out of the box, except for the following
complications:
* We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to
'content', this is because by default ccache includes the modification
time of the compiler, which would prevent us to reuse the cache
between runs.
* We specifically set CC and CXX variables, the initial approach of
setting them to 'ccache $CC' breaks boost (and possibly others)
because they try to run `ccache $CC`. A second approach that i tried
was relying on PATH ordering to make ccache execute the right thing.
When using the system compiler, we want:
PATH=$CCACHE_PATH:/usr/bin:...
CC=$CCACHE_PATH/gcc
After bootstrapping we want:
PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH
CC=$CCACHE_PATH/gcc
I think complex modification of PATH is too britle and would cause
ccache to execute the system compiler when $CCACHE_PATH == /usr/bin.
Instead we create a wrapper around ccache that executes CC and place
it at the beginning of the path, we set CC to this wrapper script.
I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"`
on the following configs:
disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k
empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k
empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k
full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k
full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k
Here's ccache -s after the full ccache build (statistics were cleared
before strting the build).
+ ccache -s
cache directory build_docker/ccache
primary config build_docker/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 138821
cache hit (preprocessed) 17437
cache miss 225
cache hit rate 99.86 %
called for link 15551
called for preprocessing 9567
multiple source files 42
compile failed 7705
preprocessor error 2647
bad compiler arguments 6047
unsupported source language 42
autoconf compile/link 18651
unsupported compiler option 342
unsupported code directive 24
no input file 4323
cleanups performed 0
files in cache 289759
cache size 30.5 GB
max cache size 50.0 GB
Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB.
Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Reviewed-on: http://gerrit.cloudera.org:8080/12574
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
---
M .gitignore
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/debian7.df
M docker/debian8.df
M docker/redhat6.df
M docker/redhat7.df
M docker/sles12.df
M docker/ubuntu1204.df
M docker/ubuntu1404.df
M docker/ubuntu1604.df
M docker/ubuntu1804.df
M functions.sh
M in-docker.py
M init-compiler.sh
M init.sh
16 files changed, 154 insertions(+), 113 deletions(-)
Approvals:
Philip Zeyliger: Looks good to me, approved
Tim Armstrong: Verified
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 3
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[native-toolchain-CR] Add ccache support
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
Looks reasonable to me. Nice speed improvement.
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df
File docker/debian7.df:
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df@43
PS1, Line 43: wget "https://www.samba.org/ftp/ccache/ccache-3.3.3.tar.gz" && \
I recommend moving this into a shell script.
We've also tended to sha-check our binaries. E.g.:
bin/bootstrap_system.sh:redhat sha512sum -c - <<< 'c8321aa223f70d7e64d3d0274263000cfffb46fbea61488534e26f9f0245d99e9872d0888e35cd3274416392a13f80c748c07750caaeffa5f9cae1220020715f apache-ant-1.9.13-bin.tar.gz'
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh
File init.sh:
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh@138
PS1, Line 138: : ${USE_CCACHE=0}
: export USE_CCACHE
:
: : ${CCACHE_MAXSIZE=50G}
: export CCACHE_MAXSIZE
:
: : ${CCACHE_DIR=$SOURCE_DIR/ccache}
: export CCACHE_DIR
:
: # Default ccache_compilercheck is mtime, which considers CC's mtime + size
: # to determine if there's a hit. Setting CCACHE_COMPILERCHECK to 'content'
: # uses the hash of the compiler instead.
: export CCACHE_COMPILERCHECK=${CCACHE_COMPILERCHECK:-content}
:
: export CCACHE_COMPRESS=1
Do you think these would be better encapsulated in a .ccache.conf file that we create when we bootstrap ccache?
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 1
Gerrit-Owner: hector.acosta@cloudera.com <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Feb 2019 17:36:38 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] Add ccache support
Posted by "Hector Acosta (Code Review)" <ge...@cloudera.org>.
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df
File docker/debian7.df:
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df@43
PS1, Line 43: wget "https://www.samba.org/ftp/ccache/ccache-3.3.3.tar.gz" && \
> I recommend moving this into a shell script.
Done
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh
File init.sh:
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh@138
PS1, Line 138: : ${USE_CCACHE=0}
: export USE_CCACHE
:
: : ${CCACHE_MAXSIZE=50G}
: export CCACHE_MAXSIZE
:
: : ${CCACHE_DIR=$SOURCE_DIR/ccache}
: export CCACHE_DIR
:
: # Default ccache_compilercheck is mtime, which considers CC's mtime + size
: # to determine if there's a hit. Setting CCACHE_COMPILERCHECK to 'content'
: # uses the hash of the compiler instead.
: export CCACHE_COMPILERCHECK=${CCACHE_COMPILERCHECK:-content}
:
: export CCACHE_COMPRESS=1
> Do you think these would be better encapsulated in a .ccache.conf file that
I think there's a trade-off here.. If we create and mount a ccache.conf, ccache would only work inside docker.
If we create the ccache.conf somewhere in functions.sh it would work both inside and outside of docker, but since someone could have already defined CCACHE_DIR, we'd need to dance around it to prevent overwriting that conf (not sure if this matters in practice).
I know being able to run this in the host helped me debugging some problems so i lean towards being able to enable ccache independent of docker.
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 1
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Mar 2019 17:58:54 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] Add ccache support
Posted by "Hector Acosta (Code Review)" <ge...@cloudera.org>.
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 2:
> Hector, are you blocked on something? Or just waiting for someone
> to merge?
Just waiting on someone to merge.
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Mar 2019 17:14:42 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Mar 2019 17:59:05 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Posted by "Hector Acosta (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12574
to look at the new patch set (#2).
Change subject: Add ccache support
......................................................................
Add ccache support
Here we add ccache support, which speeds up building the entire
toolchain. This almost works out of the box, except for the following
complications:
* We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to
'content', this is because by default ccache includes the modification
time of the compiler, which would prevent us to reuse the cache
between runs.
* We specifically set CC and CXX variables, the initial approach of
setting them to 'ccache $CC' breaks boost (and possibly others)
because they try to run `ccache $CC`. A second approach that i tried
was relying on PATH ordering to make ccache execute the right thing.
When using the system compiler, we want:
PATH=$CCACHE_PATH:/usr/bin:...
CC=$CCACHE_PATH/gcc
After bootstrapping we want:
PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH
CC=$CCACHE_PATH/gcc
I think complex modification of PATH is too britle and would cause
ccache to execute the system compiler when $CCACHE_PATH == /usr/bin.
Instead we create a wrapper around ccache that executes CC and place
it at the beginning of the path, we set CC to this wrapper script.
I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"`
on the following configs:
disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k
empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k
empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k
full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k
full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k
Here's ccache -s after the full ccache build (statistics were cleared
before strting the build).
+ ccache -s
cache directory build_docker/ccache
primary config build_docker/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 138821
cache hit (preprocessed) 17437
cache miss 225
cache hit rate 99.86 %
called for link 15551
called for preprocessing 9567
multiple source files 42
compile failed 7705
preprocessor error 2647
bad compiler arguments 6047
unsupported source language 42
autoconf compile/link 18651
unsupported compiler option 342
unsupported code directive 24
no input file 4323
cleanups performed 0
files in cache 289759
cache size 30.5 GB
max cache size 50.0 GB
Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB.
Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
---
M .gitignore
M docker/all/assert-dependencies-present.py
M docker/all/postinstall.sh
M docker/debian7.df
M docker/debian8.df
M docker/redhat6.df
M docker/redhat7.df
M docker/sles12.df
M docker/ubuntu1204.df
M docker/ubuntu1404.df
M docker/ubuntu1604.df
M docker/ubuntu1804.df
M functions.sh
M in-docker.py
M init-compiler.sh
M init.sh
16 files changed, 154 insertions(+), 113 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/74/12574/2
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[native-toolchain-CR] Add ccache support
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 2:
Hector, are you blocked on something? Or just waiting for someone to merge?
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Mar 2019 16:44:34 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12574/2/docker/all/postinstall.sh
File docker/all/postinstall.sh:
http://gerrit.cloudera.org:8080/#/c/12574/2/docker/all/postinstall.sh@54
PS2, Line 54: make -j
We often use -j$(nproc)... don't know that it matters.
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Mar 2019 21:30:59 +0000
Gerrit-HasComments: Yes