You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2024/01/08 19:23:03 UTC
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20870
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
IMPALA-12686: Build the toolchain with basic debug information (-g1)
Basic debug information (e.g. -g1) is useful for getting better
stack traces. Compressing debug information (-gz) reduces the size
of this debug information dramatically. This adds both -g1 and -gz
to the default flags for the toolchain.
In order to have the flags apply uniformly, components need to
respect the existing CFLAGS and CXXFLAGS. Several components were
setting their own CFLAGS and CXXFLAGS without keeping the existing
flags, so this fixes the components to keep existing flags.
Some components build with an extra -g after our flags and these
continue to build this way. Specifically, the following using -g:
- Thrift (-g added in this change)
- Kudu
- ORC
- CCTZ
- bzip2
These keep the same debug information they had before, except that
-gz compresses it now. This change should not reduce the debug
information for any component.
This turns off debug information for CMake as it is just a build
tool and doesn't need it.
Testing:
- Ran a build and examined what happened to the package sizes.
Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
---
M init-compiler.sh
M source/arrow/build.sh
M source/bzip2/build.sh
M source/cmake/build.sh
M source/curl/build.sh
M source/flatbuffers/build.sh
M source/gflags/build.sh
M source/glog/build.sh
M source/lz4/build.sh
M source/orc/build.sh
M source/python/build.sh
M source/thrift/build.sh
12 files changed, 27 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/70/20870/1
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 5: Code-Review+2
Rebased, carry +2
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Sep 2024 15:35:36 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 4: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Aug 2024 18:49:57 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 4: Code-Review+2
LGTM
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Sep 2024 20:48:42 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 21:32:28 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
IMPALA-12686: Build the toolchain with basic debug information (-g1)
Basic debug information (e.g. -g1) is useful for getting better
stack traces. Compressing debug information (-gz) reduces the size
of this debug information dramatically. This adds both -g1 and -gz
to the default flags for the toolchain.
In order to have the flags apply uniformly, components need to
respect the existing CFLAGS and CXXFLAGS. Several components were
setting their own CFLAGS and CXXFLAGS without keeping the existing
flags, so this either fixes the components to keep existing flags
or removes the custom CFLAGS/CXXFLAGS.
Some components build with an extra -g after our flags and these
continue to build this way. Specifically, the following using -g:
- Thrift (-g added in this change)
- Kudu
- ORC
- CCTZ
- bzip2
These keep the same debug information they had before, except that
-gz compresses it now. This change should not reduce the debug
information for any component.
This skips generating debug information for a few things:
1. Build tools like CMake and Mold don't need debuginfo
2. LLVM's release build has an enormous amount of debuginfo, even
at -g1. If this uses debuginfo for LLVM, it adds hundreds of
MB to impalad's binary size. This skips that for now.
3. Kudu's thirdparty includes a build of LLVM, so this turns off
debuginfo for Kudu's thirdparty.
Testing:
- Ran a build and examined what happened to the package sizes.
The total size of the toolchain directory for Impala went down
from 7.08GB to 6.51GB. Kudu got much smaller due to compressing
debuginfo. Most other libraries got larger.
- Ran an Impala build to look at what happened to the binary size.
This adds about 31MB to the compressed binary size. It has no
impact on the stripped size.
- Added verbosity to print the compiler commands and verified
the flags were present
Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Reviewed-on: http://gerrit.cloudera.org:8080/20870
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Joe McDonnell <jo...@cloudera.com>
---
M init-compiler.sh
M source/arrow/build.sh
M source/bzip2/build.sh
M source/calloncehack/calloncehack/CMakeLists.txt
M source/cloudflarezlib/build.sh
M source/cmake/build.sh
M source/curl/build.sh
M source/flatbuffers/build.sh
M source/gflags/build.sh
M source/glog/build.sh
M source/kudu/build.sh
M source/libpfm/build.sh
M source/llvm/build-source-tarball.sh
M source/lz4/build.sh
M source/mold/build.sh
M source/orc/build.sh
M source/python/build.sh
M source/thrift/build.sh
M source/zlib/build.sh
19 files changed, 58 insertions(+), 40 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 6
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh
File source/cmake/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh@27
PS2, Line 27: # Drop the -g1/-gz flags for CMake, as it is just a build utility and does not
> Oddly enough, it didn't, but now that I'm looking at the new binaries, this
I think it might be related to ccache. We use ccache 3.3.3, but 3.3.5 release notes mentions this:
"Fixed a regression where the original order of debug options could be lost."
https://ccache.dev/releasenotes.html#_ccache_3_3_5
So, I'm going to change this back to adding -g0 and file a separate ticket for upgrading ccache.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 21:16:59 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh
File source/cmake/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh@27
PS2, Line 27: # Drop the -g1/-gz flags for CMake, as it is just a build utility and does not
> Did adding -g0 not work?
Oddly enough, it didn't, but now that I'm looking at the new binaries, this didn't work either. I'll have to take another pass at how to do this.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 17:46:43 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh
File init-compiler.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@52
PS1, Line 52: # ARCH_FLAGS are used to convey architectur dependent flags that should
> Want to fix the typo "architectur" while you're making changes?
Done
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@102
PS1, Line 102: CFLAGS="$ARCH_FLAGS -fPIC -O3 -g1 -gz"
> Looks like CFLAGS did not previously have all the ARCH_FLAGS.
Yeah, that is odd. It seems cleaner to have CFLAGS to contain ARCH_FLAGS.
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh
File source/arrow/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh@32
PS1, Line 32: CFLAGS="$CFLAGS -fPIC -DPIC" wrap make VERBOSE=1 -j${BUILD_THREADS:-4} all install
> -fPIC is redundant now. Should we set -DPIC everywhere too?
Specifying it at the top level caused the LLVM build to fail. I removed DPIC across the board. I can't find anything indication that it is still needed. Anything that wants to check whether this is a position independent build has other things to check.
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh
File source/bzip2/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh@33
PS1, Line 33: CFLAGS="$CFLAGS -fPIC -DPIC" wrap make -f Makefile-libbz2_so CFLAGS="$CFLAGS -fPIC -DPIC"
> Why declare CFLAGS twice?
Something about the second command doesn't respect the CFLAGS environment variable. I removed one set of CFLAGS settings.
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh
File source/flatbuffers/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh@33
PS1, Line 33: CXXFLAGS=${CXXFLAGS}
> Is this line even needed?
Removed
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh
File source/lz4/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh@32
PS1, Line 32: CFLAGS="$CFLAGS -fPIC"
> This whole line is unnecessary, it already includes -fPIC.
Good point, fixed
http://gerrit.cloudera.org:8080/#/c/20870/1/source/orc/build.sh
File source/orc/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/orc/build.sh@47
PS1, Line 47: wrap make VERBOSE=1 -j${BUILD_THREADS:-4} install
> This removed -DPIC. I asked elsewhere if we should add that globally.
I'm removing DPIC everywhere as nothing really uses it.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 00:11:48 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 3:
I'm going to have a new upload for this. It turns out debug info for LLVM is very large for release builds, so I'll be turning that off. I'll also be turning off debug info for Kudu's toolchain build (which also includes LLVM).
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Jun 2024 16:30:00 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20870
to look at the new patch set (#2).
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
IMPALA-12686: Build the toolchain with basic debug information (-g1)
Basic debug information (e.g. -g1) is useful for getting better
stack traces. Compressing debug information (-gz) reduces the size
of this debug information dramatically. This adds both -g1 and -gz
to the default flags for the toolchain.
In order to have the flags apply uniformly, components need to
respect the existing CFLAGS and CXXFLAGS. Several components were
setting their own CFLAGS and CXXFLAGS without keeping the existing
flags, so this either fixes the components to keep existing flags
or removes the custom CFLAGS/CXXFLAGS.
Some components build with an extra -g after our flags and these
continue to build this way. Specifically, the following using -g:
- Thrift (-g added in this change)
- Kudu
- ORC
- CCTZ
- bzip2
These keep the same debug information they had before, except that
-gz compresses it now. This change should not reduce the debug
information for any component.
This skips generating debug information for CMake and Mold as they
are just build tools.
Testing:
- Ran a build and examined what happened to the package sizes.
- Added verbosity to print the compiler commands and verified
the flags were present
Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
---
M init-compiler.sh
M source/arrow/build.sh
M source/bzip2/build.sh
M source/calloncehack/calloncehack/CMakeLists.txt
M source/cloudflarezlib/build.sh
M source/cmake/build.sh
M source/curl/build.sh
M source/flatbuffers/build.sh
M source/gflags/build.sh
M source/glog/build.sh
M source/libpfm/build.sh
M source/llvm/build-source-tarball.sh
M source/lz4/build.sh
M source/mold/build.sh
M source/orc/build.sh
M source/python/build.sh
M source/thrift/build.sh
M source/zlib/build.sh
18 files changed, 44 insertions(+), 39 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/70/20870/2
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20870
to look at the new patch set (#4).
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
IMPALA-12686: Build the toolchain with basic debug information (-g1)
Basic debug information (e.g. -g1) is useful for getting better
stack traces. Compressing debug information (-gz) reduces the size
of this debug information dramatically. This adds both -g1 and -gz
to the default flags for the toolchain.
In order to have the flags apply uniformly, components need to
respect the existing CFLAGS and CXXFLAGS. Several components were
setting their own CFLAGS and CXXFLAGS without keeping the existing
flags, so this either fixes the components to keep existing flags
or removes the custom CFLAGS/CXXFLAGS.
Some components build with an extra -g after our flags and these
continue to build this way. Specifically, the following using -g:
- Thrift (-g added in this change)
- Kudu
- ORC
- CCTZ
- bzip2
These keep the same debug information they had before, except that
-gz compresses it now. This change should not reduce the debug
information for any component.
This skips generating debug information for a few things:
1. Build tools like CMake and Mold don't need debuginfo
2. LLVM's release build has an enormous amount of debuginfo, even
at -g1. If this uses debuginfo for LLVM, it adds hundreds of
MB to impalad's binary size. This skips that for now.
3. Kudu's thirdparty includes a build of LLVM, so this turns off
debuginfo for Kudu's thirdparty.
Testing:
- Ran a build and examined what happened to the package sizes.
The total size of the toolchain directory for Impala went down
from 7.08GB to 6.51GB. Kudu got much smaller due to compressing
debuginfo. Most other libraries got larger.
- Ran an Impala build to look at what happened to the binary size.
This adds about 31MB to the compressed binary size. It has no
impact on the stripped size.
- Added verbosity to print the compiler commands and verified
the flags were present
Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
---
M init-compiler.sh
M source/arrow/build.sh
M source/bzip2/build.sh
M source/calloncehack/calloncehack/CMakeLists.txt
M source/cloudflarezlib/build.sh
M source/cmake/build.sh
M source/curl/build.sh
M source/flatbuffers/build.sh
M source/gflags/build.sh
M source/glog/build.sh
M source/kudu/build.sh
M source/libpfm/build.sh
M source/llvm/build-source-tarball.sh
M source/lz4/build.sh
M source/mold/build.sh
M source/orc/build.sh
M source/python/build.sh
M source/thrift/build.sh
M source/zlib/build.sh
19 files changed, 58 insertions(+), 40 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/70/20870/4
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh
File source/cmake/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh@27
PS2, Line 27: # Drop the -g1/-gz flags for CMake, as it is just a build utility and does not
> I think it might be related to ccache. We use ccache 3.3.3, but 3.3.5 relea
Also, adding CMAKE_BUILD_TYPE=Release for mold seems to matter in my local tests, so I'm adding that.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 21:25:12 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh
File init-compiler.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@52
PS1, Line 52: # ARCH_FLAGS are used to convey architectur dependent flags that should
Want to fix the typo "architectur" while you're making changes?
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@102
PS1, Line 102: CFLAGS="$ARCH_FLAGS -fPIC -O3 -g1 -gz"
Looks like CFLAGS did not previously have all the ARCH_FLAGS.
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh
File source/arrow/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh@32
PS1, Line 32: CFLAGS="$CFLAGS -fPIC -DPIC" wrap make VERBOSE=1 -j${BUILD_THREADS:-4} all install
-fPIC is redundant now. Should we set -DPIC everywhere too?
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh
File source/bzip2/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh@33
PS1, Line 33: CFLAGS="$CFLAGS -fPIC -DPIC" wrap make -f Makefile-libbz2_so CFLAGS="$CFLAGS -fPIC -DPIC"
Why declare CFLAGS twice?
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh
File source/flatbuffers/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh@33
PS1, Line 33: CXXFLAGS=${CXXFLAGS}
Is this line even needed?
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh
File source/lz4/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh@32
PS1, Line 32: CFLAGS="$CFLAGS -fPIC"
This whole line is unnecessary, it already includes -fPIC.
http://gerrit.cloudera.org:8080/#/c/20870/1/source/orc/build.sh
File source/orc/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/orc/build.sh@47
PS1, Line 47: wrap make VERBOSE=1 -j${BUILD_THREADS:-4} install
This removed -DPIC. I asked elsewhere if we should add that globally.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jan 2024 22:21:35 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 5: Verified+1
Built this and ran with a corresponding Impala change, all tests pass. Going ahead with this.
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Sep 2024 15:38:30 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20870 )
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
Patch Set 2: Code-Review+1
(7 comments)
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh
File init-compiler.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@52
PS1, Line 52: # ARCH_FLAGS are used to convey architecture dependent flags that should
> Done
Ack
http://gerrit.cloudera.org:8080/#/c/20870/1/init-compiler.sh@102
PS1, Line 102: # -gdwarf-4 sets the DWARF version to 4. It is better to be explicit about the DWARF
> Yeah, that is odd. It seems cleaner to have CFLAGS to contain ARCH_FLAGS.
Ack
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh
File source/arrow/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/arrow/build.sh@32
PS1, Line 32: -DCMAKE_INSTALL_PREFIX=$LOCAL_INSTALL -DCMAKE_BUILD_TYPE=RELEASE
> Specifying it at the top level caused the LLVM build to fail. I removed DPI
Ack
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh
File source/bzip2/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/bzip2/build.sh@33
PS1, Line 33: # This command doesn't respect CFLAGS through the environment variable, so this
> Something about the second command doesn't respect the CFLAGS environment v
Ack
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh
File source/cmake/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/2/source/cmake/build.sh@27
PS2, Line 27: # Drop the -g1/-gz flags for CMake, as it is just a build utility and does not
Did adding -g0 not work?
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh
File source/flatbuffers/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/flatbuffers/build.sh@33
PS1, Line 33: GCC_MAJOR_VERSION=$(echo $GCC_VERSION | cut -d . -f1)
> Removed
Ack
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh
File source/lz4/build.sh:
http://gerrit.cloudera.org:8080/#/c/20870/1/source/lz4/build.sh@32
PS1, Line 32: cd build/cmake
> Good point, fixed
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 May 2024 17:42:43 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12686: Build the toolchain with basic debug information (-g1)
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20870
to look at the new patch set (#3).
Change subject: IMPALA-12686: Build the toolchain with basic debug information (-g1)
......................................................................
IMPALA-12686: Build the toolchain with basic debug information (-g1)
Basic debug information (e.g. -g1) is useful for getting better
stack traces. Compressing debug information (-gz) reduces the size
of this debug information dramatically. This adds both -g1 and -gz
to the default flags for the toolchain.
In order to have the flags apply uniformly, components need to
respect the existing CFLAGS and CXXFLAGS. Several components were
setting their own CFLAGS and CXXFLAGS without keeping the existing
flags, so this either fixes the components to keep existing flags
or removes the custom CFLAGS/CXXFLAGS.
Some components build with an extra -g after our flags and these
continue to build this way. Specifically, the following using -g:
- Thrift (-g added in this change)
- Kudu
- ORC
- CCTZ
- bzip2
These keep the same debug information they had before, except that
-gz compresses it now. This change should not reduce the debug
information for any component.
This skips generating debug information for CMake and Mold as they
are just build tools.
Testing:
- Ran a build and examined what happened to the package sizes.
- Added verbosity to print the compiler commands and verified
the flags were present
Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
---
M init-compiler.sh
M source/arrow/build.sh
M source/bzip2/build.sh
M source/calloncehack/calloncehack/CMakeLists.txt
M source/cloudflarezlib/build.sh
M source/cmake/build.sh
M source/curl/build.sh
M source/flatbuffers/build.sh
M source/gflags/build.sh
M source/glog/build.sh
M source/libpfm/build.sh
M source/llvm/build-source-tarball.sh
M source/lz4/build.sh
M source/mold/build.sh
M source/orc/build.sh
M source/python/build.sh
M source/thrift/build.sh
M source/zlib/build.sh
18 files changed, 43 insertions(+), 40 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/70/20870/3
--
To view, visit http://gerrit.cloudera.org:8080/20870
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee2e264b281f83ebc226d9bf7d4e5a99a52f1fc6
Gerrit-Change-Number: 20870
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>