You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2019/05/06 20:46:42 UTC
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13249
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Fix issues with toolchain Python and bump version
* Bumps the version to 2.7.16, the latest Python 2 release.
* Fixes issues where paths like
/tmp/tmp.mEyNqPNTxH-impala-toolchain/gcc got baked into the
Python metadata, which caused problems when later compiling
C/C++ extensions from source.
* Remove hardcoded version in Kudu build scripts
This is motivated by IMPALA-8508, where we want to consume the toolchain
python outside of the toolchain.
Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
---
M buildall.sh
M source/kudu/build.sh
M source/python/build.sh
3 files changed, 18 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/49/13249/1
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Hector Acosta (Code Review)" <ge...@cloudera.org>.
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh
File source/python/build.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh@37
PS1, Line 37: # a different directory from the one used during our toolchain build.
> That makes sense. I think that's the default way that ccache installs, anyw
The problem here (besides what Tim mentioned) was that then we'd have to insert paths ($PATH) after $CCACHE_PATH..
more here:
https://gerrit.cloudera.org/#/c/12574/3//COMMIT_MSG
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Hector Acosta <he...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 19:59:46 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 15:32:59 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1: Verified+1
I did a test build on all OSes.
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 18:06:03 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh@57
PS1, Line 57: PYTHON_VERSION=2.7.16 $SOURCE_DIR/source/python/build.sh
> do you know what the history of darwin also building a "fake package" for 2
It was some cruft from back when Martin was trying to get things working on OS X. It hasn't been maintained.
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh@59
PS1, Line 59: PYTHON_VERSION=2.7.16 build_fake_package "python"
> I suppose we don't care about the BUILD_HISTORICAL option to keep building
Mostly no, it's basically vestigial now. I don't think any consumers of the toolchain were actually using the Python anyway, it was only used during the build process.
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh
File source/python/build.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh@37
PS1, Line 37: # a different directory from the one used during our toolchain build.
> do we have any issues if native-toolchain gets built with ccache or distcc?
Hector recently got it working with ccache. He resorted to having a script called "gcc" that calls "ccache gcc" internally to avoid issues with packages that can't handle spaces in $CC/$CXX.
The gcc should always be the one built earlier in the toolchain.
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 17:55:37 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 2: Verified+1 Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 18:06:28 +0000
Gerrit-HasComments: No
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh@57
PS1, Line 57: PYTHON_VERSION=2.7.16 $SOURCE_DIR/source/python/build.sh
do you know what the history of darwin also building a "fake package" for 2.7.11?
http://gerrit.cloudera.org:8080/#/c/13249/1/buildall.sh@59
PS1, Line 59: PYTHON_VERSION=2.7.16 build_fake_package "python"
I suppose we don't care about the BUILD_HISTORICAL option to keep building the old versions?
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh
File source/python/build.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh@37
PS1, Line 37: # a different directory from the one used during our toolchain build.
do we have any issues if native-toolchain gets built with ccache or distcc? Or is this GCC always the gcc that was built earlier in the toolchain build?
Maybe we should explicitly check that CC == "gcc" so we have confidence it will produce a working package?
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 16:45:06 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Fix issues with toolchain Python and bump version
* Bumps the version to 2.7.16, the latest Python 2 release.
* Fixes issues where paths like
/tmp/tmp.mEyNqPNTxH-impala-toolchain/gcc got baked into the
Python metadata, which caused problems when later compiling
C/C++ extensions from source.
* Remove hardcoded version in Kudu build scripts
This is motivated by IMPALA-8508, where we want to consume the toolchain
python outside of the toolchain.
Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Reviewed-on: http://gerrit.cloudera.org:8080/13249
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
---
M buildall.sh
M source/kudu/build.sh
M source/python/build.sh
3 files changed, 18 insertions(+), 9 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 )
Change subject: Fix issues with toolchain Python and bump version
......................................................................
Patch Set 1: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh
File source/python/build.sh:
http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh@37
PS1, Line 37: # a different directory from the one used during our toolchain build.
> Hector recently got it working with ccache. He resorted to having a script
That makes sense. I think that's the default way that ccache installs, anyway -- it makes a /usr/lib/ccache/ directory with binaries like 'gcc', and then ccache itself looks at argv[0]
--
To view, visit http://gerrit.cloudera.org:8080/13249
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672
Gerrit-Change-Number: 13249
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Thomas Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 07 May 2019 18:00:56 +0000
Gerrit-HasComments: Yes