You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/09/28 17:43:31 UTC
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20519
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
IMPALA-12354: Find aarch64 packages
TODO: update toolchain to a version with aarch64 packages
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
3 files changed, 8 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/1
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 5:
https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/44/ built and passed most tests. It has a few failures that I need to look into, but I don't believe they're new with this patch.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2023 18:07:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 9:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/14148/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Oct 2023 17:19:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9818/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2023 16:42:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/10/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/20519/10/bin/bootstrap_toolchain.py@256
PS10, Line 256: # Most common return values for machine are x86_64 or aarch64
: arch = platform.machine()
: if arch not in ['aarch64', 'x86_64']:
: raise Exception("Unsupported architecture '{}' for pre-built native-toolchain. "
: "Fetch and build it locally by setting NATIVE_TOOLCHAIN_HOME".format(arch))
: toolchain_build_id = os.environ["IMPALA_TOOLCHAIN_BUILD_ID_{}".format(arch.upper())]
I see we are using a different build number per arch, but the toolchain also adds the arch to the tarball name. Where does that get incorporated?
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Oct 2023 17:52:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#10).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Testing:
- successfully built on ARM, and tests passed (with a few known
exceptions)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 19 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/10
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#4).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Adds an aarch64 (64-bit ARM) native-toolchain build.
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 13 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/4
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#3).
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
IMPALA-12354: Find aarch64 packages
TODO: update x86_64 and verify refs match
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 8 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/3
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#5).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Adds an aarch64 (64-bit ARM) native-toolchain build.
Testing:
- successfully built on ARM, and passed most tests
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 13 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/5
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11: Verified+1
Only failure was IMPALA-12500.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2023 21:12:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 10:
> Patch Set 10: Code-Review+1
>
> This is going to require a first toolchain build conforming to the new naming convention, isn't it?
We can hold it up for https://gerrit.cloudera.org/c/20539/, yes.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Oct 2023 20:43:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#11).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Updates package version to include the architecture tag to match how
native-toolchain now names them.
Testing:
- successfully built on ARM, and tests passed (exceptions noted in
IMPALA-12490)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 21 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/11
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14107/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 18:10:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 5:
(4 comments)
Thanks for the extension!
As far as the toolchain storage structure is concerned I would prefer to see a more explicit distinction than just a different TOOLCHAIN_BUILD_ID value between binary packages targeted at different CPU architectures.
This patch can get us started though.
I also added requests for comments at a few locations.
http://gerrit.cloudera.org:8080/#/c/20519/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20519/5//COMMIT_MSG@10
PS5, Line 10:
Could you add some high-level details on the structure you're building here?
Most Impala devs don't touch the toolchain often, so a summary could help their
understanding on how this changes the toolchain layout and the way Impala picks
the right bits to download.
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/bootstrap_toolchain.py@257
PS5, Line 257: arch = platform.machine().upper()
Would it make sense to add a check for the known and supported architectures
and an explicit error message if something unknown is encountered?
This would be easier to understand than having to debug a failed download.
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh@88
PS5, Line 88: export IMPALA_TOOLCHAIN_BUILD_ID_X86_64=25-ba9ab5f36c
: export IMPALA_TOOLCHAIN_BUILD_ID_AARCH64=4-ba9ab5f36c
What happens if the build numbers happened to clash?
Would it be reasonable to add the architecture tag to the directory structure/file names for non-x86 packages?
Currently the toolchain's naming convention requires two correct "coordinates" for a package download:
you have the toolchain build ID and the package version, and both are part of the package's complete URL.
If you accidentally misconfigure one of them, you end up with a download error instead of the wrong version of the package.
Ideally I'd like to see something similar for the architecture as well, since that is just another dimension for which you
need to provide a coordinate: x86 or ARM? (or maybe something else, who knows).
Adding the arch to the package name somewhere could safeguard against mismatched libraries a lot earlier than linker errors.
I do realize that this would require a change in native-toolchain, so all this can be handled in follow-up patches,
but it would be great to add some mechanism like this.
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh@95
PS5, Line 95: #*-
Could you add a brief comment here pointing out the Bash magic? It's subtle, some folks may miss it
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Oct 2023 20:07:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#8).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Testing:
- successfully built on ARM, and tests passed (with a few known
exceptions)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/impala-config.sh
2 files changed, 21 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/8
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/8/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/8/bin/impala-config.sh@104
PS8, Line 104: if [[ "${TOOLCHAIN_ARCH}" = aarch64 ]]; then
Actually I think this is worse. It's more complicated and error-prone to implement in shell, and takes the selection further from where it's used which means I forgot to skip it when NATIVE_TOOLCHAIN_HOME is set which would be the actual override).
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Oct 2023 16:59:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 10: Code-Review+1
This is going to require a first toolchain build conforming to the new naming convention, isn't it?
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Oct 2023 20:34:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Updates package version to include the architecture tag to match how
native-toolchain now names them.
Testing:
- successfully built on ARM, and tests passed (exceptions noted in
IMPALA-12490)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Reviewed-on: http://gerrit.cloudera.org:8080/20519
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 21 insertions(+), 15 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Michael Smith: Verified
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 12
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9788/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2023 21:33:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2023 02:08:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14147/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Oct 2023 22:34:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20519/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20519/5//COMMIT_MSG@10
PS5, Line 10: adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
> Could you add some high-level details on the structure you're building here
Done
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/bootstrap_toolchain.py@257
PS5, Line 257: toolchain_host = os.environ["IMPALA_TOOLCHAIN_HOST"]
> Would it make sense to add a check for the known and supported architecture
Decided to move this logic to impala-config.sh to make it easier to inspect or override if needed.
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh@88
PS5, Line 88: IMPALA_TOOLCHAIN_BUILD_ID_aarch64=4-ba9ab5f36c
: IMPALA_TOOLCHAIN_BUILD_ID_x86_64=25-ba9ab5f36c
> What happens if the build numbers happened to clash?
If I'm doing something like that, I'm tempted to change the package name since that opens up using the same TOOLCHAIN_BUILD_ID for both architectures at some point in the future.
Something like https://gerrit.cloudera.org/c/20539/. It would require a corresponding change in bootstrap_toolchain.py to download the right package.
http://gerrit.cloudera.org:8080/#/c/20519/5/bin/impala-config.sh@95
PS5, Line 95: rem
> Could you add a brief comment here pointing out the Bash magic? It's subtle
Done
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Oct 2023 22:03:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9818/
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2023 21:08:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/14109/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 21:14:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
IMPALA-12354: Find aarch64 packages
TODO: update x86_64 and verify refs match
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 8 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/2
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14110/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 21:51:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14146/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Oct 2023 22:31:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11:
https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/48/ passed with the full stack (IMPALA-12354, IMPALA-12484, IMPALA-12490).
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2023 16:42:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#9).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Testing:
- successfully built on ARM, and tests passed (with a few known
exceptions)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M buildall.sh
4 files changed, 19 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/9
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/1/bin/impala-config.sh@90
PS1, Line 90: ${IMPALA_TOOLCHAIN_REPO:-https://github.infra.cloudera.com/michaelsmith/native-toolchain.git}
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 17:44:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Find aarch64 packages
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Find aarch64 packages
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/1/bin/impala-config.sh@88
PS1, Line 88: export IMPALA_TOOLCHAIN_BUILD_ID=24-2afac3b80b
Initially going to use different BUILD_IDs for aarch64 and x86_64. We'll want to error if they use different hashes, and select the appropriate build based on system architecture.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 17:45:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14111/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Sep 2023 22:23:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14149/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Oct 2023 17:46:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20519
to look at the new patch set (#7).
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
IMPALA-12354: Add aarch64 native-toolchain build
Pre-built toolchains are identified by a TOOLCHAIN_BUILD_ID. This commit
adds an aarch64 (64-bit ARM) native-toolchain build, separate from the
x86_64 native-toolchain build, with its own environment variable set in
impala-config.sh. bootstrap_toolchain.py selects which version to use
based on 'uname -m'.
impala-config.sh also verifies that IMPALA_TOOLCHAIN_BUILD_ID_AARCH64
and IMPALA_TOOLCHAIN_BUILD_ID_X86_64 were produced from the same
native-toolchain ref by checking the 2nd token of the build ID.
Testing:
- successfully built on ARM, and tests passed (with a few known
exceptions)
Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
---
M bin/bootstrap_system.sh
M bin/impala-config.sh
2 files changed, 21 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/20519/7
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/7/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/20519/7/bin/impala-config.sh@99
PS7, Line 99: echo "IMPALA_TOOLCHAIN_BUILD_ID_aarch64 and IMPALA_TOOLCHAIN_BUILD_ID_x86_64 must come" \
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Oct 2023 22:04:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20519/10/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/20519/10/bin/bootstrap_toolchain.py@256
PS10, Line 256: # Most common return values for machine are x86_64 or aarch64
: arch = platform.machine()
: if arch not in ['aarch64', 'x86_64']:
: raise Exception("Unsupported architecture '{}' for pre-built native-toolchain. "
: "Fetch and build it locally by setting NATIVE_TOOLCHAIN_HOME".format(arch))
: toolchain_build_id = os.environ["IMPALA_TOOLCHAIN_BUILD_ID_{}".format(arch.upper())]
> I see we are using a different build number per arch, but the toolchain als
Done
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2023 04:21:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Oct 2023 04:50:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 5:
> Patch Set 5:
>
> https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/44/ built and passed most tests. It has a few failures that I need to look into, but I don't believe they're new with this patch.
Test failures match https://jenkins.impala.io/job/ubuntu-20.04-from-scratch-ARM/28/, the last ARM run we did against main.
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2023 21:17:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12354: Add aarch64 native-toolchain build
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20519 )
Change subject: IMPALA-12354: Add aarch64 native-toolchain build
......................................................................
Patch Set 11: Code-Review+2
This looks good to me
--
To view, visit http://gerrit.cloudera.org:8080/20519
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9bfa7125dbc647b33041c5572d97b7f7ccad6258
Gerrit-Change-Number: 20519
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2023 18:32:33 +0000
Gerrit-HasComments: No