You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2023/09/08 22:55:53 UTC
[native-toolchain-CR] IMPALA-12436: Build multi-platform builder images
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20469
Change subject: IMPALA-12436: Build multi-platform builder images
......................................................................
IMPALA-12436: Build multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal.
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
---
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
6 files changed, 56 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/1
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58
PS5, Line 58: build_cmd += ['--builder', args.builder]
Building everything in the same builder seems to require a fairly new version of buildx. Docker 20 wasn't new enough; I'm using Docker 24 locally.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Sep 2023 23:23:05 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: # Non-default builder only used for multi-platform bui
> I didn't think so, but I'm open to adding it.
Adding the param helped decouple this change from updating the Jenkins environment to a newer Docker version.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 22:28:37 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20469
to look at the new patch set (#5).
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
IMPALA-12436: Create multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal.
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
---
M docker/README.md
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
7 files changed, 73 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/5
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py@29
PS6, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204']
Maybe this should be an environment variable? Although we'll need to make changes to other images to build on ARM. Things I've noticed:
- the centos 7 tag we use doesn't have ARM images
- Python 2.7 seems to fail hard building PyYAML on ARM, so they all need to pull in python3-devel and pip3
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 21:39:00 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG@17
PS5, Line 17: Moves the publishing step to happen within buildx, because buildx does
: not support loading multi-platform images in the local Docker registry.
: This does mean that a build failure could result in publishing only some
: of the images. These are not updated frequently and are usually preceded
: by a test run, so this risk seems minimal.
> Can you update this to describe how we avoid this problem with the two pass
Done
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md
File docker/README.md:
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md@45
PS5, Line 45: and
> Nit: and
Done
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58
PS5, Line 58: build_cmd += ['--builder', args.builder]
> Does that impact the x86 single-arch build? Or is that basically fine becau
I think it only impacts when you use a docker-container builder (for multi-platform builds), the default "docker" builder works fine.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 17:09:07 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20469
to look at the new patch set (#7).
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
IMPALA-12436: Create multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal. The jenkins/build.sh script
avoids this issue by first running a build without publishing, then when
it succeeds running again with publishing (which will largely pull image
layers from the build cache so should be very fast).
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
---
M docker/README.md
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
7 files changed, 87 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/7
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py@29
PS6, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204']
> Maybe this should be an environment variable? Although we'll need to make c
Not going to change it, if we're producing multi-platform containers we should always do it for all platforms we support.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Sep 2023 17:52:20 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29
PS2, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204']
:
: def main():
> Yeah, the normal path is fine, so it's only the --multi path that would be
Done
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32
PS2, Line 32: nfig(leve
> I agree that we don't really need Redhat 9 ARM at the moment, so even if we
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Sep 2023 21:52:53 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Build multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Build multi-platform builder images
......................................................................
Patch Set 1:
This is the direction I'm thinking for creating multi-platform builder images.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Sep 2023 22:56:45 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20469
to look at the new patch set (#6).
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
IMPALA-12436: Create multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal. The jenkins/build.sh script
avoids this issue by first running a build without publishing, then when
it succeeds running again with publishing (which will largely pull image
layers from the build cache so should be very fast).
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
---
M docker/README.md
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
7 files changed, 78 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/6
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Joe McDonnell,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20469
to look at the new patch set (#3).
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
IMPALA-12436: Create multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal.
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
---
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
6 files changed, 54 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/3
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29
PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86
: # then create a docker-container builder
: # docker buildx create --use
> I think the errors you get from Docker are pretty clear. These are only req
Yeah, the normal path is fine, so it's only the --multi path that would be a problem. I noticed that we have a README.md file in this directory with some instructions for building the docker images. Maybe we could add a section there about what to install, how to get a builder, how to invoke buildall.py?
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32
PS2, Line 32: 'redhat8'
> We could if we think we'll move CDW to it sometime soon. I was mostly tryin
I agree that we don't really need Redhat 9 ARM at the moment, so even if we produce the docker images, we can probably skip the full build until later.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Sep 2023 21:34:20 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: ./buildall.py --multi --builder=${BUILDER} ${DISTRO_PARAM}
Would it make sense to have an environment variable / Jenkins parameter that toggles whether to do the multi-arch build or a single arch build?
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:23:21 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 7: Code-Review+2
This looks good to me. Sometimes we use the docker images for older builds of native toolchain for older Impala branches. Probably we want to verify that these images continue to work for older toolchains (this is mainly about Redhat 8 and Ubuntu 20). I don't think the version of Python 3 or any of the other changes will matter for those purposes.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Sep 2023 19:36:48 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
IMPALA-12436: Create multi-platform builder images
Updates the toolchain build image creation process to support publishing
multi-platform (aarch64 and amd64) images.
Uses docker buildx to produce multi-platform images. This requires QEMU
on the builder; these builds are quick, so the overhead of emulating arm
is minimal. Currently assumes images are built on amd64 as not all
builds specify `--platform`.
Moves the publishing step to happen within buildx, because buildx does
not support loading multi-platform images in the local Docker registry.
This does mean that a build failure could result in publishing only some
of the images. These are not updated frequently and are usually preceded
by a test run, so this risk seems minimal. The jenkins/build.sh script
avoids this issue by first running a build without publishing, then when
it succeeds running again with publishing (which will largely pull image
layers from the build cache so should be very fast).
Python 2.7 ran into build issues installing awscli on ARM, so OS
targeting ARM were updated to ensure they include Python 3.8+ and
install a newer version of awscli in Python 3.
New options were added to buildall.py:
- registry: publish images to the specified registry
- multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22
Multi-platform builds require QEMU and a non-default buildx builder.
Jenkins job is updated to install QEMU, create a builder, and build
multi-platform images.
Multi-platform images can't be loaded into a local Docker context (where
they would be listed via docker images); we continue to load them there
by default with --load, but disable it when doing multi-platform builds.
The only way to get images out of the builder for multi-platform builds
is to specify a registry.
Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Reviewed-on: http://gerrit.cloudera.org:8080/20469
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
---
M docker/README.md
M docker/all/postinstall.sh
M docker/buildall.py
M docker/redhat8.df
M docker/ubuntu2004.df
M docker/ubuntu2204.df
M jenkins/build.sh
7 files changed, 87 insertions(+), 32 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Michael Smith: Verified
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 7: Verified+1
Verified these containers can be used to build current and older versions of native-toolchain.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Sep 2023 20:50:22 +0000
Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG@17
PS5, Line 17: Moves the publishing step to happen within buildx, because buildx does
: not support loading multi-platform images in the local Docker registry.
: This does mean that a build failure could result in publishing only some
: of the images. These are not updated frequently and are usually preceded
: by a test run, so this risk seems minimal.
Can you update this to describe how we avoid this problem with the two passes?
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md
File docker/README.md:
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md@45
PS5, Line 45: acd
Nit: and
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58
PS5, Line 58: build_cmd += ['--builder', args.builder]
> Building everything in the same builder seems to require a fairly new versi
Does that impact the x86 single-arch build? Or is that basically fine because it is using the default builder?
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:00:41 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: ./buildall.py --multi --builder=${BUILDER} ${DISTRO_PARAM}
> Would it make sense to have an environment variable / Jenkins parameter tha
I didn't think so, but I'm open to adding it.
If you push to a registry, then we want to build with multi. If you don't, then we don't really need the output. Main scenario might be to test building a single platform, but if you want to test that without multi it's easy enough to do locally.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Sep 2023 17:05:07 +0000
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py
File docker/buildall.py:
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29
PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86
: # then create a docker-container builder
: # docker buildx create --use
> Nit: Is there anything we can do to detect if these are appropriately speci
I think the errors you get from Docker are pretty clear. These are only required if specifying '--multi', I'll update the comment to clarify that.
I made sure most local development workflows would not require QEMU or a new builder, so these should only be needed if testing out multi-platform builds specifically.
http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32
PS2, Line 32: 'redhat8'
> Should we include redhat9?
We could if we think we'll move CDW to it sometime soon. I was mostly trying to keep native-toolchain build varieties down to ones we would use.
http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh@65
PS2, Line 65: popd
> Something we could do to make sure we only push if all images succeeded is
Done
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Sep 2023 21:12:04 +0000
Gerrit-HasComments: Yes