You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Robert Munteanu <ro...@apache.org> on 2022/12/06 12:43:53 UTC
Multi-arch container images on DockerHub
Hi,
We had a user report that our official Docker image does not support
architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't run
the image with our current setup.
On our side, we set up automated builds on Docker Hub using automated
builds. Unforunately, Docker Hub autobuilds don't support `docker
buildx` or another form of multi-arch builds. It is not the roadmap
though [2], but here is no guarantee on when (or if) it will become
available.
I see two alternatives so far:
1. Moving to GitHub actions
2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
Hub
To be honest, none is too appealing to me, we have a simple process
that works.
How are other projects handling this? Or does anyone have any ideas
that they can share?
Thanks,
Robert
[1]: https://issues.apache.org/jira/browse/SLING-11714
[2]: https://github.com/docker/roadmap/issues/109
Re: Multi-arch container images on DockerHub
Posted by Robert Munteanu <ro...@apache.org>.
Hi Jarek, Zach, and Allen,
That is a lot of good material, thank you. Looks like most people are
using GH actions, so I'll take this to our dev list, we will probably
do the same.
Thanks,
Robert
On Tue, 2022-12-06 at 18:56 +0100, Jarek Potiuk wrote:
> Good one Zach. Yeah, I also played with building separately and
> combining
> the resulting image (`regctl` is a great tool that might help with
> it).
>
> But I found it quite brittle and "manual" - I really like the buildx
> approach when you can have multiple builder instances running in
> parallel.
> And ssh-forwarding of docker sockets is a cool thing :). I even used
> reverse ssh port forwarding to docker on my Mac to build images in
> parallel
> when I ssh-ed from my Mac M1 to my home desktop linux and run the
> buildx
> command on Linux to build both images (this triggered the build back
> on my
> Mac).
>
> That was fun :)
>
> J.
>
>
>
> On Tue, Dec 6, 2022 at 5:48 PM Zach Hoffman <zr...@apache.org>
> wrote:
>
> > Never mind, the first link was better
> >
> > -Zach
> >
> > On Tue, Dec 6, 2022 at 9:46 AM Zach Hoffman <zr...@apache.org>
> > wrote:
> >
> > > Oops, the container link got mangled, it was supposed to be <
> > >
> > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine
> > > > .
> > >
> > > On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman
> > > <zr...@apache.org>
> > wrote:
> > >
> > > > in Traffic Control, we build an Alpine image <
> > > >
> > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine
> > > ,
> > > > with Traffic Server baked into it, for amd64 and arm64 . Rather
> > > > than
> > > > building them both from the same machine, the amd64 and arm64
> > > > images in
> > > > separate GitHub Actions jobs, then combined using `docker
> > > > manifest` in a
> > > > third GitHub Actions job <
> > > >
> > https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml
> > > > > .
> > > >
> > > > The arm64 part still takes much longer (over 2 hours, compare
> > > > to 10
> > > > minutes for amd64), but this approach eliminates the need to
> > > > build for
> > more
> > > > than one platform at a time, so if the arm64 job were to run on
> > > > an arm64
> > > > runner in the future, that wouldn't need to further complicate
> > > > the other
> > > > jobs in order to take advantage of that speedup.
> > > >
> > > > -Zach
> > > >
> > > > On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > >
> > > > > In Airflow we have a bit more complex setting (we are
> > > > > building 2x5x2
> > > > > different images and they are different sets of them for
> > > > > different
> > > > > branches), Building images for Airflow takes quite some time
> > (installing
> > > > > many dependencies) so qemu was out of the question (several
> > > > > hours to
> > > > > build
> > > > > single image). Unfortunately qemu is ~ 15 times (!) slower
> > > > > than
> > hardware
> > > > > builds from our experience. Good enough for small images, but
> > > > > unacceptable
> > > > > if your image normally takes time
> > > > >
> > > > > The built-in actions like the one mentioned by Jacob have the
> > limitations
> > > > > that they depende on qemu - I have not yet found an easy
> > > > > "Action" that
> > > > > could utilise AMD and ARM hardware together. But maybe there
> > > > > are other
> > > > > ways.
> > > > >
> > > > > We developed our own scripting and tooling:
> > > > >
> > > > > * we have self-hosted runners in GitHub and we only use those
> > > > > to build
> > > > > images (with Astronomer money and Amazon-sponsored credits on
> > > > > AWS)
> > > > > * for the time when we build multi-platform build we start
> > > > > ARM
> > instances
> > > > > (they have built-in timed auto-kill-switch so that they are
> > > > > not run
> > > > > indefinitely and take resources)
> > > > > * we configured ssh-forwarded docker port from our AMD
> > > > > instance to
> > > > > forward
> > > > > docker socket
> > > > > * we configured two builder with buildx (local for AMD and
> > > > > the
> > forwarded
> > > > > one for ARM)
> > > > > * we run multi-platform buildx build to use both builders.
> > > > > * we have our own Python build/development environment
> > > > > (breeze) that
> > > > > wraps
> > > > > the docker commands that are executed so the "code" doing it
> > > > > is not
> > easy
> > > > > to
> > > > > copy elsewhere sometimes, but I copied below the actual
> > > > > "meat" what
> > > > > happens
> > > > > under the hood.
> > > > > * we run it as part of our Github Action workflows
> > > > >
> > > > > Examples:
> > > > >
> > > > > * Starting ARM instance and configuring builders:
> > > > >
> > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
> > > > >
> > > > > This is the relevant part of establishing the worker for
> > > > > buildx (I omit
> > > > > starting the instance as it is amazon-specific):
> > > > >
> > > > > export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
> > > > > autossh -f "-L12357:/var/run/docker.sock" \
> > > > > -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no"
> > > > > \
> > > > > -i "${WORKING_DIR}/my_key" "${EC2_USER}@
> > > > > ${INSTANCE_PRIVATE_DNS_NAME}"
> > > > >
> > > > > bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1
> > > > > 40`; do echo
> > -n
> > > > > .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit
> > > > > ; done;
> > > > > echo
> > > > > " Timeout!" >&2; exit 1'
> > > > >
> > > > > docker buildx rm --force airflow_cache || true
> > > > > docker buildx create --name airflow_cache
> > > > > docker buildx create --name airflow_cache --append
> > > > > localhost:12357
> > > > >
> > > > > * Releasing image:
> > > > >
> > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
> > > > >
> > > > > This is the command that is run under-the-hood
> > > > >
> > > > > docker buildx build --builder airflow_cache --build-arg
> > > > > PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
> > > > > AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
> > > > > apache/airflow:2.5.0-python3.10 --push
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak
> > > > > <ja...@voltrondata.com.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hello Robert,
> > > > > >
> > > > > > I would suggest using GItHub Actions, there you can use the
> > > > > > official
> > > > > suite
> > > > > > of docker actions to build multiplatform images with little
> > > > > > need for
> > > > > custom
> > > > > > scripting [1].
> > > > > > Feel free to ping me in the ASF slack.
> > > > > >
> > > > > > Best
> > > > > > Jacob
> > > > > >
> > > > > > [1]: https://github.com/docker/build-push-action
> > > > > >
> > > > > > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu
> > > > > > <ro...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We had a user report that our official Docker image does
> > > > > > > not
> > support
> > > > > > > architectures other than AMD64 [1]. M1 Macs and Raspberry
> > > > > > > Pis can't
> > > > > run
> > > > > > > the image with our current setup.
> > > > > > >
> > > > > > > On our side, we set up automated builds on Docker Hub
> > > > > > > using
> > automated
> > > > > > > builds. Unforunately, Docker Hub autobuilds don't support
> > > > > > > `docker
> > > > > > > buildx` or another form of multi-arch builds. It is not
> > > > > > > the roadmap
> > > > > > > though [2], but here is no guarantee on when (or if) it
> > > > > > > will become
> > > > > > > available.
> > > > > > >
> > > > > > > I see two alternatives so far:
> > > > > > >
> > > > > > > 1. Moving to GitHub actions
> > > > > > > 2. Use hooks to install qemu and 'fake' a multi-arch
> > > > > > > build on
> > Docker
> > > > > > > Hub
> > > > > > >
> > > > > > > To be honest, none is too appealing to me, we have a
> > > > > > > simple process
> > > > > > > that works.
> > > > > > >
> > > > > > > How are other projects handling this? Or does anyone have
> > > > > > > any ideas
> > > > > > > that they can share?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Robert
> > > > > > >
> > > > > > > [1]: https://issues.apache.org/jira/browse/SLING-11714
> > > > > > > [2]: https://github.com/docker/roadmap/issues/109
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: Multi-arch container images on DockerHub
Posted by Jarek Potiuk <ja...@potiuk.com>.
Good one Zach. Yeah, I also played with building separately and combining
the resulting image (`regctl` is a great tool that might help with it).
But I found it quite brittle and "manual" - I really like the buildx
approach when you can have multiple builder instances running in parallel.
And ssh-forwarding of docker sockets is a cool thing :). I even used
reverse ssh port forwarding to docker on my Mac to build images in parallel
when I ssh-ed from my Mac M1 to my home desktop linux and run the buildx
command on Linux to build both images (this triggered the build back on my
Mac).
That was fun :)
J.
On Tue, Dec 6, 2022 at 5:48 PM Zach Hoffman <zr...@apache.org> wrote:
> Never mind, the first link was better
>
> -Zach
>
> On Tue, Dec 6, 2022 at 9:46 AM Zach Hoffman <zr...@apache.org> wrote:
>
> > Oops, the container link got mangled, it was supposed to be <
> >
> https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine
> > >.
> >
> > On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman <zr...@apache.org>
> wrote:
> >
> >> in Traffic Control, we build an Alpine image <
> >>
> https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine
> >,
> >> with Traffic Server baked into it, for amd64 and arm64 . Rather than
> >> building them both from the same machine, the amd64 and arm64 images in
> >> separate GitHub Actions jobs, then combined using `docker manifest` in a
> >> third GitHub Actions job <
> >>
> https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml
> >> >.
> >>
> >> The arm64 part still takes much longer (over 2 hours, compare to 10
> >> minutes for amd64), but this approach eliminates the need to build for
> more
> >> than one platform at a time, so if the arm64 job were to run on an arm64
> >> runner in the future, that wouldn't need to further complicate the other
> >> jobs in order to take advantage of that speedup.
> >>
> >> -Zach
> >>
> >> On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >>> In Airflow we have a bit more complex setting (we are building 2x5x2
> >>> different images and they are different sets of them for different
> >>> branches), Building images for Airflow takes quite some time
> (installing
> >>> many dependencies) so qemu was out of the question (several hours to
> >>> build
> >>> single image). Unfortunately qemu is ~ 15 times (!) slower than
> hardware
> >>> builds from our experience. Good enough for small images, but
> >>> unacceptable
> >>> if your image normally takes time
> >>>
> >>> The built-in actions like the one mentioned by Jacob have the
> limitations
> >>> that they depende on qemu - I have not yet found an easy "Action" that
> >>> could utilise AMD and ARM hardware together. But maybe there are other
> >>> ways.
> >>>
> >>> We developed our own scripting and tooling:
> >>>
> >>> * we have self-hosted runners in GitHub and we only use those to build
> >>> images (with Astronomer money and Amazon-sponsored credits on AWS)
> >>> * for the time when we build multi-platform build we start ARM
> instances
> >>> (they have built-in timed auto-kill-switch so that they are not run
> >>> indefinitely and take resources)
> >>> * we configured ssh-forwarded docker port from our AMD instance to
> >>> forward
> >>> docker socket
> >>> * we configured two builder with buildx (local for AMD and the
> forwarded
> >>> one for ARM)
> >>> * we run multi-platform buildx build to use both builders.
> >>> * we have our own Python build/development environment (breeze) that
> >>> wraps
> >>> the docker commands that are executed so the "code" doing it is not
> easy
> >>> to
> >>> copy elsewhere sometimes, but I copied below the actual "meat" what
> >>> happens
> >>> under the hood.
> >>> * we run it as part of our Github Action workflows
> >>>
> >>> Examples:
> >>>
> >>> * Starting ARM instance and configuring builders:
> >>>
> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
> >>>
> >>> This is the relevant part of establishing the worker for buildx (I omit
> >>> starting the instance as it is amazon-specific):
> >>>
> >>> export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
> >>> autossh -f "-L12357:/var/run/docker.sock" \
> >>> -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \
> >>> -i "${WORKING_DIR}/my_key" "${EC2_USER}@
> >>> ${INSTANCE_PRIVATE_DNS_NAME}"
> >>>
> >>> bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo
> -n
> >>> .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done;
> >>> echo
> >>> " Timeout!" >&2; exit 1'
> >>>
> >>> docker buildx rm --force airflow_cache || true
> >>> docker buildx create --name airflow_cache
> >>> docker buildx create --name airflow_cache --append localhost:12357
> >>>
> >>> * Releasing image:
> >>>
> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
> >>>
> >>> This is the command that is run under-the-hood
> >>>
> >>> docker buildx build --builder airflow_cache --build-arg
> >>> PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
> >>> AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
> >>> apache/airflow:2.5.0-python3.10 --push
> >>>
> >>> J.
> >>>
> >>>
> >>>
> >>> On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak
> >>> <ja...@voltrondata.com.invalid>
> >>> wrote:
> >>>
> >>> > Hello Robert,
> >>> >
> >>> > I would suggest using GItHub Actions, there you can use the official
> >>> suite
> >>> > of docker actions to build multiplatform images with little need for
> >>> custom
> >>> > scripting [1].
> >>> > Feel free to ping me in the ASF slack.
> >>> >
> >>> > Best
> >>> > Jacob
> >>> >
> >>> > [1]: https://github.com/docker/build-push-action
> >>> >
> >>> > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org>
> >>> wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > We had a user report that our official Docker image does not
> support
> >>> > > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't
> >>> run
> >>> > > the image with our current setup.
> >>> > >
> >>> > > On our side, we set up automated builds on Docker Hub using
> automated
> >>> > > builds. Unforunately, Docker Hub autobuilds don't support `docker
> >>> > > buildx` or another form of multi-arch builds. It is not the roadmap
> >>> > > though [2], but here is no guarantee on when (or if) it will become
> >>> > > available.
> >>> > >
> >>> > > I see two alternatives so far:
> >>> > >
> >>> > > 1. Moving to GitHub actions
> >>> > > 2. Use hooks to install qemu and 'fake' a multi-arch build on
> Docker
> >>> > > Hub
> >>> > >
> >>> > > To be honest, none is too appealing to me, we have a simple process
> >>> > > that works.
> >>> > >
> >>> > > How are other projects handling this? Or does anyone have any ideas
> >>> > > that they can share?
> >>> > >
> >>> > > Thanks,
> >>> > > Robert
> >>> > >
> >>> > > [1]: https://issues.apache.org/jira/browse/SLING-11714
> >>> > > [2]: https://github.com/docker/roadmap/issues/109
> >>> > >
> >>> >
> >>>
> >>
>
Re: Multi-arch container images on DockerHub
Posted by Zach Hoffman <zr...@apache.org>.
Never mind, the first link was better
-Zach
On Tue, Dec 6, 2022 at 9:46 AM Zach Hoffman <zr...@apache.org> wrote:
> Oops, the container link got mangled, it was supposed to be <
> https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine
> >.
>
> On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman <zr...@apache.org> wrote:
>
>> in Traffic Control, we build an Alpine image <
>> https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine>,
>> with Traffic Server baked into it, for amd64 and arm64 . Rather than
>> building them both from the same machine, the amd64 and arm64 images in
>> separate GitHub Actions jobs, then combined using `docker manifest` in a
>> third GitHub Actions job <
>> https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml
>> >.
>>
>> The arm64 part still takes much longer (over 2 hours, compare to 10
>> minutes for amd64), but this approach eliminates the need to build for more
>> than one platform at a time, so if the arm64 job were to run on an arm64
>> runner in the future, that wouldn't need to further complicate the other
>> jobs in order to take advantage of that speedup.
>>
>> -Zach
>>
>> On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> In Airflow we have a bit more complex setting (we are building 2x5x2
>>> different images and they are different sets of them for different
>>> branches), Building images for Airflow takes quite some time (installing
>>> many dependencies) so qemu was out of the question (several hours to
>>> build
>>> single image). Unfortunately qemu is ~ 15 times (!) slower than hardware
>>> builds from our experience. Good enough for small images, but
>>> unacceptable
>>> if your image normally takes time
>>>
>>> The built-in actions like the one mentioned by Jacob have the limitations
>>> that they depende on qemu - I have not yet found an easy "Action" that
>>> could utilise AMD and ARM hardware together. But maybe there are other
>>> ways.
>>>
>>> We developed our own scripting and tooling:
>>>
>>> * we have self-hosted runners in GitHub and we only use those to build
>>> images (with Astronomer money and Amazon-sponsored credits on AWS)
>>> * for the time when we build multi-platform build we start ARM instances
>>> (they have built-in timed auto-kill-switch so that they are not run
>>> indefinitely and take resources)
>>> * we configured ssh-forwarded docker port from our AMD instance to
>>> forward
>>> docker socket
>>> * we configured two builder with buildx (local for AMD and the forwarded
>>> one for ARM)
>>> * we run multi-platform buildx build to use both builders.
>>> * we have our own Python build/development environment (breeze) that
>>> wraps
>>> the docker commands that are executed so the "code" doing it is not easy
>>> to
>>> copy elsewhere sometimes, but I copied below the actual "meat" what
>>> happens
>>> under the hood.
>>> * we run it as part of our Github Action workflows
>>>
>>> Examples:
>>>
>>> * Starting ARM instance and configuring builders:
>>> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
>>>
>>> This is the relevant part of establishing the worker for buildx (I omit
>>> starting the instance as it is amazon-specific):
>>>
>>> export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
>>> autossh -f "-L12357:/var/run/docker.sock" \
>>> -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \
>>> -i "${WORKING_DIR}/my_key" "${EC2_USER}@
>>> ${INSTANCE_PRIVATE_DNS_NAME}"
>>>
>>> bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo -n
>>> .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done;
>>> echo
>>> " Timeout!" >&2; exit 1'
>>>
>>> docker buildx rm --force airflow_cache || true
>>> docker buildx create --name airflow_cache
>>> docker buildx create --name airflow_cache --append localhost:12357
>>>
>>> * Releasing image:
>>> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
>>>
>>> This is the command that is run under-the-hood
>>>
>>> docker buildx build --builder airflow_cache --build-arg
>>> PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
>>> AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
>>> apache/airflow:2.5.0-python3.10 --push
>>>
>>> J.
>>>
>>>
>>>
>>> On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak
>>> <ja...@voltrondata.com.invalid>
>>> wrote:
>>>
>>> > Hello Robert,
>>> >
>>> > I would suggest using GItHub Actions, there you can use the official
>>> suite
>>> > of docker actions to build multiplatform images with little need for
>>> custom
>>> > scripting [1].
>>> > Feel free to ping me in the ASF slack.
>>> >
>>> > Best
>>> > Jacob
>>> >
>>> > [1]: https://github.com/docker/build-push-action
>>> >
>>> > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org>
>>> wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > We had a user report that our official Docker image does not support
>>> > > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't
>>> run
>>> > > the image with our current setup.
>>> > >
>>> > > On our side, we set up automated builds on Docker Hub using automated
>>> > > builds. Unforunately, Docker Hub autobuilds don't support `docker
>>> > > buildx` or another form of multi-arch builds. It is not the roadmap
>>> > > though [2], but here is no guarantee on when (or if) it will become
>>> > > available.
>>> > >
>>> > > I see two alternatives so far:
>>> > >
>>> > > 1. Moving to GitHub actions
>>> > > 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
>>> > > Hub
>>> > >
>>> > > To be honest, none is too appealing to me, we have a simple process
>>> > > that works.
>>> > >
>>> > > How are other projects handling this? Or does anyone have any ideas
>>> > > that they can share?
>>> > >
>>> > > Thanks,
>>> > > Robert
>>> > >
>>> > > [1]: https://issues.apache.org/jira/browse/SLING-11714
>>> > > [2]: https://github.com/docker/roadmap/issues/109
>>> > >
>>> >
>>>
>>
Re: Multi-arch container images on DockerHub
Posted by Zach Hoffman <zr...@apache.org>.
Oops, the container link got mangled, it was supposed to be <
https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine
>.
On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman <zr...@apache.org> wrote:
> in Traffic Control, we build an Alpine image <
> https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine>,
> with Traffic Server baked into it, for amd64 and arm64 . Rather than
> building them both from the same machine, the amd64 and arm64 images in
> separate GitHub Actions jobs, then combined using `docker manifest` in a
> third GitHub Actions job <
> https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml
> >.
>
> The arm64 part still takes much longer (over 2 hours, compare to 10
> minutes for amd64), but this approach eliminates the need to build for more
> than one platform at a time, so if the arm64 job were to run on an arm64
> runner in the future, that wouldn't need to further complicate the other
> jobs in order to take advantage of that speedup.
>
> -Zach
>
> On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> In Airflow we have a bit more complex setting (we are building 2x5x2
>> different images and they are different sets of them for different
>> branches), Building images for Airflow takes quite some time (installing
>> many dependencies) so qemu was out of the question (several hours to build
>> single image). Unfortunately qemu is ~ 15 times (!) slower than hardware
>> builds from our experience. Good enough for small images, but unacceptable
>> if your image normally takes time
>>
>> The built-in actions like the one mentioned by Jacob have the limitations
>> that they depende on qemu - I have not yet found an easy "Action" that
>> could utilise AMD and ARM hardware together. But maybe there are other
>> ways.
>>
>> We developed our own scripting and tooling:
>>
>> * we have self-hosted runners in GitHub and we only use those to build
>> images (with Astronomer money and Amazon-sponsored credits on AWS)
>> * for the time when we build multi-platform build we start ARM instances
>> (they have built-in timed auto-kill-switch so that they are not run
>> indefinitely and take resources)
>> * we configured ssh-forwarded docker port from our AMD instance to forward
>> docker socket
>> * we configured two builder with buildx (local for AMD and the forwarded
>> one for ARM)
>> * we run multi-platform buildx build to use both builders.
>> * we have our own Python build/development environment (breeze) that wraps
>> the docker commands that are executed so the "code" doing it is not easy
>> to
>> copy elsewhere sometimes, but I copied below the actual "meat" what
>> happens
>> under the hood.
>> * we run it as part of our Github Action workflows
>>
>> Examples:
>>
>> * Starting ARM instance and configuring builders:
>> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
>>
>> This is the relevant part of establishing the worker for buildx (I omit
>> starting the instance as it is amazon-specific):
>>
>> export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
>> autossh -f "-L12357:/var/run/docker.sock" \
>> -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \
>> -i "${WORKING_DIR}/my_key" "${EC2_USER}@${INSTANCE_PRIVATE_DNS_NAME}"
>>
>> bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo -n
>> .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done; echo
>> " Timeout!" >&2; exit 1'
>>
>> docker buildx rm --force airflow_cache || true
>> docker buildx create --name airflow_cache
>> docker buildx create --name airflow_cache --append localhost:12357
>>
>> * Releasing image:
>> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
>>
>> This is the command that is run under-the-hood
>>
>> docker buildx build --builder airflow_cache --build-arg
>> PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
>> AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
>> apache/airflow:2.5.0-python3.10 --push
>>
>> J.
>>
>>
>>
>> On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak
>> <ja...@voltrondata.com.invalid>
>> wrote:
>>
>> > Hello Robert,
>> >
>> > I would suggest using GItHub Actions, there you can use the official
>> suite
>> > of docker actions to build multiplatform images with little need for
>> custom
>> > scripting [1].
>> > Feel free to ping me in the ASF slack.
>> >
>> > Best
>> > Jacob
>> >
>> > [1]: https://github.com/docker/build-push-action
>> >
>> > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org>
>> wrote:
>> >
>> > > Hi,
>> > >
>> > > We had a user report that our official Docker image does not support
>> > > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't
>> run
>> > > the image with our current setup.
>> > >
>> > > On our side, we set up automated builds on Docker Hub using automated
>> > > builds. Unforunately, Docker Hub autobuilds don't support `docker
>> > > buildx` or another form of multi-arch builds. It is not the roadmap
>> > > though [2], but here is no guarantee on when (or if) it will become
>> > > available.
>> > >
>> > > I see two alternatives so far:
>> > >
>> > > 1. Moving to GitHub actions
>> > > 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
>> > > Hub
>> > >
>> > > To be honest, none is too appealing to me, we have a simple process
>> > > that works.
>> > >
>> > > How are other projects handling this? Or does anyone have any ideas
>> > > that they can share?
>> > >
>> > > Thanks,
>> > > Robert
>> > >
>> > > [1]: https://issues.apache.org/jira/browse/SLING-11714
>> > > [2]: https://github.com/docker/roadmap/issues/109
>> > >
>> >
>>
>
Re: Multi-arch container images on DockerHub
Posted by Zach Hoffman <zr...@apache.org>.
in Traffic Control, we build an Alpine image <
https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine>,
with Traffic Server baked into it, for amd64 and arm64 . Rather than
building them both from the same machine, the amd64 and arm64 images in
separate GitHub Actions jobs, then combined using `docker manifest` in a
third GitHub Actions job <
https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml
>.
The arm64 part still takes much longer (over 2 hours, compare to 10 minutes
for amd64), but this approach eliminates the need to build for more than
one platform at a time, so if the arm64 job were to run on an arm64 runner
in the future, that wouldn't need to further complicate the other jobs in
order to take advantage of that speedup.
-Zach
On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> In Airflow we have a bit more complex setting (we are building 2x5x2
> different images and they are different sets of them for different
> branches), Building images for Airflow takes quite some time (installing
> many dependencies) so qemu was out of the question (several hours to build
> single image). Unfortunately qemu is ~ 15 times (!) slower than hardware
> builds from our experience. Good enough for small images, but unacceptable
> if your image normally takes time
>
> The built-in actions like the one mentioned by Jacob have the limitations
> that they depende on qemu - I have not yet found an easy "Action" that
> could utilise AMD and ARM hardware together. But maybe there are other
> ways.
>
> We developed our own scripting and tooling:
>
> * we have self-hosted runners in GitHub and we only use those to build
> images (with Astronomer money and Amazon-sponsored credits on AWS)
> * for the time when we build multi-platform build we start ARM instances
> (they have built-in timed auto-kill-switch so that they are not run
> indefinitely and take resources)
> * we configured ssh-forwarded docker port from our AMD instance to forward
> docker socket
> * we configured two builder with buildx (local for AMD and the forwarded
> one for ARM)
> * we run multi-platform buildx build to use both builders.
> * we have our own Python build/development environment (breeze) that wraps
> the docker commands that are executed so the "code" doing it is not easy to
> copy elsewhere sometimes, but I copied below the actual "meat" what happens
> under the hood.
> * we run it as part of our Github Action workflows
>
> Examples:
>
> * Starting ARM instance and configuring builders:
> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
>
> This is the relevant part of establishing the worker for buildx (I omit
> starting the instance as it is amazon-specific):
>
> export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
> autossh -f "-L12357:/var/run/docker.sock" \
> -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \
> -i "${WORKING_DIR}/my_key" "${EC2_USER}@${INSTANCE_PRIVATE_DNS_NAME}"
>
> bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo -n
> .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done; echo
> " Timeout!" >&2; exit 1'
>
> docker buildx rm --force airflow_cache || true
> docker buildx create --name airflow_cache
> docker buildx create --name airflow_cache --append localhost:12357
>
> * Releasing image:
> https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
>
> This is the command that is run under-the-hood
>
> docker buildx build --builder airflow_cache --build-arg
> PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
> AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
> apache/airflow:2.5.0-python3.10 --push
>
> J.
>
>
>
> On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak <jacob@voltrondata.com.invalid
> >
> wrote:
>
> > Hello Robert,
> >
> > I would suggest using GItHub Actions, there you can use the official
> suite
> > of docker actions to build multiplatform images with little need for
> custom
> > scripting [1].
> > Feel free to ping me in the ASF slack.
> >
> > Best
> > Jacob
> >
> > [1]: https://github.com/docker/build-push-action
> >
> > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org>
> wrote:
> >
> > > Hi,
> > >
> > > We had a user report that our official Docker image does not support
> > > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't run
> > > the image with our current setup.
> > >
> > > On our side, we set up automated builds on Docker Hub using automated
> > > builds. Unforunately, Docker Hub autobuilds don't support `docker
> > > buildx` or another form of multi-arch builds. It is not the roadmap
> > > though [2], but here is no guarantee on when (or if) it will become
> > > available.
> > >
> > > I see two alternatives so far:
> > >
> > > 1. Moving to GitHub actions
> > > 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
> > > Hub
> > >
> > > To be honest, none is too appealing to me, we have a simple process
> > > that works.
> > >
> > > How are other projects handling this? Or does anyone have any ideas
> > > that they can share?
> > >
> > > Thanks,
> > > Robert
> > >
> > > [1]: https://issues.apache.org/jira/browse/SLING-11714
> > > [2]: https://github.com/docker/roadmap/issues/109
> > >
> >
>
Re: Multi-arch container images on DockerHub
Posted by Jarek Potiuk <ja...@potiuk.com>.
In Airflow we have a bit more complex setting (we are building 2x5x2
different images and they are different sets of them for different
branches), Building images for Airflow takes quite some time (installing
many dependencies) so qemu was out of the question (several hours to build
single image). Unfortunately qemu is ~ 15 times (!) slower than hardware
builds from our experience. Good enough for small images, but unacceptable
if your image normally takes time
The built-in actions like the one mentioned by Jacob have the limitations
that they depende on qemu - I have not yet found an easy "Action" that
could utilise AMD and ARM hardware together. But maybe there are other ways.
We developed our own scripting and tooling:
* we have self-hosted runners in GitHub and we only use those to build
images (with Astronomer money and Amazon-sponsored credits on AWS)
* for the time when we build multi-platform build we start ARM instances
(they have built-in timed auto-kill-switch so that they are not run
indefinitely and take resources)
* we configured ssh-forwarded docker port from our AMD instance to forward
docker socket
* we configured two builder with buildx (local for AMD and the forwarded
one for ARM)
* we run multi-platform buildx build to use both builders.
* we have our own Python build/development environment (breeze) that wraps
the docker commands that are executed so the "code" doing it is not easy to
copy elsewhere sometimes, but I copied below the actual "meat" what happens
under the hood.
* we run it as part of our Github Action workflows
Examples:
* Starting ARM instance and configuring builders:
https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757
This is the relevant part of establishing the worker for buildx (I omit
starting the instance as it is amazon-specific):
export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log"
autossh -f "-L12357:/var/run/docker.sock" \
-N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \
-i "${WORKING_DIR}/my_key" "${EC2_USER}@${INSTANCE_PRIVATE_DNS_NAME}"
bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo -n
.; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done; echo
" Timeout!" >&2; exit 1'
docker buildx rm --force airflow_cache || true
docker buildx create --name airflow_cache
docker buildx create --name airflow_cache --append localhost:12357
* Releasing image:
https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776
This is the command that is run under-the-hood
docker buildx build --builder airflow_cache --build-arg
PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg
AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t
apache/airflow:2.5.0-python3.10 --push
J.
On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak <ja...@voltrondata.com.invalid>
wrote:
> Hello Robert,
>
> I would suggest using GItHub Actions, there you can use the official suite
> of docker actions to build multiplatform images with little need for custom
> scripting [1].
> Feel free to ping me in the ASF slack.
>
> Best
> Jacob
>
> [1]: https://github.com/docker/build-push-action
>
> On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org> wrote:
>
> > Hi,
> >
> > We had a user report that our official Docker image does not support
> > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't run
> > the image with our current setup.
> >
> > On our side, we set up automated builds on Docker Hub using automated
> > builds. Unforunately, Docker Hub autobuilds don't support `docker
> > buildx` or another form of multi-arch builds. It is not the roadmap
> > though [2], but here is no guarantee on when (or if) it will become
> > available.
> >
> > I see two alternatives so far:
> >
> > 1. Moving to GitHub actions
> > 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
> > Hub
> >
> > To be honest, none is too appealing to me, we have a simple process
> > that works.
> >
> > How are other projects handling this? Or does anyone have any ideas
> > that they can share?
> >
> > Thanks,
> > Robert
> >
> > [1]: https://issues.apache.org/jira/browse/SLING-11714
> > [2]: https://github.com/docker/roadmap/issues/109
> >
>
Re: Multi-arch container images on DockerHub
Posted by Jacob Wujciak <ja...@voltrondata.com.INVALID>.
Hello Robert,
I would suggest using GItHub Actions, there you can use the official suite
of docker actions to build multiplatform images with little need for custom
scripting [1].
Feel free to ping me in the ASF slack.
Best
Jacob
[1]: https://github.com/docker/build-push-action
On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <ro...@apache.org> wrote:
> Hi,
>
> We had a user report that our official Docker image does not support
> architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't run
> the image with our current setup.
>
> On our side, we set up automated builds on Docker Hub using automated
> builds. Unforunately, Docker Hub autobuilds don't support `docker
> buildx` or another form of multi-arch builds. It is not the roadmap
> though [2], but here is no guarantee on when (or if) it will become
> available.
>
> I see two alternatives so far:
>
> 1. Moving to GitHub actions
> 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
> Hub
>
> To be honest, none is too appealing to me, we have a simple process
> that works.
>
> How are other projects handling this? Or does anyone have any ideas
> that they can share?
>
> Thanks,
> Robert
>
> [1]: https://issues.apache.org/jira/browse/SLING-11714
> [2]: https://github.com/docker/roadmap/issues/109
>
Re: Multi-arch container images on DockerHub
Posted by Allen Wittenauer <aw...@apache.org>.
> On Dec 6, 2022, at 4:43 AM, Robert Munteanu <ro...@apache.org> wrote:
>
> I see two alternatives so far:
>
> 1. Moving to GitHub actions
Apache Yetus did the move from docker hub builds to Github Actions because ...
> 2. Use hooks to install qemu and 'fake' a multi-arch build on Docker
> Hub
... when I tried to do this a bit over a year ago, the kernel on the docker hub machines didn't support qemu. https://github.com/docker/roadmap/issues/109 seems to still be open so that functionality is likely still missing. That said, the project kept the hook in place in case it is ever supported. So...
> How are other projects handling this? Or does anyone have any ideas
> that they can share?
... the build is pretty much contained in two files:
https://github.com/apache/yetus/blob/main/.github/workflows/ghcr.yml
https://github.com/apache/yetus/blob/main/hooks/build
The build file does a lot of extra work that may/may not be desired (such as building cascading container images), but it should at least work on Linux and macOS. Being able to run it locally after a bit of multi-arch setup is a _huge_ debugging win vs. going all-in on a native GH Action method.
The hooks/build file is also still run on docker hub so as to not break users who are still pulling from there. At some point, we'll have a discussion in the project about getting rid of it but for now, everything is relatively consistent between the two container repos. Just that docker hub repo has one built for only x86 and the GHCR has both arm and x86 as a multi-arch image.
Some unsolicited advice: keep in mind the bus factor. A lot of projects have wildly complex build systems that are maintained by one maybe two people. While those build processes may be faster or better or more functional or whatever... at some point other people will need to understand it.