You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/14 21:43:08 UTC
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24433: First attempt to have CI-controlled process of releasing PROD image
jedcunningham commented on code in PR #24433:
URL: https://github.com/apache/airflow/pull/24433#discussion_r897340795
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
+[this guideline](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
Review Comment:
```suggestion
[this blog post](https://www.docker.com/blog/speed-up-building-with-docker-buildx-and-graviton2-ec2/) and
```
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
Review Comment:
```suggestion
remote).
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -494,76 +491,23 @@ is not supposed to be used by and advertised to the end-users who do not read th
Production Docker images should be manually prepared and pushed by the release manager or another committer
who has access to Airflow's DockerHub. Note that we started releasing a multi-platform build, so you need
-to have an environment prepared to build multi-platform images. You can achieve it with either emulation
-(very slow) or if you have two types of hardware (AMD64 and ARM64) you can configure Hardware builders.
+to have an environment prepared to build multi-platform images. You can achieve it with:
-## Prerequisites
+* GitHub Actions Manual Job (easiest)
+* Emulation (very slow)
+* Hardware builders if you have both AMD64 and ARM64 hardware locally
-You need to have buildx plugin installed to run the build. Also, you need to have regctl
-installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
-DockerHub. The script to build images will refuse to work if you do not have those two installed.
+Building the image is triggered By running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions)
-You also need to have the right permissions to push the images, so you should run
-`docker login` before and authenticate with your DockerHub token.
+When you trigger it you need to pass:
-## Setting environment with emulation
+* Airflow Version
+* Additional "true" in skip field if you do not release latest image
Review Comment:
```suggestion
* Optional "true" in skip latest field if you do not want to retag the latest image
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -494,76 +491,23 @@ is not supposed to be used by and advertised to the end-users who do not read th
Production Docker images should be manually prepared and pushed by the release manager or another committer
who has access to Airflow's DockerHub. Note that we started releasing a multi-platform build, so you need
-to have an environment prepared to build multi-platform images. You can achieve it with either emulation
-(very slow) or if you have two types of hardware (AMD64 and ARM64) you can configure Hardware builders.
+to have an environment prepared to build multi-platform images. You can achieve it with:
-## Prerequisites
+* GitHub Actions Manual Job (easiest)
+* Emulation (very slow)
+* Hardware builders if you have both AMD64 and ARM64 hardware locally
-You need to have buildx plugin installed to run the build. Also, you need to have regctl
-installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
-DockerHub. The script to build images will refuse to work if you do not have those two installed.
+Building the image is triggered By running the `Release PROD image` workflow via
Review Comment:
```suggestion
Building the image is triggered by running the `Release PROD image` workflow via
```
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
Review Comment:
```suggestion
If you plan to build a number of images, it's probably better to set up a hardware remote builder
```
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
Review Comment:
```suggestion
More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/).
```
##########
.github/workflows/release_dockerhub_image.yml:
##########
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Release PROD image"
+on: # yamllint disable-line rule:truthy
+ workflow_dispatch:
+ inputs:
+ airflowVersion:
+ description: 'Airflow version'
+ required: true
+ skipLatest:
+ description: 'Set below to true if not latest.'
Review Comment:
```suggestion
description: 'Skip Latest: Set to true if not latest.'
```
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
Review Comment:
```suggestion
In order to build images on local hardware, you need to have the buildx plugin installed to run the build.
```
##########
dev/MANUALLY_BUILDING_IMAGES.md:
##########
@@ -0,0 +1,96 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Building docker images](#building-docker-images)
+- [Setting environment with emulation](#setting-environment-with-emulation)
+- [Setting up cache refreshing with hardware ARM/AMD support](#setting-up-cache-refreshing-with-hardware-armamd-support)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Building docker images
+
+In order to build images on local hardware,you need to have buildx plugin installed to run the build.
+Also, you need to have regctl installed from https://github.com/regclient/regclient in order to tag
+the multi-platform images in DockerHub. The script to build images will refuse to work if
+you do not have those two installed.
+
+You also need to have the right permissions to push the images, so you should run
+`docker login` before and authenticate with your DockerHub token.
+
+## Setting environment with emulation
+
+According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
+this can be achieved via:
+
+```shell
+docker run --privileged --rm tonistiigi/binfmt --install all
+```
+
+More information can be found [here](https://docs.docker.com/engine/reference/commandline/buildx_create/)
+
+However, emulation is very slow - more than 10x slower than hardware-backed builds.
+
+## Setting up cache refreshing with hardware ARM/AMD support
+
+If you plan to build a number of images, probably better solution is to set up a hardware remote builder
+for your ARM or AMD builds (depending which platform you build images on - the "other" platform should be
+remote.
+
+This can be achieved by settings build as described in
Review Comment:
```suggestion
This can be achieved by settings build as described in
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -494,76 +491,23 @@ is not supposed to be used by and advertised to the end-users who do not read th
Production Docker images should be manually prepared and pushed by the release manager or another committer
who has access to Airflow's DockerHub. Note that we started releasing a multi-platform build, so you need
-to have an environment prepared to build multi-platform images. You can achieve it with either emulation
-(very slow) or if you have two types of hardware (AMD64 and ARM64) you can configure Hardware builders.
+to have an environment prepared to build multi-platform images. You can achieve it with:
-## Prerequisites
+* GitHub Actions Manual Job (easiest)
+* Emulation (very slow)
+* Hardware builders if you have both AMD64 and ARM64 hardware locally
-You need to have buildx plugin installed to run the build. Also, you need to have regctl
-installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
-DockerHub. The script to build images will refuse to work if you do not have those two installed.
+Building the image is triggered By running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions)
Review Comment:
```suggestion
[GitHub Actions](https://github.com/apache/airflow/actions).
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -1014,33 +958,22 @@ At this point we release an official package:
## Manually prepare production Docker Image
-Note that this scripts prepares multi-platform image, so you need to fulfill prerequisites as
-described above in the preparation of RC images.
+Building the image is triggered By running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions)
+
+When you trigger it you need to pass:
+
+* Airflow Version
+* Additional "true" in skip field if you do not release latest image
Review Comment:
```suggestion
* Optional "true" in skip latest field if you do not want to retag the latest image
```
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -494,76 +491,23 @@ is not supposed to be used by and advertised to the end-users who do not read th
Production Docker images should be manually prepared and pushed by the release manager or another committer
who has access to Airflow's DockerHub. Note that we started releasing a multi-platform build, so you need
-to have an environment prepared to build multi-platform images. You can achieve it with either emulation
-(very slow) or if you have two types of hardware (AMD64 and ARM64) you can configure Hardware builders.
+to have an environment prepared to build multi-platform images. You can achieve it with:
-## Prerequisites
+* GitHub Actions Manual Job (easiest)
+* Emulation (very slow)
+* Hardware builders if you have both AMD64 and ARM64 hardware locally
-You need to have buildx plugin installed to run the build. Also, you need to have regctl
-installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
-DockerHub. The script to build images will refuse to work if you do not have those two installed.
+Building the image is triggered By running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions)
-You also need to have the right permissions to push the images, so you should run
-`docker login` before and authenticate with your DockerHub token.
+When you trigger it you need to pass:
-## Setting environment with emulation
+* Airflow Version
+* Additional "true" in skip field if you do not release latest image
Review Comment:
Since we are talking RC images here, should we change this to advise setting this to true?
##########
dev/README_RELEASE_AIRFLOW.md:
##########
@@ -1014,33 +958,22 @@ At this point we release an official package:
## Manually prepare production Docker Image
-Note that this scripts prepares multi-platform image, so you need to fulfill prerequisites as
-described above in the preparation of RC images.
+Building the image is triggered By running the `Release PROD image` workflow via
+[GitHub Actions](https://github.com/apache/airflow/actions)
Review Comment:
```suggestion
Building the image is triggered by running the `Release PROD image` workflow via
[GitHub Actions](https://github.com/apache/airflow/actions).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org