You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/03 08:30:10 UTC
[GitHub] [airflow] Bowrna opened a new pull request #21956: prod image build first draft changes
Bowrna opened a new pull request #21956:
URL: https://github.com/apache/airflow/pull/21956
closes : [#21100](https://github.com/apache/airflow/issues/21100)
related : [#21100](https://github.com/apache/airflow/issues/21100)
This helps to build prod image using Breeze2
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r830478682
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import contextlib
+import sys
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE, DOCKER_CONTEXT_DIR
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ run_command(
+ [
+ 'docker',
+ 'login',
+ '--username',
+ build_params.github_username,
+ '--password-stdin',
+ 'ghcr.io',
+ ],
+ verbose=True,
+ text=True,
+ input=build_params.github_token,
Review comment:
Nice :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1073213869
> I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it
Sure - this is the key - lambda (in this case) is really an anonymous function:
This
```
read_from_cache_mock.side_effect = lambda param_name: cached_values.get(param_name)
```
Is equivalent of this (just written in a shorter way):
```
def read_from_cache_mocked_function(param_name):
return cached_values.get(param_name)
read_from_cache_mock.side_effect = read_from_cache_mocked_function
```
The "side-effect" really points to the function that will be called when the "read_from_cache_mock" funciton will be actually called. This allows to implement some logic in your test "mocks" - they become a bit smarter this way. Usually (in simple cases) your mocks can be told to return specific value:
```
read_from_cache_mock.return_value = x
```
Also you can tell it to return different values if called multiple times by assigning an iterable to return value:
```
read_from_cache_mock.return_value = [return_when_called_first_time, return_when_called_second_time, return_when_called_third_time]
```
And `side_effect` is the "smartest" way - you just provide your own (usually very simple) implementation of a function that can return different return values - for example based on the parameters that are passed.
In the case above, The "read_from_cache_mock" and "check_cache_and_write_mock" values are really very simple implementation tha cache the values in memory and replace the more "file-based" cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r830478622
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import contextlib
+import sys
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE, DOCKER_CONTEXT_DIR
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ run_command(
+ [
+ 'docker',
+ 'login',
+ '--username',
+ build_params.github_username,
+ '--password-stdin',
+ 'ghcr.io',
+ ],
+ verbose=True,
+ text=True,
+ input=build_params.github_token,
+ )
+ else:
+ console.print('\n[blue]Skip Login to GitHub Container Registry as token is missing')
+
+
+def clean_docker_context_files():
+ extensions_to_delete = ['whl', 'tar.gz']
Review comment:
I think we should just skip README.md. There might be other files in ยง`docker-context-files` that might pollute the cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066528782
> Can you give me a few example airflow version that matches this pattern?
* `2.0.1alpha1`
* `2.2.2a10`
* `2.3.0rc1`
* `2.2.0.rc3`
* `2.2.3pre0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1061812104
Looks cool :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068976592
> @potiuk Could you tell me what we are trying to do in this part? I checked the file and content would be available only in specific cases I think.
We check if there are any files in "docker-context-files" folder except the `README.md`. The idea is that if you planned to install something from there ("INSTALL_DOCKER_CONTEXT_FILES" is set) then you wanted to have something to install.
By default `docker-context-files` only contains `README.md` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1071124149
> I am getting similar issue when i run `Breeze2 build-ci-image --platform linux/amd64` too :(
Can't reproduce it :(. Just wondering - what kind of network connection you have. It almost looks like there is someone (government ? hacker?) executing a Man-in-the-middle attack on you and trying to replace the packages you are installing with something else (or blocking some URLs) . Can you try with a different network?
What do you see when you go to http://security.debian.org/debian-security/ from your browser: http://security.debian.org/debian-security
I see this:
<img width="650" alt="Screenshot 2022-03-17 at 18 26 31" src="https://user-images.githubusercontent.com/595491/158859594-2afcd059-b352-4830-a333-eddc03ea4878.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1071124149
> I am getting similar issue when i run `Breeze2 build-ci-image --platform linux/amd64` too :(
Can't reproduce it :(. Just wondering - what kind of network connection you have. It almost looks like there is someone (government ? hacker?) executing a Man-in-the-middle attack on you and trying to replace the packages you are installing with something else (or blocking some URLs) . Can you try with a different network?
What do you see when you go to http://security.debian.org/debian-security/ from your browser: http://security.debian.org/debian-security
I see this:
<img width="650" alt="Screenshot 2022-03-17 at 18 26 31" src="https://user-images.githubusercontent.com/595491/158859594-2afcd059-b352-4830-a333-eddc03ea4878.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070534325
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070920654
I am getting similar issue when i run `Breeze2 build-ci-image --platform linux/amd64` too :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #21956:
URL: https://github.com/apache/airflow/pull/21956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1071124149
> I am getting similar issue when i run `Breeze2 build-ci-image --platform linux/amd64` too :(
Can't reproduce it :(. Just wondering - what kind of network connection you have. It almost looks like there is someone (government ? hacker?) executing a Man-in-the-middle attack on you and trying to replace the packages you are installing with something else (or blocking some URLs) . Can you try with a different network.
What do you see when you go to http://security.debian.org/debian-security/ from your browser: http://security.debian.org/debian-security
I see this:
<img width="650" alt="Screenshot 2022-03-17 at 18 26 31" src="https://user-images.githubusercontent.com/595491/158859594-2afcd059-b352-4830-a333-eddc03ea4878.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r833119782
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import contextlib
+import sys
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE, DOCKER_CONTEXT_DIR
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ run_command(
+ [
+ 'docker',
+ 'login',
+ '--username',
+ build_params.github_username,
+ '--password-stdin',
+ 'ghcr.io',
+ ],
+ verbose=True,
+ text=True,
+ input=build_params.github_token,
+ )
+ else:
+ console.print('\n[blue]Skip Login to GitHub Container Registry as token is missing')
+
+
+def clean_docker_context_files():
+ extensions_to_delete = ['whl', 'tar.gz']
Review comment:
@potiuk I fixed this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1080002230
Rebasing to re-run all tests :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066528782
> Can you give me a few example airflow version that matches this pattern?
* `2.0.1alpha1`
* '2.2.2a`
* '2.3.0rc1`
* '2.2.0.rc`
* '2.2.3pre'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066548288
UPDATED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r826972705
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -312,11 +314,156 @@ def build_ci_image(
@option_verbose
@main.command(name='build-prod-image')
-def build_prod_image(verbose: bool):
+@click.option(
+ '--cleanup-docker-context-files', help='Preserves data volumes when stopping airflow.', is_flag=True
+)
+@click.option('--disable-mysql-client-installation', is_flag=True)
+@click.option('--disable-mssql-client-installation', is_flag=True)
+@click.option('--disable-postgres-client-installation', is_flag=True)
+@click.option('--disable-pip-cache', is_flag=True)
+@click.option('-t', '--install-airflow-reference')
+@click.option('-a', '--install-airflow-version', type=click.Choice(ALLOWED_INSTALL_AIRFLOW_VERSIONS))
+@click.option('-r', '--skip-rebuild-check', is_flag=True)
+@click.option('-L', '--build-cache-local', is_flag=True)
+@click.option('-U', '--build-cache-pulled', is_flag=True)
+@click.option('-X', '--build-cache-disabled', is_flag=True)
+@click.option(
+ '--additional-extras',
+ help='This installs additional extra package while installing airflow in the image.',
+)
+@option_python_version
+@click.option(
+ '--additional-dev-apt-deps', help='Additional apt dev dependencies to use when building the images.'
+)
+@click.option(
+ '--additional-runtime-apt-deps',
+ help='Additional apt runtime dependencies to use when building the images.',
+)
+@click.option(
+ '--additional-python-deps', help='Additional python dependencies to use when building the images.'
+)
+@click.option(
+ '--additional_dev_apt_command', help='Additional command executed before dev apt deps are installed.'
+)
+@click.option(
+ '--additional_runtime_apt_command',
+ help='Additional command executed before runtime apt deps are installed.',
+)
+@click.option(
+ '--additional_dev_apt_env', help='Additional environment variables set when adding dev dependencies.'
+)
+@click.option(
+ '--additional_runtime_apt_env',
+ help='Additional environment variables set when adding runtime dependencies.',
+)
+@click.option('--dev-apt-command', help='The basic command executed before dev apt deps are installed.')
+@click.option(
+ '--dev-apt-deps',
+ help='The basic apt dev dependencies to use when building the images.',
+)
+@click.option(
+ '--runtime-apt-command', help='The basic command executed before runtime apt deps are installed.'
+)
+@click.option(
+ '--runtime-apt-deps',
+ help='The basic apt runtime dependencies to use when building the images.',
+)
+@click.option('-g', '--github-repository', help='Choose repository to push/pull image.')
+@click.option('--platform', help='Builds image for the platform specified.')
+@click.option(
+ '-d',
+ '--debian-version',
+ help='Debian version used for the image.',
+ type=click.Choice(ALLOWED_DEBIAN_VERSIONS),
+)
+@click.option('--upgrade-to-newer-dependencies', is_flag=True)
Review comment:
Let's add `--prepare-buildx-cache` as additional flag here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068953380
> @potiuk Could you tell me why this is done and why there is OR operator?
>
> https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L298
```
docker_v buildx inspect airflow_cache || docker_v buildx create --name airflow_cache
```
It works in the way that it fill tries to run the left side `docker_v buildx inspect airflow_cache` and only if it returns an error, it executes the righte side: `docker_v buildx create --name airflow_cache`. This is often used shorthand for:
```
if "left side" !=0 then "right side".
```
It's based on the fact that "or" stops being evaluated when the left side is `0` because anything "or" with 0 is 0.
Similarly this:
```
docker_v buildx inspect airflow_cache && docker_v buildx create --name airflow_cache
```
would work the opposite way:
```
if "left_side" == "0" then "right side".
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r826958095
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,153 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ # echo "${token}" | docker_v login \
+ # --username "${GITHUB_USERNAME:-apache}" \
+ # --password-stdin \
+ # "ghcr.io" || true
+ run_command(
Review comment:
We should pass the token here to login from `GITHUB_TOKEN` variable (which might be actually added as a new `--github-token` option with GITHUB_TOKEN envvar). This can be done by adding a new feature to run_command to pass. Apparently this can be easily done with the `input` parameter:
https://stackoverflow.com/questions/163542/how-do-i-pass-a-string-into-subprocess-popen-using-the-stdin-argument
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,154 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ # echo "${token}" | docker_v login \
+ # --username "${GITHUB_USERNAME:-apache}" \
+ # --password-stdin \
+ # "ghcr.io" || true
+ run_command(
+ ['docker', 'login', '--username', '', '--password-stdin', 'ghcr.io'], verbose=True, text=True
+ )
+ else:
+ console.print('\n[blue]Skip Login to GitHub Container Registry as token is missing')
+
+
+def check_docker_context_files(install_from_docker_context_files: bool):
+ pass
+
+
+def build_production_image(verbose, **kwargs):
+ parameters_passed = filter_out_none(**kwargs)
+ prod_params = get_image_build_params(parameters_passed)
+ prod_params.print_info()
+ if prod_params.skip_building_prod_image:
+ console.print('[bright_yellow]\nSkip building production image. Assume the one we have is good!')
+ console.print('bright_yellow]\nYou must run Breeze2 build-prod-image before for all python versions!')
+ if prod_params.prepare_buildx_cache:
+ login_to_docker_registry(prod_params)
+
+ cmd = construct_docker_command(prod_params)
+ print(cmd)
+ output = run_command(cmd, verbose=verbose, cwd=AIRFLOW_SOURCE, text=True)
+ console.print(f"[blue]{output}")
+ # if prod_params.prepare_buildx_cache:
Review comment:
we can uncomment this one I think, once the login to registry is sorted out.
##########
File path: dev/breeze/src/airflow_breeze/breeze.py
##########
@@ -312,11 +314,156 @@ def build_ci_image(
@option_verbose
@main.command(name='build-prod-image')
-def build_prod_image(verbose: bool):
+@click.option(
Review comment:
I think it woudl be great to add ENVVARS for all the parameters here already similarly to what @edithturn added in CI build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1067900951
@potiuk Could you explain to me what's happening in this code part? If `GITHUB_USERNAME` is not specified then will it take `apache` as its value?
https://github.com/apache/airflow/blob/408a7d64f0e049279fb40048556d04b4037c78ee/scripts/ci/libraries/_build_images.sh#L336-L339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066541844
> * 2.2.3pre
@potiuk
* 2.0.1alpha1
* 2.3.0rc1
Only these patterns of version matches for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070919983
@potiuk I am getting following issue
```
sathishkannan@Sathishs-MacBook-Air airflow % Breeze2 build-prod-image --platform linux/amd64
Airflow 2.3.0.dev0 Python: 3.7. Image description: Airflow production
Working directory /Users/sathishkannan/code/airflow
docker buildx version
['docker', 'buildx', 'build', '--builder', 'default', '--progress=tty', '--build-arg', 'AIRFLOW_SOURCES_FROM=.', '--build-arg', 'AIRFLOW_SOURCES_TO=/opt/airflow', '--build-arg', 'AIRFLOW_SOURCES_WWW_FROM=./airflow/www', '--build-arg', 'AIRFLOW_SOURCES_WWW_TO=/opt/airflow/airflow/www', '--build-arg', 'AIRFLOW_INSTALLATION_METHOD=.', '--build-arg', 'AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main', '--pull', '--build-arg', 'PYTHON_BASE_IMAGE=python:3.7-slim-bullseye', '--build-arg', 'INSTALL_MYSQL_CLIENT=true', '--build-arg', 'INSTALL_MSSQL_CLIENT=true', '--build-arg', 'INSTALL_POSTGRES_CLIENT=true', '--build-arg', 'AIRFLOW_VERSION=2.3.0.dev0', '--build-arg', 'AIRFLOW_BRANCH=main', '--build-arg', 'AIRFLOW_EXTRAS=amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv', '--build-arg', 'AIRFLOW_PRE_CACHED_PIP_PACKAGES=true', '--build-arg', 'AD
DITIONAL_AIRFLOW_EXTRAS=', '--build-arg', 'ADDITIONAL_PYTHON_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_COMMAND=', '--build-arg', 'ADDITIONAL_DEV_APT_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_ENV=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_COMMAND=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_DEPS=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_ENV=', '--build-arg', 'UPGRADE_TO_NEWER_DEPENDENCIES=false', '--build-arg', 'CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow', '--build-arg', 'AIRFLOW_CONSTRAINTS=constraints-source-providers', '--build-arg', 'AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow', '--build-arg', 'AIRFLOW_IMAGE_DATE_CREATED=2022-03-17T18:50:29Z', '--build-arg', 'BUILD_ID=0', '--build-arg', 'COMMIT_SHA=eee8281a0f81c053e749c5f85931924afb1211c5', '--build-arg', 'AIRFLOW_IMAGE_README_URL=https://raw.githubusercontent.com/apache/airflow/eee8281a0f81c053e749c5f85931924afb1211c5/docs/docker-stack/README.md', '--build-arg', 'INSTALL_PROVIDERS_FROM_SOURCES=true', '--build-arg'
, 'INSTALL_FROM_PYPI=true', '--build-arg', 'INSTALL_FROM_DOCKER_CONTEXT_FILES=false', '--cache-from=ghcr.io/apache/airflow/main/prod/python3.7:cache', '-t', 'ghcr.io/apache/airflow/main/prod/python3.7', '--target', 'main', '.', '-f', 'Dockerfile', '--platform', 'linux/amd64']
[+] Building 6.7s (9/36)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 24.80kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 35B 0.0s
=> [internal] load metadata for docker.io/library/python:3.7-slim-bullseye 1.3s
=> importing cache manifest from ghcr.io/apache/airflow/main/prod/python3.7:cache 1.6s
=> [internal] load build context 0.6s
=> => transferring context: 413.77kB 0.6s
=> [airflow-build-image 1/19] FROM docker.io/library/python:3.7-slim-bullseye@sha256:6165225dd1566eccfbe60e642014bbca96ac752e09e7d3163235f62ef5ad89cc 0.0s
=> => resolve docker.io/library/python:3.7-slim-bullseye@sha256:6165225dd1566eccfbe60e642014bbca96ac752e09e7d3163235f62ef5ad89cc 0.0s
=> CACHED [airflow-build-image 2/19] COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/ 0.0s
=> ERROR [main 3/14] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 lsb-release 3.0s
=> CANCELED [airflow-build-image 3/19] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnup 3.0s
------
> [main 3/14] RUN apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 lsb-release && export && source /scripts/docker/determine_debian_version_specific_variables.sh && bash -o pipefail -o errexit -o nounset -o nolog -c "echo" && bash -o pipefail -o errexit -o nounset -o nolog -c "" && apt-get update && apt-get install -y --no-install-recommends apt-transport-https apt-utils ca-certificates curl dumb-init freetds-bin gosu krb5-user ldap-utils libldap-2.4-2 libsasl2-2 libsasl2-modules libssl1.1 locales lsb-release netcat openssh-client rsync sasl2-bin sqlite3 sudo unixodbc "${DISTRO_LIBFFI}" && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/
lib/apt/lists/* && rm -rf /var/log/*:
#8 0.630 Get:1 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
#8 0.636 Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB]
#8 0.778 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [39.3 kB]
#8 1.435 Err:1 http://security.debian.org/debian-security bullseye-security InRelease
#8 1.435 At least one invalid signature was encountered.
#8 2.121 Err:2 http://deb.debian.org/debian bullseye InRelease
#8 2.121 At least one invalid signature was encountered.
#8 2.800 Err:3 http://deb.debian.org/debian bullseye-updates InRelease
#8 2.800 At least one invalid signature was encountered.
#8 2.845 Reading package lists...
#8 2.951 W: GPG error: http://security.debian.org/debian-security bullseye-security InRelease: At least one invalid signature was encountered.
#8 2.951 E: The repository 'http://security.debian.org/debian-security bullseye-security InRelease' is not signed.
#8 2.951 W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered.
#8 2.952 E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed.
#8 2.952 W: GPG error: http://deb.debian.org/debian bullseye-updates InRelease: At least one invalid signature was encountered.
#8 2.952 E: The repository 'http://deb.debian.org/debian bullseye-updates InRelease' is not signed.
------
error: failed to solve: executor failed running [/bin/bash -o pipefail -o errexit -o nounset -o nolog -c apt-get update && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 && apt-get install -y --no-install-recommends curl gnupg2 lsb-release && export ${ADDITIONAL_RUNTIME_APT_ENV?} && source /scripts/docker/determine_debian_version_specific_variables.sh && bash -o pipefail -o errexit -o nounset -o nolog -c "${RUNTIME_APT_COMMAND}" && bash -o pipefail -o errexit -o nounset -o nolog -c "${ADDITIONAL_RUNTIME_APT_COMMAND}" && apt-get update && apt-get install -y --no-install-recommends ${RUNTIME_APT_DEPS} "${DISTRO_LIBFFI}" ${ADDITIONAL_RUNTIME_APT_DEPS} && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* && rm -rf /var/log/*]: exit code: 100
========================= OUTPUT start ============================
None
None
========================= OUTPUT end ============================
Traceback (most recent call last):
File "/Users/sathishkannan/.pyenv/versions/airflow-env/bin/Breeze2", line 33, in <module>
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts', 'Breeze2')())
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/rich_click/rich_click.py", line 512, in main
return super().main(*args, standalone_mode=False, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/envs/airflow-env/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/breeze.py", line 426, in build_prod_image
build_production_image(
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/prod/build_prod_image.py", line 176, in build_production_image
output = run_command(cmd, verbose=verbose, cwd=AIRFLOW_SOURCE, text=True)
File "/Users/sathishkannan/code/airflow/dev/breeze/src/airflow_breeze/utils/run_utils.py", line 65, in run_command
return subprocess.run(cmd, check=check, env=cmd_env, cwd=workdir, **kwargs)
File "/Users/sathishkannan/.pyenv/versions/3.8.10/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'buildx', 'build', '--builder', 'default', '--progress=tty', '--build-arg', 'AIRFLOW_SOURCES_FROM=.', '--build-arg', 'AIRFLOW_SOURCES_TO=/opt/airflow', '--build-arg', 'AIRFLOW_SOURCES_WWW_FROM=./airflow/www', '--build-arg', 'AIRFLOW_SOURCES_WWW_TO=/opt/airflow/airflow/www', '--build-arg', 'AIRFLOW_INSTALLATION_METHOD=.', '--build-arg', 'AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main', '--pull', '--build-arg', 'PYTHON_BASE_IMAGE=python:3.7-slim-bullseye', '--build-arg', 'INSTALL_MYSQL_CLIENT=true', '--build-arg', 'INSTALL_MSSQL_CLIENT=true', '--build-arg', 'INSTALL_POSTGRES_CLIENT=true', '--build-arg', 'AIRFLOW_VERSION=2.3.0.dev0', '--build-arg', 'AIRFLOW_BRANCH=main', '--build-arg', 'AIRFLOW_EXTRAS=amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv', '--build-arg', 'AIRFLOW_PRE_CACHE
D_PIP_PACKAGES=true', '--build-arg', 'ADDITIONAL_AIRFLOW_EXTRAS=', '--build-arg', 'ADDITIONAL_PYTHON_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_COMMAND=', '--build-arg', 'ADDITIONAL_DEV_APT_DEPS=', '--build-arg', 'ADDITIONAL_DEV_APT_ENV=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_COMMAND=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_DEPS=', '--build-arg', 'ADDITIONAL_RUNTIME_APT_ENV=', '--build-arg', 'UPGRADE_TO_NEWER_DEPENDENCIES=false', '--build-arg', 'CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow', '--build-arg', 'AIRFLOW_CONSTRAINTS=constraints-source-providers', '--build-arg', 'AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow', '--build-arg', 'AIRFLOW_IMAGE_DATE_CREATED=2022-03-17T18:50:29Z', '--build-arg', 'BUILD_ID=0', '--build-arg', 'COMMIT_SHA=eee8281a0f81c053e749c5f85931924afb1211c5', '--build-arg', 'AIRFLOW_IMAGE_README_URL=https://raw.githubusercontent.com/apache/airflow/eee8281a0f81c053e749c5f85931924afb1211c5/docs/docker-stack/README.md', '--build-arg', 'INSTALL_PRO
VIDERS_FROM_SOURCES=true', '--build-arg', 'INSTALL_FROM_PYPI=true', '--build-arg', 'INSTALL_FROM_DOCKER_CONTEXT_FILES=false', '--cache-from=ghcr.io/apache/airflow/main/prod/python3.7:cache', '-t', 'ghcr.io/apache/airflow/main/prod/python3.7', '--target', 'main', '.', '-f', 'Dockerfile', '--platform', 'linux/amd64']' returned non-zero exit status 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070534736
> @potiuk Could you review the changes made when you get the time? I tried running the command in both old Breeze and Breeze2 . Both of them fail when building the production image docker process. Could you verify and tell me where I am going wrong?
Hmm. It works for me :). what kind of erors you have? Generally speaking - this one looks cool and I am ready to approve it. some old tests for ci build are failing after the change and needs fixing and maybe you should add similar tests for prod image and I think we should be ready to merge that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070934839
I am working on adding tests @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068953380
> @potiuk Could you tell me why this is done and why there is OR operator?
>
> https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L298
It literally means: "run 'create' if inspect' returns error".
```
docker_v buildx inspect airflow_cache || docker_v buildx create --name airflow_cache
```
It works in the way that it fill tries to run the left side `docker_v buildx inspect airflow_cache` and only if it returns an error, it executes the right side: `docker_v buildx create --name airflow_cache`. This is often used shorthand for:
```
if "left side" !=0 then "right side".
```
It's based on the fact that "or" stops being evaluated when the left side is `0` because anything "or" with 0 is 0.
Similarly this:
```
docker_v buildx inspect airflow_cache && docker_v buildx create --name airflow_cache
```
would work the opposite way:
```
if "left_side" == "0" then "right side".
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1080995704
@potiuk Fixing the Breeze2 tests failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068890069
@potiuk Could you tell me why this is done and why there is OR operator?
https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1071124149
> I am getting similar issue when i run `Breeze2 build-ci-image --platform linux/amd64` too :(
Can't reproduce it :(. Just wondering - what kind of network connection you have. It almost looks like there is someone (government ? hacker?) executing a Man-in-the-middle attack on you and trying to replace the packages you are installing with something else (or blocking some URLs) . Can you try with a different network.
What do you see when you go to http://security.debian.org/debian-security/ from your browser: http://security.debian.org/debian-security
I see this:
<img width="650" alt="Screenshot 2022-03-17 at 18 26 31" src="https://user-images.githubusercontent.com/595491/158859594-2afcd059-b352-4830-a333-eddc03ea4878.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1082019640
Wooho! :tada:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066528782
> Can you give me a few example airflow version that matches this pattern?
* `2.0.1alpha1`
* `2.2.2a`
* `2.3.0rc1`
* `2.2.0.rc`
* `2.2.3pre`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066517733
@potiuk
https://github.com/apache/airflow/blob/408a7d64f0e049279fb40048556d04b4037c78ee/scripts/ci/libraries/_build_images.sh#L542
Can you give me a few example airflow version that matches this pattern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1073011967
@potiuk Could you explain to me how this side effect is put to use?
https://github.com/apache/airflow/blob/5eb63357426598f99ed50b002b72aebdf8790f73/dev/breeze/tests/test_build_image.py#L47-L52
I could understand that you are mocking the return of those functions but I couldn't understand how it's further used in the code. If possible could you tell me or direct me to some relevant place to learn about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1071989707
I see this when I check the URL http://security.debian.org/debian-security/
<img width="526" alt="Screenshot 2022-03-18 at 8 49 59 AM" src="https://user-images.githubusercontent.com/10162465/158931342-e7ec2d1f-4393-4b4e-974e-29e062e2cab6.png">
Let me try with different network and update you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068972948
@potiuk Could you tell me what we are trying to do in this part? I checked the file and content would be available only in specific cases I think.
https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L224
I am not sure how to verify this whole function
https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L221-L244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1076226070
@potiuk I have tried to add test for prod image similar to what you have done for CI image. Do you think I can further improve with more tests into prod image? Could you give me suggestion on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1068953380
> @potiuk Could you tell me why this is done and why there is OR operator?
>
> https://github.com/apache/airflow/blob/df6058c862a910a99fbb86858502d9d93fdbe1e5/scripts/ci/libraries/_build_images.sh#L298
It literally means: "run 'create' if 'inspect' returns error".
```
docker_v buildx inspect airflow_cache || docker_v buildx create --name airflow_cache
```
It works in the way that it fill tries to run the left side `docker_v buildx inspect airflow_cache` and only if it returns an error, it executes the right side: `docker_v buildx create --name airflow_cache`. This is often used shorthand for:
```
if "left side" !=0 then "right side".
```
It's based on the fact that "or" stops being evaluated when the left side is `0` because anything "or" with 0 is 0.
Similarly this:
```
docker_v buildx inspect airflow_cache && docker_v buildx create --name airflow_cache
```
would work the opposite way:
```
if "left_side" == "0" then "right side".
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21956: prod image build first draft changes
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1066547944
Ah. right - we assume we always have number :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070513944
@potiuk Could you review the changes made when you get the time? I tried running the command in both old Breeze and Breeze2 . Both of them fail when building the production image docker process. Could you verify and tell me where I am going wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1070919983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a change in pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on a change in pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#discussion_r830554200
##########
File path: dev/breeze/src/airflow_breeze/prod/build_prod_image.py
##########
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import contextlib
+import sys
+from typing import Dict, List
+
+from airflow_breeze.cache import check_cache_and_write_if_not_cached, write_to_cache_file
+from airflow_breeze.console import console
+from airflow_breeze.prod.prod_params import ProdParams
+from airflow_breeze.utils.path_utils import AIRFLOW_SOURCE, DOCKER_CONTEXT_DIR
+from airflow_breeze.utils.run_utils import filter_out_none, run_command
+
+PARAMS_PROD_IMAGE = [
+ "python_base_image",
+ "install_mysql_client",
+ "install_mssql_client",
+ "install_postgres_client",
+ "airflow_version",
+ "airflow_branch",
+ "airflow_extras",
+ "airflow_pre_cached_pip_packages",
+ "additional_airflow_extras",
+ "additional_python_deps",
+ "additional_dev_apt_command",
+ "additional_dev_apt_deps",
+ "additional_dev_apt_env",
+ "additional_runtime_apt_command",
+ "additional_runtime_apt_deps",
+ "additional_runtime_apt_env",
+ "upgrade_to_newer_dependencies",
+ "constraints_github_repository",
+ "airflow_constraints",
+ "airflow_image_repository",
+ "airflow_image_date_created",
+ "build_id",
+ "commit_sha",
+ "airflow_image_readme_url",
+ "install_providers_from_sources",
+ "install_from_pypi",
+ "install_from_docker_context_files",
+]
+
+PARAMS_TO_VERIFY_PROD_IMAGE = [
+ "dev_apt_command",
+ "dev_apt_deps",
+ "runtime_apt_command",
+ "runtime_apt_deps",
+]
+
+
+def construct_arguments_docker_command(prod_image: ProdParams) -> List[str]:
+ args_command = []
+ for param in PARAMS_PROD_IMAGE:
+ args_command.append("--build-arg")
+ args_command.append(param.upper() + "=" + str(getattr(prod_image, param)))
+ for verify_param in PARAMS_TO_VERIFY_PROD_IMAGE:
+ param_value = str(getattr(prod_image, verify_param))
+ if len(param_value) > 0:
+ args_command.append("--build-arg")
+ args_command.append(verify_param.upper() + "=" + param_value)
+ docker_cache = prod_image.docker_cache_prod_directive
+ if len(docker_cache) > 0:
+ args_command.extend(prod_image.docker_cache_prod_directive)
+ return args_command
+
+
+def construct_docker_command(prod_image: ProdParams) -> List[str]:
+ arguments = construct_arguments_docker_command(prod_image)
+ build_command = prod_image.check_buildx_plugin_build_command()
+ build_flags = prod_image.extra_docker_build_flags
+ final_command = []
+ final_command.extend(["docker"])
+ final_command.extend(build_command)
+ final_command.extend(build_flags)
+ final_command.extend(["--pull"])
+ final_command.extend(arguments)
+ final_command.extend(["-t", prod_image.airflow_prod_image_name, "--target", "main", "."])
+ final_command.extend(["-f", 'Dockerfile'])
+ final_command.extend(["--platform", prod_image.platform])
+ return final_command
+
+
+def login_to_docker_registry(build_params: ProdParams):
+ if build_params.ci == "true":
+ if len(build_params.github_token) == 0:
+ console.print("\n[blue]Skip logging in to GitHub Registry. No Token available!")
+ elif build_params.airflow_login_to_github_registry != "true":
+ console.print(
+ "\n[blue]Skip logging in to GitHub Registry.\
+ AIRFLOW_LOGIN_TO_GITHUB_REGISTRY is set as false"
+ )
+ elif len(build_params.github_token) > 0:
+ run_command(['docker', 'logout', 'ghcr.io'], verbose=True, text=True)
+ run_command(
+ [
+ 'docker',
+ 'login',
+ '--username',
+ build_params.github_username,
+ '--password-stdin',
+ 'ghcr.io',
+ ],
+ verbose=True,
+ text=True,
+ input=build_params.github_token,
+ )
+ else:
+ console.print('\n[blue]Skip Login to GitHub Container Registry as token is missing')
+
+
+def clean_docker_context_files():
+ extensions_to_delete = ['whl', 'tar.gz']
Review comment:
ok @potiuk I will these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21956: prod image build changes
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #21956:
URL: https://github.com/apache/airflow/pull/21956#issuecomment-1073235230
I see @potiuk thank you. I could better understand the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org