You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/01/26 23:36:49 UTC
[PR] Move CI documentation to inside Breeze docs [airflow]
potiuk opened a new pull request, #37039:
URL: https://github.com/apache/airflow/pull/37039
This PR moves the documentation of CI of ours to inside Breeze doc folder and splits the documentation in separate docs / chapters following similar changes done for Breeze docs #36936 and the contributing docs #36969.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move CI documentation to inside Breeze docs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37039:
URL: https://github.com/apache/airflow/pull/37039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move CI documentation to inside Breeze docs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37039:
URL: https://github.com/apache/airflow/pull/37039#discussion_r1468367132
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
+robust and stable. We are running a lot of tests for every pull request,
+for main and v2-\*-test branches and regularly as scheduled jobs.
+
+Our execution environment for CI is [GitHub
+Actions](https://github.com/features/actions). GitHub Actions (GA) are
+very well integrated with GitHub code and Workflow and it has evolved
+fast in 2019/202 to become a fully-fledged CI environment, easy to use
Review Comment:
removed it altogether and updated. It's prehistory, not relevant any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move CI documentation to inside Breeze docs [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #37039:
URL: https://github.com/apache/airflow/pull/37039#discussion_r1468361184
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
+robust and stable. We are running a lot of tests for every pull request,
Review Comment:
We run* a lot of tests
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
+robust and stable. We are running a lot of tests for every pull request,
+for main and v2-\*-test branches and regularly as scheduled jobs.
+
+Our execution environment for CI is [GitHub
+Actions](https://github.com/features/actions). GitHub Actions (GA) are
+very well integrated with GitHub code and Workflow and it has evolved
+fast in 2019/202 to become a fully-fledged CI environment, easy to use
+and develop for, so we decided to switch to it. Our previous CI system
+was Travis CI.
+
+However part of the philosophy we have is that we are not tightly
+coupled with any of the CI environments we use. Most of our CI jobs are
+written as bash scripts which are executed as steps in the CI jobs. And
+we have a number of variables determine build behaviour.
+
+
+## GitHub Actions runs
+
+Our CI builds are highly optimized, leveraging the latest features
+provided by the GitHub Actions environment to reuse parts of the build
+process across different jobs.
+
+A significant portion of our CI runs utilize container images. Given
+that Airflow has numerous dependencies, we use Docker containers to
+ensure tests run in a well-configured and consistent environment. This
+approach is used for most tests, documentation building, and some
+advanced static checks. The environment comprises two types of images:
+CI images and PROD images. CI images are used for most tests and checks,
+while PROD images are used for Kubernetes tests.
+
+To run the tests, we need to ensure that the images are built using the
+latest sources and that the build process is efficient. A full rebuild
+of such an image from scratch might take approximately 15 minutes.
+Therefore, we've implemented optimization techniques that efficiently
+use the cache from the GitHub Docker registry. In most cases, this
+reduces the time needed to rebuild the image to about 4 minutes.
+However, when dependencies change, it can take around 6-7 minutes, and
+if the base image of Python releases a new patch-level, it can take
+approximately 12 minutes.
+
+## Container Registry used as cache
+
+We are using GitHub Container Registry to store the results of the
+`Build Images` workflow which is used in the `Tests` workflow.
+
+Currently in main version of Airflow we run tests in all versions of
+Python supported, which means that we have to build multiple images (one
+CI and one PROD for each Python version). Yet we run many jobs (\>15) -
+for each of the CI images. That is a lot of time to just build the
+environment to run. Therefore we are utilising `pull_request_target`
Review Comment:
utilising the* `pull_request_target`
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
+robust and stable. We are running a lot of tests for every pull request,
+for main and v2-\*-test branches and regularly as scheduled jobs.
+
+Our execution environment for CI is [GitHub
+Actions](https://github.com/features/actions). GitHub Actions (GA) are
+very well integrated with GitHub code and Workflow and it has evolved
+fast in 2019/202 to become a fully-fledged CI environment, easy to use
Review Comment:
2019/2020* to become
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
Review Comment:
Not: is an* important component
##########
dev/breeze/doc/ci/01_ci_environment.md:
##########
@@ -0,0 +1,132 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+
+- [CI Environment](#ci-environment)
+ - [GitHub Actions runs](#github-actions-runs)
+ - [Container Registry used as cache](#container-registry-used-as-cache)
+ - [Authentication in GitHub Registry](#authentication-in-github-registry)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# CI Environment
+
+Continuous Integration is important component of making Apache Airflow
+robust and stable. We are running a lot of tests for every pull request,
+for main and v2-\*-test branches and regularly as scheduled jobs.
+
+Our execution environment for CI is [GitHub
+Actions](https://github.com/features/actions). GitHub Actions (GA) are
+very well integrated with GitHub code and Workflow and it has evolved
+fast in 2019/202 to become a fully-fledged CI environment, easy to use
+and develop for, so we decided to switch to it. Our previous CI system
+was Travis CI.
+
+However part of the philosophy we have is that we are not tightly
+coupled with any of the CI environments we use. Most of our CI jobs are
+written as bash scripts which are executed as steps in the CI jobs. And
+we have a number of variables determine build behaviour.
+
+
+## GitHub Actions runs
+
+Our CI builds are highly optimized, leveraging the latest features
+provided by the GitHub Actions environment to reuse parts of the build
+process across different jobs.
+
+A significant portion of our CI runs utilize container images. Given
+that Airflow has numerous dependencies, we use Docker containers to
+ensure tests run in a well-configured and consistent environment. This
+approach is used for most tests, documentation building, and some
+advanced static checks. The environment comprises two types of images:
+CI images and PROD images. CI images are used for most tests and checks,
+while PROD images are used for Kubernetes tests.
+
+To run the tests, we need to ensure that the images are built using the
+latest sources and that the build process is efficient. A full rebuild
+of such an image from scratch might take approximately 15 minutes.
+Therefore, we've implemented optimization techniques that efficiently
+use the cache from the GitHub Docker registry. In most cases, this
+reduces the time needed to rebuild the image to about 4 minutes.
+However, when dependencies change, it can take around 6-7 minutes, and
+if the base image of Python releases a new patch-level, it can take
+approximately 12 minutes.
+
+## Container Registry used as cache
+
+We are using GitHub Container Registry to store the results of the
+`Build Images` workflow which is used in the `Tests` workflow.
+
+Currently in main version of Airflow we run tests in all versions of
+Python supported, which means that we have to build multiple images (one
+CI and one PROD for each Python version). Yet we run many jobs (\>15) -
+for each of the CI images. That is a lot of time to just build the
+environment to run. Therefore we are utilising `pull_request_target`
+feature of GitHub Actions.
+
+This feature allows to run a separate, independent workflow, when the
Review Comment:
allows us* to run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org