You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/11/14 16:34:16 UTC
[airflow] branch v1-10-test updated (0b8bb3e -> 00f320f)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
omit 0b8bb3e Synchronize INTHEWILD.md with master
omit 09fe8f1 Simplifies check whether the CI image should be rebuilt (#12181)
omit a950aac For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
omit 2f59506 Deploy was not working from Breeze (#12319)
omit ca91871 Python base image is shared between CI and PROD image (#12280)
omit 330e4d3 Added k9s as integrated tool to help with kubernetes testing (#12163)
omit f1b8c7e Docker context files should be available earlier (#12219)
omit b40568e Fixes continuous image rebuilding with Breeze (#12256)
omit 07298ba Add Markdown linting to pre-commit (#11465)
omit c89257c Beautify Output of setup-installation pre-commit (#12218)
omit 7d40471 Fix permissions of mounted /tmp directory for Breeze (#12157)
omit 7eac201 Remove popd which is a remnant from past (#12211)
omit 4f8473d Fixes timeout in helm chart tests (#12209)
omit 330f0b9 Fixed path of the test_core.py file in docs (#12191)
omit 6370a91 Adds extra check while the selective checks are run (#12178)
omit c7d64d5 Uses always the same Python base image as used for CI image (#12177)
omit ae033b9 Fixes "--force-clean-images" flag in Breeze (#12156)
omit 67f2c97 Fixes undefined variables (#12155)
omit 4166311 Update to new helm stable repo (#12137)
omit 61346ca Work properly if some variables are not defined (#12135)
omit 5914d12 Add Kubernetes files to selective checks (#12114)
omit 4c8f860 Update install_mysql.sh (#12101)
omit 96f0a77 Fix proper SHA in check preventing accidentally merging PR (#12083)
omit 6b1078b Use sys.exit() instead of exit() (#12084)
omit b1b608e Fixes problem with building a PROD image (#12080)
omit cdc26fc If we build a new image, we should run more than basic checks (#12070)
omit ea58242 Adds a forgotten word in a README.md (#12066)
omit 4e0b9fe Uses DOCKER_TAG when building image in DockerHub (#12050)
omit 5ab6cd5 Fixes documentation-only selective checks (#12038)
omit 56727d0 Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
omit 46d778a Turns failure of PR label when approved action into warning (#12017)
omit 56d5960 Fix canceling of CodeQL workflow (#12024)
omit d10eece Fixes problem with non-iterable data returned by GH API (#12021)
omit 86c7b7b Revise "Project Focus" copy (#12011)
omit 9051a87 Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
omit 08c0275 Add contributor-targeted description of the PR workflow. (#12016)
omit a8d602b Adds documentation about the optimized PR workflow (#12006)
omit 310cbd6 Fix K8S CI job name rendering (#12007)
omit c45c045 Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
omit 6454b94 Implements canceling of future duplicate runs (but the latest) (#11980)
omit 4acdc8e Move Project focus and Principles higher in the README (#11973)
omit b310082 Pin `kubernetes` to a max version of 11.0.0. (#11974)
omit eb11556 Updated documentation for the CI with mermaid sequence diagrams (#10380)
omit a13366d Migrate from helm-unittest to python unittest (#11827)
new 572387d Migrate from helm-unittest to python unittest (#11827)
new 132e899 Updated documentation for the CI with mermaid sequence diagrams (#10380)
new d5e7b51 Pin `kubernetes` to a max version of 11.0.0. (#11974)
new 6f6b560 Move Project focus and Principles higher in the README (#11973)
new ad56790 Implements canceling of future duplicate runs (but the latest) (#11980)
new d3ab3df Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
new 33a14e7 Fix K8S CI job name rendering (#12007)
new 8084584 Adds documentation about the optimized PR workflow (#12006)
new b2633ff Add contributor-targeted description of the PR workflow. (#12016)
new 99aa883 Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
new 887c231 Revise "Project Focus" copy (#12011)
new e18d675 Fixes problem with non-iterable data returned by GH API (#12021)
new df68fb7 Fix canceling of CodeQL workflow (#12024)
new 8e58291 Turns failure of PR label when approved action into warning (#12017)
new 43c72f2 Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
new ad8aa65 Fixes documentation-only selective checks (#12038)
new 7512344 Uses DOCKER_TAG when building image in DockerHub (#12050)
new 26d001f Adds a forgotten word in a README.md (#12066)
new 6f64e98 If we build a new image, we should run more than basic checks (#12070)
new f432309 Fixes problem with building a PROD image (#12080)
new 2be4d56 Use sys.exit() instead of exit() (#12084)
new 64393ad Fix proper SHA in check preventing accidentally merging PR (#12083)
new b062b92 Update install_mysql.sh (#12101)
new 49430a0 Add Kubernetes files to selective checks (#12114)
new 38d5582 Work properly if some variables are not defined (#12135)
new 2ddafd0 Update to new helm stable repo (#12137)
new fc19025 Fixes undefined variables (#12155)
new 815f18b Fixes "--force-clean-images" flag in Breeze (#12156)
new a27701a Uses always the same Python base image as used for CI image (#12177)
new 20512af Adds extra check while the selective checks are run (#12178)
new 10700af Fixed path of the test_core.py file in docs (#12191)
new 6e74f1b Fixes timeout in helm chart tests (#12209)
new 7318701 Remove popd which is a remnant from past (#12211)
new 40ebe6e Fix permissions of mounted /tmp directory for Breeze (#12157)
new eb7d914 Beautify Output of setup-installation pre-commit (#12218)
new 64af945 Add Markdown linting to pre-commit (#11465)
new 4ca85a6 Fixes continuous image rebuilding with Breeze (#12256)
new 10b6c48 Docker context files should be available earlier (#12219)
new d8db36a Added k9s as integrated tool to help with kubernetes testing (#12163)
new 76f51e1 Python base image is shared between CI and PROD image (#12280)
new 891a01d Deploy was not working from Breeze (#12319)
new 4ab76df For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
new 23c6206 Simplifies check whether the CI image should be rebuilt (#12181)
new 00f320f Synchronize INTHEWILD.md with master
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (0b8bb3e)
\
N -- N -- N refs/heads/v1-10-test (00f320f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 44 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.pre-commit-config.yaml | 2 +-
airflow/www_rbac/webpack.config.js | 2 +-
docs/static/exampleinclude.css | 2 +-
docs/static/jira-links.js | 2 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 5 ++---
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 5 +++++
6 files changed, 11 insertions(+), 7 deletions(-)
[airflow] 12/44: Fixes problem with non-iterable data returned by
GH API (#12021)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e18d6752814ea17cf596048cb8196224c46586a7
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 1 19:25:29 2020 +0100
Fixes problem with non-iterable data returned by GH API (#12021)
The action to cancel workflow switched from deprecated
method of retrieving jobs to a 'better' one but it caused
some unexpected failures as some of the job data is not iterable and failures in 'failedJobs" matching
Version 4.6 fixed the problem.
(cherry picked from commit aa5213b3609388b7b77e362dd9ad0bb05a8b5bc6)
---
.github/workflows/build-images-workflow-run.yml | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index a3880e7..752e577 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event + Repo + Branch as the
# current Event/Repo/Branch combination.
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that we
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.cancel-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then we try to match
# all the jobs with the same Event + Repo + Branch match and cancel all the duplicates for those
# This might cancel own run, so this is the last step in the job
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 09/44: Add contributor-targeted description of the PR
workflow. (#12016)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b2633ff5231198363fcf6ac64ed388bc4125a12f
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 1 14:22:23 2020 +0100
Add contributor-targeted description of the PR workflow. (#12016)
* Add contributor-targeted description of the PR workflow.
Simpler/shorter description of the PR workflow targeted for new
contributors and contributors who did not follow the recent changes
in the PR workflow.
* Update CONTRIBUTING.rst
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
* Update CONTRIBUTING.rst
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
Co-authored-by: Xiaodong DENG <xd...@gmail.com>
(cherry picked from commit 6c8c3089e9f7fc00debe031b8c1f0000f576470a)
---
CONTRIBUTING.rst | 552 ++++++++++++++++++++++++++++---------------------------
1 file changed, 285 insertions(+), 267 deletions(-)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 40e4f89..6d3aa91 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -42,8 +42,7 @@ to follow it and apply to the programme and follow up with the community.
Report Bugs
-----------
-Report bugs through `Apache
-JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__.
+Report bugs through `GitHub <https://github.com/apache/airflow/issues>`__.
Please report relevant information and preferably code that exhibits the
problem.
@@ -51,16 +50,16 @@ problem.
Fix Bugs
--------
-Look through the JIRA issues for bugs. Anything is open to whoever wants to
+Look through the GitHub issues for bugs. Anything is open to whoever wants to
implement it.
Implement Features
------------------
-Look through the `Apache
-JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__ for features.
+Look through the `GitHub issues labeled "kind:feature"
+<https://github.com/apache/airflow/labels/kind%3Afeature>`__ for features.
-Any unassigned "Improvement" issue is open to whoever wants to implement it.
+Any unassigned feature request issue is open to whoever wants to implement it.
We've created the operators, hooks, macros and executors we needed, but we've
made sure that this part of Airflow is extensible. New operators, hooks, macros
@@ -76,8 +75,7 @@ articles.
Submit Feedback
---------------
-The best way to send feedback is to open an issue on `Apache
-JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__.
+The best way to send feedback is to `open an issue on GitHub <https://github.com/apache/airflow/issues/new/choose>`__.
If you are proposing a new feature:
@@ -150,69 +148,6 @@ Contributors are responsible for:
* Adding features
* Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__.
-=============
-
-There are several roles within the Airflow Open-Source community.
-
-
-PMC Member
------------
-The PMC (Project Management Committee) is a group of maintainers that drives changes in the way that
-Airflow is managed as a project.
-
-Considering Apache, the role of the PMC is primarily to ensure that Airflow conforms to Apache's processes
-and guidelines.
-
-Committers/Maintainers
-----------------------
-
-Committers are community members that have write access to the project’s repositories, i.e., they can modify the code,
-documentation, and website by themselves and also accept other contributions.
-
-The official list of committers can be found `here <https://airflow.apache.org/docs/stable/project.html#committers>`__.
-
-Additionally, committers are listed in a few other places (some of these may only be visible to existing committers):
-
-* https://whimsy.apache.org/roster/ppmc/airflow
-* https://github.com/orgs/apache/teams/airflow-committers/members
-
-Committers are responsible for:
-
-* Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__
-* Reviewing & Merging Pull-Requests
-* Scanning and responding to Github issues
-* Responding to questions on the dev mailing list (dev@airflow.apache.org)
-
-Becoming a Committer
---------------------
-
-There is no strict protocol for becoming a committer.
-Candidates for new committers are typically people that are active contributors and community members.
-
-The key aspects of a committer are:
-
-* Consistent contributions over the past 6 months
-* Understanding of Airflow Core or has displayed a holistic understanding of a particular part and made
- contributions towards a more strategic goal
-* Understanding of contributor/committer guidelines: `Contributors' Guide <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst>`__
-* Quality of the commits
-* Visibility in community discussions (dev mailing list, Slack and Github)
-* Testing Release Candidates
-
-
-Contributors
-------------
-
-A contributor is anyone who wants to contribute code, documentation, tests, ideas, or anything to the
-Apache Airflow project.
-
-Contributors are responsible for:
-
-* Fixing bugs
-* Adding features
-* Championing one or more items on the `Roadmap <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`__.
-
-
Contribution Workflow
=====================
@@ -240,7 +175,7 @@ In general, your contribution includes the following stages:
If you want to add more changes in the future, set up your fork and enable GitHub Actions.
3. Join `devlist <ht...@airflow.apache.org>`__
- and set up a `Slack account <https://apache-airflow-slack.herokuapp.com>`__.
+ and set up a `Slack account <https://s.apache.org/airflow-slack>`__.
4. Make the change and create a `Pull Request from your fork <https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork>`__.
@@ -262,21 +197,24 @@ Configure the Docker-based Breeze development environment and run tests.
You can use the default Breeze configuration as follows:
-The latest API documentation is usually available
-`here <https://airflow.apache.org/>`__.
+1. Install the latest versions of the Docker Community Edition
+ and Docker Compose and add them to the PATH.
+
+2. Enter Breeze: ``./breeze``
-To generate a local version:
+ Breeze starts with downloading the Airflow CI image from
+ the Docker Hub and installing all required dependencies.
-1. Set up an Airflow development environment.
+3. Enter the Docker environment and mount your local sources
+ to make them immediately visible in the environment.
-2. Install the ``doc`` extra.
+4. Create a local virtualenv, for example:
.. code-block:: bash
- pip install -e '.[doc]'
-
+ mkvirtualenv myenv --python=python3.6
-3. Generate and serve the documentation as follows:
+5. Initialize the created environment:
.. code-block:: bash
@@ -314,7 +252,7 @@ Step 4: Prepare PR
* Read about `email configuration in Airflow <https://airflow.readthedocs.io/en/latest/howto/email-config.html>`__.
- * Find the class you should modify. For the example github issue,
+ * Find the class you should modify. For the example GitHub issue,
this is `email.py <https://github.com/apache/airflow/blob/master/airflow/utils/email.py>`__.
* Find the test class where you should add tests. For the example ticket,
@@ -351,13 +289,50 @@ Step 4: Prepare PR
review/fix. This will make rebase process much easier and less painful and the more often you do it,
the more comfortable you will feel doing it.
-.. note::
- The docs build script ``build`` requires Python 3.6 or greater.
+3. Re-run static code checks again.
-**Known issues:**
+4. Make sure your commit has a good title and description of the context of your change, enough
+ for the committer reviewing it to understand why you are proposing a change. Make sure to follow other
+ PR guidelines described in `pull request guidelines <#pull-request-guidelines>`_.
+ Create Pull Request! Make yourself ready for the discussion!
+
+5. Depending on "scope" of your changes, your Pull Request might go through one of few paths after approval.
+ We run some non-standard workflow with high degree of automation that allows us to optimize the usage
+ of queue slots in Github Actions. Our automated workflows determine the "scope" of changes in your PR
+ and send it through the right path:
+
+ * In case of a "no-code" change, approval will generate a comment that the PR can be merged and no
+ tests are needed. This is usually when the change modifies some non-documentation related rst
+ files (such as this file). No python tests are run and no CI images are built for such PR. Usually
+ it can be approved and merged few minutes after it is submitted (unless there is a big queue of jobs).
+
+ * In case of change involving python code changes or documentation changes, a subset of full test matrix
+ will be executed. This subset of tests perform relevant tests for single combination of python, backend
+ version and only builds one CI image and one PROD image. Here the scope of tests depends on the
+ scope of your changes:
+
+ * when your change does not change "core" of Airflow (Providers, CLI, WWW, Helm Chart) you will get the
+ comment that PR is likely ok to be merged without running "full matrix" of tests. However decision
+ for that is left to committer who approves your change. The committer might set a "full tests needed"
+ label for your PR and ask you to rebase your request or re-run all jobs. PRs with "full tests needed"
+ run full matrix of tests.
+
+ * when your change changes the "core" of Airflow you will get the comment that PR needs full tests and
+ the "full tests needed" label is set for your PR. Additional check is set that prevents from
+ accidental merging of the request until full matrix of tests succeeds for the PR.
+
+ More details about the PR workflow be found in `PULL_REQUEST_WORKFLOW.rst <PULL_REQUEST_WORKFLOW.rst>`_.
+
+
+Step 5: Pass PR Review
+----------------------
+.. image:: images/review.png
+ :align: center
+ :alt: PR Review
-If you are creating ``example_dags`` directory, you need to create ``example_dags/__init__.py`` with Apache license or copy another ``__init__.py`` file that contains the necessary license.
+Note that committers will use **Squash and Merge** instead of **Rebase and Merge**
+when merging PRs and your commit will be squashed to single commit.
You need to have review of at least one committer (if you are committer yourself, it has to be
another committer). Ideally you should have 2 or more committers reviewing the code that touches
@@ -378,23 +353,16 @@ these guidelines:
coverage. You can set up both for free on your fork. It will help you make sure you do not
break the build with your PR and that you help increase coverage.
+- Follow our project's `Coding style and best practices`_.
+
+ These are things that aren't currently enforced programmatically (either because they are too hard or just
+ not yet done.)
+
- `Rebase your fork <http://stackoverflow.com/a/7244456/1110993>`__, squash
commits, and resolve all conflicts.
- When merging PRs, wherever possible try to use **Squash and Merge** instead of **Rebase and Merge**.
-- Make sure every pull request introducing code changes has an associated
- `JIRA <https://issues.apache.org/jira/browse/AIRFLOW/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>`__
- ticket. The JIRA link should also be added to the PR description. In case of documentation only changes
- the JIRA ticket is not necessary.
-
-- Preface your commit's subject & PR title with **[AIRFLOW-NNNN] COMMIT_MSG** where *NNNN*
- is the JIRA number. For example: [AIRFLOW-5574] Fix Google Analytics script loading. In case of
- documentation only changes you should put "[AIRFLOW-XXXX]" instead.
- We compose Airflow release notes from all commit titles in a release. By placing the JIRA number in the
- commit title and hence in the release notes, we let Airflow users look into
- JIRA and GitHub PRs for more details about a particular change.
-
- Add an `Apache License <http://www.apache.org/legal/src-headers.html>`__ header
to all new files.
@@ -411,7 +379,7 @@ these guidelines:
- Run tests locally before opening PR.
-- Make sure the pull request works for Python 2.7, 3.5 and 3.6.
+- Make sure the pull request works for Python 2.7, 3.5, 3.6, 3.7 and 3.8.
- Adhere to guidelines for commit messages described in this `article <http://chris.beams.io/posts/git-commit/>`__.
This makes the lives of those who come after you a lot easier.
@@ -534,7 +502,7 @@ Benefits:
- Breeze environment is almost the same as used in the CI automated builds.
So, if the tests run in your Breeze environment, they will work in the CI as well.
- See `<CI.yml>`_ for details about Airflow CI.
+ See `<CI.rst>`_ for details about Airflow CI.
Limitations:
@@ -555,12 +523,16 @@ Limitations:
They are optimized for repeatability of tests, maintainability and speed of building rather
than production performance. The production images are not yet officially published.
+
+Airflow dependencies
+====================
+
Extras
------
There are a number of extras that can be specified when installing Airflow. Those
extras can be specified after the usual pip install - for example
-``pip install -e .[gcp]``. For development purpose there is a ``devel`` extra that
+``pip install -e .[ssh]``. For development purpose there is a ``devel`` extra that
installs all development dependencies. There is also ``devel_ci`` that installs
all dependencies needed in the CI environment.
@@ -578,9 +550,94 @@ vertica, virtualenv, webhdfs, winrm
.. END EXTRAS HERE
+Provider packages
+-----------------
+
+Airflow 2.0 is split into core and providers. They are delivered as separate packages:
+
+* ``apache-airflow`` - core of Apache Airflow
+* ``apache-airflow-providers-*`` - More than 50 provider packages to communicate with external services
+
+In Airflow 1.10 all those providers were installed together within one single package and when you installed
+airflow locally, from sources, they were also installed. In Airflow 2.0, providers are separated out,
+and not installed together with the core, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment
+variable to ``true``.
+
+In Breeze - which is a development environment, ``INSTALL_PROVIDERS_FROM_SOURCES`` variable is set to true,
+but you can add ``--skip-installing-airflow-providers`` flag to Breeze to skip installing providers when
+building the images.
+
+One watch-out - providers are still always installed (or rather available) if you install airflow from
+sources using ``-e`` (or ``--editable``) flag. In such case airflow is read directly from the sources
+without copying airflow packages to the usual installation location, and since 'providers' folder is
+in this airflow folder - the providers package is importable.
+
+Some of the packages have cross-dependencies with other providers packages. This typically happens for
+transfer operators where operators use hooks from the other providers in case they are transferring
+data between the providers. The list of dependencies is maintained (automatically with pre-commits)
+in the ``airflow/providers/dependencies.json``. Pre-commits are also used to generate dependencies.
+The dependency list is automatically used during pypi packages generation.
+
+Cross-dependencies between provider packages are converted into extras - if you need functionality from
+the other provider package you can install it adding [extra] after the
+``apache-airflow-backport-providers-PROVIDER`` for example:
+``pip install apache-airflow-backport-providers-google[amazon]`` in case you want to use GCP
+transfer operators from Amazon ECS.
+
+If you add a new dependency between different providers packages, it will be detected automatically during
+pre-commit phase and pre-commit will fail - and add entry in dependencies.json so that the package extra
+dependencies are properly added when package is installed.
+
+You can regenerate the whole list of provider dependencies by running this command (you need to have
+``pre-commits`` installed).
+
+.. code-block:: bash
+
+ pre-commit run build-providers-dependencies
+
+
+Here is the list of packages and their extras:
-Airflow dependencies
---------------------
+
+ .. START PACKAGE DEPENDENCIES HERE
+
+========================== ===========================
+Package Extras
+========================== ===========================
+amazon apache.hive,google,imap,mongo,mysql,postgres,ssh
+apache.druid apache.hive
+apache.hive amazon,microsoft.mssql,mysql,presto,samba,vertica
+apache.livy http
+dingding http
+discord http
+google amazon,apache.cassandra,cncf.kubernetes,facebook,microsoft.azure,microsoft.mssql,mysql,postgres,presto,salesforce,sftp
+hashicorp google
+microsoft.azure google,oracle
+microsoft.mssql odbc
+mysql amazon,presto,vertica
+opsgenie http
+postgres amazon
+sftp ssh
+slack http
+snowflake slack
+========================== ===========================
+
+ .. END PACKAGE DEPENDENCIES HERE
+
+Backport providers
+------------------
+
+You can also build backport provider packages for Airflow 1.10. They aim to provide a bridge when users
+of Airflow 1.10 want to migrate to Airflow 2.0. The backport packages are named similarly to the
+provider packages, but with "backport" added:
+
+* ``apache-airflow-backport-provider-*``
+
+Those backport providers are automatically refactored to work with Airflow 1.10.* and have a few
+limitations described in those packages.
+
+Dependency management
+=====================
Airflow is not a standard python project. Most of the python projects fall into one of two types -
application or library. As described in
@@ -599,7 +656,7 @@ This - seemingly unsolvable - puzzle is solved by having pinned constraints file
as of airflow 1.10.10 and further improved with 1.10.12 (moved to separate orphan branches)
Pinned constraint files
------------------------
+=======================
By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to
@@ -635,7 +692,7 @@ This works also with extras - for example:
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt"
-As of apache-airflow 1.10.12 it is also possible to use constraints directly from github using specific
+As of apache-airflow 1.10.12 it is also possible to use constraints directly from GitHub using specific
tag/hash name. We tag commits working for particular release with constraints-<version> tag. So for example
fixed valid constraints 1.10.12 can be used by using ``constraints-1.10.12`` tag:
@@ -713,6 +770,51 @@ slack http
.. END PACKAGE DEPENDENCIES HERE
+Documentation
+=============
+
+The latest API documentation (for the master branch) is usually available
+`here <https://airflow.readthedocs.io/en/latest/>`__.
+
+To generate a local version you can use `<BREEZE.rst>`_.
+
+The documentation build consists of verifying consistency of documentation and two steps:
+
+* spell checking
+* building documentation
+
+You can only run one of the steps via ``--spellcheck-only`` or ``--docs-only``.
+
+.. code-block:: bash
+
+ ./breeze build-docs
+
+or just to run spell-check
+
+.. code-block:: bash
+
+ ./breeze build-docs -- --spellcheck-only
+
+or just to run documentation building
+
+.. code-block:: bash
+
+ ./breeze build-docs
+
+Also documentation is available as downloadable artifact in GitHub Actions after the CI builds your PR.
+
+**Known issues:**
+
+If you are creating a new directory for new integration in the ``airflow.providers`` package,
+you should also update the ``docs/autoapi_templates/index.rst`` file.
+
+If you are creating new ``hooks``, ``sensors``, ``operators`` directory in
+the ``airflow.providers`` package, you should also update
+the ``docs/operators-and-hooks-ref.rst`` file.
+
+If you are creating ``example_dags`` directory, you need to create ``example_dags/__init__.py`` with Apache
+license or copy another ``__init__.py`` file that contains the necessary license.
+
Static code checks
==================
@@ -764,6 +866,67 @@ If this function is designed to be called by "end-users" (i.e. DAG authors) then
...
# You SHOULD not commit the session here. The wrapper will take care of commit()/rollback() if exception
+Naming Conventions for provider packages
+----------------------------------------
+
+In Airflow 2.0 we standardized and enforced naming for provider packages, modules and classes.
+those rules (introduced as AIP-21) were not only introduced but enforced using automated checks
+that verify if the naming conventions are followed. Here is a brief summary of the rules, for
+detailed discussion you can go to [AIP-21 Changes in import paths](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths)
+
+The rules are as follows:
+
+* Provider packages are all placed in 'airflow.providers'
+
+* Providers are usually direct sub-packages of the 'airflow.providers' package but in some cases they can be
+ further split into sub-packages (for example 'apache' package has 'cassandra', 'druid' ... providers ) out
+ of which several different provider packages are produced (apache.cassandra, apache.druid). This is
+ case when the providers are connected under common umbrella but very loosely coupled on the code level.
+
+* In some cases the package can have sub-packages but they are all delivered as single provider
+ package (for example 'google' package contains 'ads', 'cloud' etc. sub-packages). This is in case
+ the providers are connected under common umbrella and they are also tightly coupled on the code level.
+
+* Typical structure of provider package:
+ * example_dags -> example DAGs are stored here (used for documentation and System Tests)
+ * hooks -> hooks are stored here
+ * operators -> operators are stored here
+ * sensors -> sensors are stored here
+ * secrets -> secret backends are stored here
+ * transfers -> transfer operators are stored here
+
+* Module names do not contain word "hooks", "operators" etc. The right type comes from
+ the package. For example 'hooks.datastore' module contains DataStore hook and 'operators.datastore'
+ contains DataStore operators.
+
+* Class names contain 'Operator', 'Hook', 'Sensor' - for example DataStoreHook, DataStoreExportOperator
+
+* Operator name usually follows the convention: ``<Subject><Action><Entity>Operator``
+ (BigQueryExecuteQueryOperator) is a good example
+
+* Transfer Operators are those that actively push data from one service/provider and send it to another
+ service (might be for the same or another provider). This usually involves two hooks. The convention
+ for those ``<Source>To<Destination>Operator``. They are not named *TransferOperator nor *Transfer.
+
+* Operators that use external service to perform transfer (for example CloudDataTransferService operators
+ are not placed in "transfers" package and do not have to follow the naming convention for
+ transfer operators.
+
+* It is often debatable where to put transfer operators but we agreed to the following criteria:
+
+ * We use "maintainability" of the operators as the main criteria - so the transfer operator
+ should be kept at the provider which has highest "interest" in the transfer operator
+
+ * For Cloud Providers or Service providers that usually means that the transfer operators
+ should land at the "target" side of the transfer
+
+* Secret Backend name follows the convention: ``<SecretEngine>Backend``.
+
+* Tests are grouped in parallel packages under "tests.providers" top level package. Module name is usually
+ ``test_<object_to_test>.py``,
+
+* System tests (not yet fully automated but allowing to run e2e testing of particular provider) are
+ named with _system.py suffix.
Test Infrastructure
===================
@@ -779,12 +942,12 @@ We support the following types of tests:
additional services running, such as Postgres, Mysql, Kerberos, etc.
* **System tests** are automatic tests that use external systems like
- Google Cloud Platform. These tests are intended for an end-to-end DAG execution.
+ Google Cloud. These tests are intended for an end-to-end DAG execution.
For details on running different types of Airflow tests, see `TESTING.rst <TESTING.rst>`_.
Metadata Database Updates
-==============================
+=========================
When developing features, you may need to persist information to the metadata
database. Airflow has `Alembic <https://github.com/sqlalchemy/alembic>`__ built-in
@@ -864,7 +1027,7 @@ could get a reproducible build. See the `Yarn docs
Generate Bundled Files with yarn
-----------------------------------
+--------------------------------
To parse and generate bundled files for Airflow, run either of the following
commands:
@@ -878,13 +1041,13 @@ commands:
yarn run dev
-Follow Javascript Style Guide
+Follow JavaScript Style Guide
-----------------------------
We try to enforce a more consistent style and follow the JS community
guidelines.
-Once you add or modify any javascript code in the project, please make sure it
+Once you add or modify any JavaScript code in the project, please make sure it
follows the guidelines defined in `Airbnb
JavaScript Style Guide <https://github.com/airbnb/javascript>`__.
@@ -900,164 +1063,18 @@ commands:
# Check JS code in .js and .html files, report any errors/warnings and fix them if possible
yarn run lint:fix
-Contribution Workflow Example
-==============================
-
-Typically, you start your first contribution by reviewing open tickets
-at `Apache JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__.
-
-If you create pull-request, you don't have to create an issue first, but if you want, you can do it.
-Creating an issue will allow you to collect feedback or share plans with other people.
-
-If you create pull-request, you don't have to create an issue first, but if you want, you can do it.
-Creating an issue will allow you to collect feedback or share plans with other people.
-
-For example, you want to have the following sample ticket assigned to you:
-`AIRFLOW-5934: Add extra CC: to the emails sent by Aiflow <https://issues.apache.org/jira/browse/AIRFLOW-5934>`_.
-
-In general, your contribution includes the following stages:
-
-.. image:: images/workflow.png
- :align: center
- :alt: Contribution Workflow
-
-1. Make your own `fork <https://help.github.com/en/github/getting-started-with-github/fork-a-repo>`__ of
- the Apache Airflow `main repository <https://github.com/apache/airflow>`__.
-
-2. Create a `local virtualenv <LOCAL_VIRTUALENV.rst>`_,
- initialize the `Breeze environment <BREEZE.rst>`__, and
- install `pre-commit framework <STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__.
- If you want to add more changes in the future, set up your fork and enable Github Actions.
-
-3. Join `devlist <ht...@airflow.apache.org>`__
- and set up a `Slack account <https://apache-airflow-slack.herokuapp.com>`__.
-
-4. Make the change and create a `Pull Request from your fork <https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request-from-a-fork>`__.
-
-5. Ping @ #development slack, comment @people. Be annoying. Be considerate.
-
-Step 1: Fork the Apache Repo
-----------------------------
-From the `apache/airflow <https://github.com/apache/airflow>`_ repo,
-`create a fork <https://help.github.com/en/github/getting-started-with-github/fork-a-repo>`_:
-
-.. image:: images/fork.png
- :align: center
- :alt: Creating a fork
-
-
-Step 2: Configure Your Environment
-----------------------------------
-Configure the Docker-based Breeze development environment and run tests.
-
-You can use the default Breeze configuration as follows:
-
-1. Install the latest versions of the Docker Community Edition
- and Docker Compose and add them to the PATH.
-
-2. Enter Breeze: ``./breeze``
-
- Breeze starts with downloading the Airflow CI image from
- the Docker Hub and installing all required dependencies.
-
-3. Enter the Docker environment and mount your local sources
- to make them immediately visible in the environment.
-
-4. Create a local virtualenv, for example:
-
-.. code-block:: bash
-
- mkvirtualenv myenv --python=python3.6
-
-5. Initialize the created environment:
-
-.. code-block:: bash
-
- ./breeze --initialize-local-virtualenv
-
-6. Open your IDE (for example, PyCharm) and select the virtualenv you created
- as the project's default virtualenv in your IDE.
-
-Step 3: Connect with People
----------------------------
-
-For effective collaboration, make sure to join the following Airflow groups:
-
-- Mailing lists:
-
- - Developer’s mailing list `<de...@airflow.apache.org>`_
- (quite substantial traffic on this list)
-
- - All commits mailing list: `<co...@airflow.apache.org>`_
- (very high traffic on this list)
-
- - Airflow users mailing list: `<us...@airflow.apache.org>`_
- (reasonably small traffic on this list)
-
-- `Issues on Apache’s JIRA <https://issues.apache.org/jira/browse/AIRFLOW>`__
-
-- `Slack (chat) <https://apache-airflow-slack.herokuapp.com/>`__
-
-Step 4: Prepare PR
-------------------
-
-1. Update the local sources to address the JIRA ticket.
-
- For example, to address this example JIRA ticket, do the following:
-
- * Read about `email configuration in Airflow <https://airflow.readthedocs.io/en/latest/howto/email-config.html>`__.
-
- * Find the class you should modify. For the example ticket,
- this is `email.py <https://github.com/apache/airflow/blob/master/airflow/utils/email.py>`__.
-
- * Find the test class where you should add tests. For the example ticket,
- this is `test_email.py <https://github.com/apache/airflow/blob/master/tests/utils/test_email.py>`__.
-
- * Create a local branch for your development. Make sure to use latest
- ``apache/master`` as base for the branch. See `How to Rebase PR <#how-to-rebase-pr>`_ for some details
- on setting up the ``apache`` remote. Note - some people develop their changes directy in their own
- ``master`` branches - this is OK and you can make PR from your master to ``apache/master`` but we
- recommend to always create a local branch for your development. This allows you to easily compare
- changes, have several changes that you work on at the same time and many more.
- If you have ``apache`` set as remote then you can make sure that you have latest changes in your master
- by ``git pull apache master`` when you are in the local ``master`` branch. If you have conflicts and
- want to override your locally changed master you can override your local changes with
- ``git fetch apache; git reset --hard apache/master``.
-
- * Modify the class and add necessary code and unit tests.
-
- * Run the unit tests from the `IDE <TESTING.rst#running-unit-tests-from-ide>`__
- or `local virtualenv <TESTING.rst#running-unit-tests-from-local-virtualenv>`__ as you see fit.
-
- * Run the tests in `Breeze <TESTING.rst#running-unit-tests-inside-breeze>`__.
-
- * Run and fix all the `static checks <STATIC_CODE_CHECKS>`__. If you have
- `pre-commits installed <STATIC_CODE_CHECKS.rst#pre-commit-hooks>`__,
- this step is automatically run while you are committing your code. If not, you can do it manually
- via ``git add`` and then ``pre-commit run``.
-
-2. Rebase your fork, squash commits, and resolve all conflicts. See `How to rebase PR <#how-to-rebase-pr>`_
- if you need help with rebasing your change. Remember to rebase often if your PR takes a lot of time to
- review/fix. This will make rebase process much easier and less painful - and the more often you do it,
- the more comfortable you will feel doing it.
-
-3. Re-run static code checks again.
-
-4. Create a pull request with the following title for the sample ticket:
- ``[AIRFLOW-5934] Added extra CC: field to the Airflow emails.``
-
-Make sure to follow other PR guidelines described in `this document <#pull-request-guidelines>`_.
-
+How to sync your fork
+=====================
-Step 5: Pass PR Review
-----------------------
+When you have your fork, you should periodically synchronize the master of your fork with the
+Apache Airflow master. In order to do that you can ``git pull --rebase`` to your local git repository from
+apache remote and push the master (often with ``--force`` to your fork). There is also an easy
+way using ``Force sync master from apache/airflow`` workflow. You can go to "Actions" in your repository and
+choose the workflow and manually trigger the workflow using "Run workflow" command.
-.. image:: images/review.png
- :align: center
- :alt: PR Review
+This will force-push the master from apache/airflow to the master in your fork. Note that in case you
+modified the master in your fork, you might loose those changes.
-Note that committers will use **Squash and Merge** instead of **Rebase and Merge**
-when merging PRs and your commit will be squashed to single commit.
How to rebase PR
================
@@ -1065,7 +1082,6 @@ How to rebase PR
A lot of people are unfamiliar with the rebase workflow in Git, but we think it is an excellent workflow,
providing a better alternative to the merge workflow. We've therefore written a short guide for those who would like to learn it.
-
As opposed to the merge workflow, the rebase workflow allows us to
clearly separate your changes from the changes of others. It puts the responsibility of rebasing on the
author of the change. It also produces a "single-line" series of commits on the master branch. This
@@ -1162,6 +1178,8 @@ community are far more important than their contribution.
This means that communication plays a big role in it, and this chapter is all about it.
+In our communication, everyone is expected to follow the `ASF Code of Conduct <https://www.apache.org/foundation/policies/conduct>`_.
+
We have various channels of communication - starting from the official devlist, comments
in the Pull Requests, Slack, wiki.
@@ -1187,7 +1205,7 @@ You can join the channels via links at the `Airflow Community page <https://airf
**IMPORTANT**
We don't create new issues on JIRA anymore. The reason we still look at JIRA issues is that there are valuable tickets inside of it. However, each new PR should be created on `GitHub issues <https://github.com/apache/airflow/issues>`_ as stated in `Contribution Workflow Example <https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example>`_
-* The `Apache Airflow Slack <https://apache-airflow-slack.herokuapp.com/>`_ for:
+* The `Apache Airflow Slack <https://s.apache.org/airflow-slack>`_ for:
* ad-hoc questions related to development (#development channel)
* asking for review (#development channel)
* asking for help with PRs (#how-to-pr channel)
@@ -1269,7 +1287,7 @@ This means that committers should:
* Review PRs in a timely and reliable fashion
* They should also help to actively whittle down the PR backlog
-* Answer questions (i.e. on the dev list, in PRs, in Github Issues, slack, etc...)
+* Answer questions (i.e. on the dev list, in PRs, in GitHub Issues, slack, etc...)
* Take on core changes/bugs/feature requests
* Some changes are important enough that a committer needs to ensure it gets done. This is especially
the case if no one from the community is taking it on.
[airflow] 38/44: Docker context files should be available earlier
(#12219)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10b6c48b67ec2c23b256897a6e5b562178a1eb9b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Nov 11 11:00:16 2020 +0100
Docker context files should be available earlier (#12219)
If you want to override constraints with local version,
the docker-context-files should be earlier in the Dockerfile
(cherry picked from commit 9b7e7603c4a01c6c943fa68df3b20a8859cdde1e)
---
.gitignore | 2 ++
Dockerfile.ci | 8 ++++----
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/.gitignore b/.gitignore
index 27b6f0c..6b2650c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -188,6 +188,8 @@ dmypy.json
log.txt*
/backport_packages/CHANGELOG.txt
+# Docker context files
+/docker-context-files
# Local .terraform directories
**/.terraform/*
diff --git a/Dockerfile.ci b/Dockerfile.ci
index fae86b2..7699706 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -260,6 +260,10 @@ ENV AIRFLOW_LOCAL_PIP_WHEELS=${AIRFLOW_LOCAL_PIP_WHEELS}
ARG INSTALL_AIRFLOW_VIA_PIP="true"
ENV INSTALL_AIRFLOW_VIA_PIP=${INSTALL_AIRFLOW_VIA_PIP}
+# If wheel files are found in /docker-context-files during installation
+# they are also installed additionally to whatever is installed from Airflow.
+COPY docker-context-files /docker-context-files
+
# In case of CI builds we want to pre-install master version of airflow dependencies so that
# We do not have to always reinstall it from the scratch.
# This can be reinstalled from latest master by increasing PIP_DEPENDENCIES_EPOCH_NUMBER.
@@ -311,10 +315,6 @@ RUN if [[ ${INSTALL_AIRFLOW_VIA_PIP} == "true" ]]; then \
fi; \
fi
-# If wheel files are found in /docker-context-files sduring installation
-# they are also installed additionally to whatever is installed from Airflow.
-COPY docker-context-files /docker-context-files
-
RUN if [[ ${AIRFLOW_LOCAL_PIP_WHEELS} != "true" ]]; then \
if ls /docker-context-files/*.whl 1> /dev/null 2>&1; then \
pip install --no-deps /docker-context-files/*.whl; \
[airflow] 42/44: For v1-10-test PRs and pushes,
use target branch scripts for images (#12339)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ab76df0eb547fc2af670473d38536ab9b1774b1
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Nov 13 14:28:36 2020 +0100
For v1-10-test PRs and pushes, use target branch scripts for images (#12339)
Previously, always master scripts were used to build images
for workflow_run, because workflow_run always runs from master
branch. However that causes some surprising effects becuase the
sripts from master had to support both master and 1.10.
This change utilises a new feature in the "get-workflow-origin"
action - to get the target branch of PR and uses ci scripts from that
target branch.
This is perfectly secure, because both v1-10-test, v1-10-stable
and future 2-0 branches can only be updated by committers,
either by direct push or by merge.
(cherry picked from commit 7c4fe19e41ae02a1df1c0a217501cae2e0e84819)
---
.github/workflows/build-images-workflow-run.yml | 12 +++++--
.github/workflows/ci.yml | 38 +++++++---------------
.../workflows/label_when_reviewed_workflow_run.yml | 2 +-
3 files changed, 22 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index a3277a7..af71710 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -53,13 +53,14 @@ jobs:
targetCommitSha: ${{ steps.source-run-info.outputs.targetCommitSha }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
+ targetBranch: ${{ steps.source-run-info.outputs.targetBranch }}
sourceEvent: ${{ steps.source-run-info.outputs.sourceEvent }}
cacheDirective: ${{ steps.cache-directive.outputs.docker-cache }}
buildImages: ${{ steps.build-images.outputs.buildImages }}
upgradeToLatestConstraints: ${{ steps.upgrade-constraints.outputs.upgradeToLatestConstraints }}
steps:
- name: "Get information about the original trigger of the run"
- uses: potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses: potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -326,17 +327,22 @@ jobs:
[Image Build](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
for details" }
if: steps.defaults.outputs.proceed == 'true'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} ) to 'main-airflow' to use main scripts"
+ - name: >
+ Checkout "${{ needs.cancel-workflow-runs.outputs.targetBranch }}" branch to 'main-airflow' folder
+ to use ci/scripts from there.
uses: actions/checkout@v2
with:
path: "main-airflow"
+ ref: "${{ needs.cancel-workflow-runs.outputs.targetBranch }}"
if: steps.defaults.outputs.proceed == 'true'
- name: "Setup python"
uses: actions/setup-python@v2
with:
python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
if: steps.defaults.outputs.proceed == 'true'
- - name: "Override 'scripts/ci' with the ${{ github.ref }} version so that the PR cannot override it."
+ - name: >
+ Override "scripts/ci" with the "${{ needs.cancel-workflow-runs.outputs.targetBranch }}" branch
+ so that the PR does not override it
# We should not override those scripts which become part of the image as they will not be
# changed in the image built - we should only override those that are executed to build
# the image.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 81890a7..df5f53e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -95,27 +95,15 @@ jobs:
image-build: ${{ steps.selective-checks.outputs.image-build }}
docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
- needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
pullRequestLabels: ${{ steps.source-run-info.outputs.pullRequestLabels }}
steps:
- name: "Get information about the PR"
- uses: potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses: potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
- # First fetch the sha of merge commit in case it is pull request so that we can
- # Run selective tests
- - name: >
- Fetch merge commit ${{ github.ref }} ( ${{ github.sha }}:
- merge_commit ${{ github.event.pull_request.merge_commit_sha }} )
- uses: actions/checkout@v2
- with:
- ref: ${{ github.event.pull_request.merge_commit_sha }}
- fetch-depth: 2
- if: github.event_name == 'pull_request'
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }} )"
uses: actions/checkout@v2
- name: >
Event: ${{ github.event_name }}
@@ -123,7 +111,6 @@ jobs:
Branch: ${{ github.head_ref }}
Run id: ${{ github.run_id }}
Sha: ${{ github.sha }}
- Merge commit sha: ${{ github.merge_commit_sha }}
Ref: ${{ github.ref }}
run: printenv
- name: Set wait for image
@@ -164,7 +151,7 @@ jobs:
env:
BACKEND: sqlite
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }} )"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
if: needs.build-info.outputs.waitForImage == 'true'
- name: "Setup python"
@@ -199,7 +186,7 @@ jobs:
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }} )"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -267,7 +254,7 @@ jobs:
needs: [build-info, ci-images]
if: needs.build-info.outputs.docs-build == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
@@ -348,7 +335,7 @@ jobs:
TEST_TYPE: ""
if: needs.build-info.outputs.run-tests == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -400,7 +387,7 @@ jobs:
TEST_TYPE: ""
if: needs.build-info.outputs.run-tests == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -449,7 +436,7 @@ jobs:
TEST_TYPE: ""
if: needs.build-info.outputs.run-tests == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -503,7 +490,7 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
if: needs.build-info.outputs.run-tests == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -628,7 +615,7 @@ jobs:
HELM_VERSION: "${{ needs.build-info.outputs.defaultHelmVersion }}"
if: needs.build-info.outputs.run-kubernetes-tests == 'true'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -736,7 +723,7 @@ jobs:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
GITHUB_REGISTRY_PUSH_IMAGE_TAG: "latest"
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -764,7 +751,7 @@ jobs:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test'
steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: "Setup python"
uses: actions/setup-python@v2
@@ -790,7 +777,6 @@ jobs:
- build-info
- constraints
- static-checks
- - static-checks-pylint
- tests-sqlite
- tests-mysql
- tests-postgres
@@ -814,7 +800,7 @@ jobs:
- name: "Commit changed constraint files for ${{needs.build-info.outputs.pythonVersions}}"
run: ./scripts/ci/constraints/ci_commit_constraints.sh
- name: "Push changes"
- uses: ad-m/github-push-action@master
+ uses: ad-m/github-push-action@40bf560936a8022e68a3c00e7d2abefaf01305a6 # v0.6.0
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: ${{ steps.constraints-branch.outputs.branch }}
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml b/.github/workflows/label_when_reviewed_workflow_run.yml
index c2359c8..f943609 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -30,7 +30,7 @@ jobs:
labelSet: ${{ steps.label-when-reviewed.outputs.labelSet }}
steps:
- name: "Get information about the original trigger of the run"
- uses: potiuk/get-workflow-origin@2ef0b065db6b688a2231f8a7f464df1aac254328 # v1_2
+ uses: potiuk/get-workflow-origin@588cc14f9f1cdf1b8be3db816855e96422204fec # v1_3
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
[airflow] 05/44: Implements canceling of future duplicate runs (but
the latest) (#11980)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad5679025d84378a54ac22a9203c3cc75f7aedd8
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Oct 30 21:48:40 2020 +0100
Implements canceling of future duplicate runs (but the latest) (#11980)
* Implements canceling of future duplicate runs (but the latest)
Previous version of the cancel-workflow-runs action implemented
canceling of only past duplicates, but when there are queues
involved, some future "cancel-workflow-runs" might be in a queue
for long time. This change has the effect that cancel-workflow-runs
for duplicates will also allow future runs of the same branch/repo
sparing only the most recent run - no matter if the duplicates
were older than my own run.
This should handle the case where we have queues blocking the
"cancel-workflow-runs" from running.
* Update .github/workflows/build-images-workflow-run.yml
Co-authored-by: Kaxil Naik <ka...@gmail.com>
Co-authored-by: Kaxil Naik <ka...@gmail.com>
(cherry picked from commit 36b5cc6df0c7b49bcd9d38c37c2ed4b881089d6d)
---
.github/workflows/build-images-workflow-run.yml | 19 ++++++++-----------
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index a4f2dbe..81c4fb4 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -21,7 +21,6 @@ on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ["CI Build"]
types: ['requested']
-
env:
MOUNT_LOCAL_SOURCES: "false"
MOUNT_FILES: "true"
@@ -66,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
@@ -84,7 +83,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,9 +92,7 @@ jobs:
[".*Event: ${{ steps.source-run-info.outputs.sourceEvent }}
Repo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
- if: >
- env.BUILD_IMAGES == 'true' && steps.source-run-info.outputs.sourceEvent != 'schedule'
- && steps.source-run-info.outputs.sourceEvent != 'push'
+ if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
# We find any of the "CI Build" workflow runs, where any of the important jobs
@@ -104,7 +101,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -138,14 +135,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -376,7 +373,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@cancel_message # v3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -391,7 +388,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@0acb1c01f6740dfbca6eab6e21a5b5066e8bafb3 # v3_3
+ uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 41/44: Deploy was not working from Breeze (#12319)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 891a01d0ee3b68c8a91b60c6f82fe907aac0c4b1
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Nov 12 19:53:35 2020 +0100
Deploy was not working from Breeze (#12319)
The get_cluster_name was called twice resulting in redonly
error after rebasing/fixing CI failure in #12163.
This PR is fxing it.
(cherry picked from commit af19b126e94876c371553f6a7cfae6b1102f79fd)
---
scripts/ci/libraries/_kind.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index defa4de..3a170cb 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -173,7 +173,6 @@ function kind::perform_kind_cluster_operation() {
echo "Deploying Airflow to KinD"
echo
kind::build_image_for_kubernetes_tests
- kind::get_kind_cluster_name
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
kind::deploy_test_kubernetes_resources
[airflow] 37/44: Fixes continuous image rebuilding with Breeze
(#12256)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 4ca85a6a315c5c46aad95a67482e09de707c0293
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Nov 10 17:34:52 2020 +0100
Fixes continuous image rebuilding with Breeze (#12256)
There was a problem that even if we pulled the right image
from the Airflow repository, we have not tagged it properly.
Also added protection for people who have not yet at all pulled
the Python image from airflow, to force pull for the first time.
(cherry picked from commit 09febee4c1caf4e430fcbad2386f80aab52a0b15)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index e853f42..4c51f0a 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -105,6 +105,11 @@ function push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to be pulled
function push_pull_remove_images::pull_ci_images_if_needed() {
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
+ local python_image_hash
+ python_image_hash=$(docker images -q "${AIRFLOW_CI_PYTHON_IMAGE}" 2> /dev/null || true)
+ if [[ -z "${python_image_hash=}" ]]; then
+ FORCE_PULL_IMAGES="true"
+ fi
if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
echo
echo "Force pull base image ${PYTHON_BASE_IMAGE}"
@@ -122,6 +127,7 @@ Docker pulling ${PYTHON_BASE_IMAGE}.
push_pull_remove_images::pull_image_github_dockerhub "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+ docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 17/44: Uses DOCKER_TAG when building image in DockerHub
(#12050)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7512344582888106d46fb871ecd9804d2c706f6b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 2 22:00:51 2020 +0100
Uses DOCKER_TAG when building image in DockerHub (#12050)
DockerHub uses `hooks/build` to build the image and it passes
DOCKER_TAG variable when the script is called.
This PR makes the DOCKER_TAG to provide the default valuei for tag
that is calculated from sources (taking the default branch and
python version). Since it is only set in the DockerHub build, it
should be safe.
Fixes #11937
(cherry picked from commit 5c199fbddfaf9f83915e84225313169a0486c3a6)
---
scripts/ci/libraries/_build_images.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 444fff9..387372a 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,7 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
- export AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
+ export AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
# CI image to build
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +336,7 @@ function build_images::get_docker_image_names() {
export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
- export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+ export AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
# PROD image to build
export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 36/44: Add Markdown linting to pre-commit (#11465)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64af9450e37faeb5b08e9568938d6cdf2d214f6d
Author: John Bampton <jb...@users.noreply.github.com>
AuthorDate: Tue Nov 10 12:37:45 2020 +1000
Add Markdown linting to pre-commit (#11465)
(cherry picked from commit 7463b6bcc7a823a9d02c7026d9b03b81c6421654)
---
.markdownlint.yml | 77 +++
.pre-commit-config.yaml | 32 +-
BREEZE.rst | 8 +-
STATIC_CODE_CHECKS.rst | 4 +-
breeze-complete | 2 +
chart/README.md | 2 +-
dev/README.md | 1522 ++++++++++++++++++++++++++++++++++++++++++++---
7 files changed, 1552 insertions(+), 95 deletions(-)
diff --git a/.markdownlint.yml b/.markdownlint.yml
new file mode 100644
index 0000000..dae8217
--- /dev/null
+++ b/.markdownlint.yml
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+# MD003/heading-style/header-style
+MD003: false
+
+# MD004/ul-style
+MD004: false
+
+# MD007/ul-indent
+MD007: false
+
+# MD012/no-multiple-blanks
+MD012: false
+
+# MD013 Line length
+MD013: false
+
+# MD014/commands-show-output
+MD014: false
+
+# MD022/blanks-around-headings/blanks-around-headers
+MD022: false
+
+# MD024/no-duplicate-heading/no-duplicate-header
+MD024: false
+
+# MD026/no-trailing-punctuation
+MD026: false
+
+# MD029/ol-prefix
+MD029: false
+
+# MD030/list-marker-space
+MD030: false
+
+# MD031/blanks-around-fences
+MD031: false
+
+# MD032/blanks-around-lists
+MD032: false
+
+# MD033/no-inline-html
+MD033: false
+
+# MD034/no-bare-urls
+MD034: false
+
+# MD036/no-emphasis-as-heading/no-emphasis-as-header
+MD036: false
+
+# MD040/fenced-code-language
+MD040: false
+
+# MD041/first-line-heading/first-line-h1
+MD041: false
+
+# MD045/no-alt-text
+MD045: false
+
+# MD046/code-block-style
+MD046: false
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index e626cd4..4dbf84c 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -333,6 +333,26 @@ repos:
files: ^breeze$|^breeze-complete$|\.sh$|\.bash$|\.bats$
exclude: ^tests/bats/in_container/.*bats$|^scripts/in_container/.*sh$
pass_filenames: false
+ - id: pre-commit-descriptions
+ name: Check if pre-commits are described
+ entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh
+ language: system
+ files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$
+ require_serial: true
+ - id: sort-in-the-wild
+ name: Sort INTHEWILD.md alphabetically
+ entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh
+ language: system
+ files: ^.pre-commit-config.yaml$|^INTHEWILD.md$
+ require_serial: true
+ - id: markdownlint
+ name: Run markdownlint
+ description: "Checks the style of Markdown files."
+ entry: markdownlint
+ language: node
+ types: [markdown]
+ files: \.(md|mdown|markdown)$
+ additional_dependencies: ['markdownlint-cli']
- id: build
name: Check if image build is needed
entry: ./scripts/ci/pre_commit/pre_commit_ci_build.sh 3.6 false
@@ -364,15 +384,3 @@ repos:
entry: "./scripts/ci/pre_commit/pre_commit_in_container_bats_test.sh"
files: ^tests/bats/in_container/.*.bats$|^scripts/in_container/.*sh
pass_filenames: false
- - id: pre-commit-descriptions
- name: Check if pre-commits are described
- entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commits.sh
- language: system
- files: ^.pre-commit-config.yaml$|^STATIC_CODE_CHECKS.rst|^breeze-complete$
- require_serial: true
- - id: sort-in-the-wild
- name: Sort INTHEWILD.md alphabetically
- entry: ./scripts/ci/pre_commit/pre_commit_sort_in_the_wild.sh
- language: system
- files: ^.pre-commit-config.yaml$|^INTHEWILD.md$
- require_serial: true
diff --git a/BREEZE.rst b/BREEZE.rst
index b66c7d2..c3c2d95 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1863,10 +1863,10 @@ This is the current syntax for `./breeze <./breeze>`_:
check-hooks-apply check-integrations check-merge-conflict check-xml debug-statements
detect-private-key doctoc dont-use-safe-filter end-of-file-fixer fix-encoding-pragma
flake8 forbid-tabs helm-lint incorrect-use-of-LoggingMixin insert-license
- language-matters lint-dockerfile lint-openapi mixed-line-ending mypy mypy-helm
- no-relative-imports pre-commit-descriptions pydevd python2-compile python2-fastcheck
- python-no-log-warn rst-backticks setup-order setup-installation shellcheck
- sort-in-the-wild trailing-whitespace update-breeze-file update-extras
+ language-matters lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending
+ mypy mypy-helm no-relative-imports pre-commit-descriptions pydevd python2-compile
+ python2-fastcheck python-no-log-warn rst-backticks setup-order setup-installation
+ shellcheck sort-in-the-wild trailing-whitespace update-breeze-file update-extras
update-local-yml-file yamllint
You can pass extra arguments including options to to the pre-commit framework as
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index ce68e0a..3fb49c3 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -108,10 +108,12 @@ require Breeze Docker images to be installed locally:
----------------------------------- ---------------------------------------------------------------- ------------
``lint-openapi`` Lints openapi specification.
----------------------------------- ---------------------------------------------------------------- ------------
-``mixed-line-ending`` Detects if mixed line ending is used (\r vs. \r\n).
+``markdownlint`` Lints Markdown files.
----------------------------------- ---------------------------------------------------------------- ------------
``mermaid`` Generates diagrams from mermaid files.
----------------------------------- ---------------------------------------------------------------- ------------
+``mixed-line-ending`` Detects if mixed line ending is used (\r vs. \r\n).
+----------------------------------- ---------------------------------------------------------------- ------------
``mypy`` Runs mypy. *
----------------------------------- ---------------------------------------------------------------- ------------
``mypy-helm`` Runs mypy. *
diff --git a/breeze-complete b/breeze-complete
index 1b89b70..4855c86 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -91,6 +91,8 @@ insert-license
language-matters
lint-dockerfile
lint-openapi
+markdownlint
+mermaid
mixed-line-ending
mypy
mypy-helm
diff --git a/chart/README.md b/chart/README.md
index 7d23ce3..11dc632 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -208,7 +208,7 @@ helm install --name my-release \
--set enablePodLaunching=false .
```
-## Autoscaling with KEDA
+## Autoscaling with KEDA
KEDA stands for Kubernetes Event Driven Autoscaling. [KEDA](https://github.com/kedacore/keda) is a custom controller that allows users to create custom bindings
to the Kubernetes [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
diff --git a/dev/README.md b/dev/README.md
index 85248f6..b5ad9a3 100644
--- a/dev/README.md
+++ b/dev/README.md
@@ -16,111 +16,1441 @@
specific language governing permissions and limitations
under the License.
-->
-# Development Tools
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of contents**
+
+- [Apache Airflow source releases](#apache-airflow-source-releases)
+ - [Apache Airflow Package](#apache-airflow-package)
+ - [Backport Provider packages](#backport-provider-packages)
+- [Prerequisites for the release manager preparing the release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the packages](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Airflow packages](#apache-airflow-packages)
+ - [Prepare the Apache Airflow Package RC](#prepare-the-apache-airflow-package-rc)
+ - [Vote and verify the Apache Airflow release candidate](#vote-and-verify-the-apache-airflow-release-candidate)
+ - [Publish the final Apache Airflow release](#publish-the-final-apache-airflow-release)
+- [Provider Packages](#provider-packages)
+ - [Decide when to release](#decide-when-to-release)
+ - [Prepare the Backport Provider Packages RC](#prepare-the-backport-provider-packages-rc)
+ - [Vote and verify the Backport Providers release candidate](#vote-and-verify-the-backport-providers-release-candidate)
+ - [Publish the final releases of backport packages](#publish-the-final-releases-of-backport-packages)
+ - [Prepare the Regular Provider Packages Alpha](#prepare-the-regular-provider-packages-alpha)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Apache Airflow source releases
+
+The Apache Airflow releases are one of the two types:
+
+* Releases of the Apache Airflow package
+* Releases of the Backport Providers Packages
+
+## Apache Airflow Package
+
+This package contains sources that allow the user building fully-functional Apache Airflow 2.0 package.
+They contain sources for:
+
+ * "apache-airflow" python package that installs "airflow" Python package and includes
+ all the assets required to release the webserver UI coming with Apache Airflow
+ * Dockerfile and corresponding scripts that build and use an official DockerImage
+ * Breeze development environment that helps with building images and testing locally
+ apache airflow built from sources
+
+In the future (Airflow 2.0) this package will be split into separate "core" and "providers" packages that
+will be distributed separately, following the mechanisms introduced in Backport Package Providers. We also
+plan to release the official Helm Chart sources that will allow the user to install Apache Airflow
+via helm 3.0 chart in a distributed fashion.
+
+The Source releases are the only "official" Apache Software Foundation releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
+
+Following source releases Apache Airflow release manager also distributes convenience packages:
+
+* PyPI packages released via https://pypi.org/project/apache-airflow/
+* Docker Images released via https://hub.docker.com/repository/docker/apache/airflow
+
+Those convenience packages are not "official releases" of Apache Airflow, but the users who
+cannot or do not want to build the packages themselves can use them as a convenient way of installing
+Apache Airflow, however they are not considered as "official source releases". You can read more
+details about it in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html).
+
+This document describes the process of releasing both - official source packages and convenience
+packages for Apache Airflow packages.
+
+## Backport Provider packages
+
+The Backport Provider packages are packages (per provider) that make it possible to easily use Hooks,
+Operators, Sensors, and Secrets from the 2.0 version of Airflow in the 1.10.* series.
+
+Once you release the packages, you can simply install them with:
+
+```
+pip install apache-airflow-backport-providers-<PROVIDER>[<EXTRAS>]
+```
+
+Where `<PROVIDER>` is the provider id and `<EXTRAS>` are optional extra packages to install.
+You can find the provider packages dependencies and extras in the README.md files in each provider
+package (in `airflow/providers/<PROVIDER>` folder) as well as in the PyPI installation page.
+
+Backport providers are a great way to migrate your DAGs to Airflow-2.0 compatible DAGs. You can
+switch to the new Airflow-2.0 packages in your DAGs, long before you attempt to migrate
+airflow to 2.0 line.
+
+The sources released in SVN allow to build all the provider packages by the user, following the
+instructions and scripts provided. Those are also "official_source releases" as described in the
+[ASF Release Policy](http://www.apache.org/legal/release-policy.html) and they are available
+via [Official Apache Download sources](https://downloads.apache.org/airflow/backport-providers/).
+
+There are also 50+ convenience packages released as "apache-airflow-backport-providers" separately in
+PyPI. You can find them all by [PyPI query](https://pypi.org/search/?q=apache-airflow-backport-providers)
+
+The document describes the process of releasing both - official source packages and convenience
+packages for Backport Provider Packages.
+
+# Prerequisites for the release manager preparing the release
+
+The person acting as release manager has to fulfill certain pre-requisites. More details and FAQs are
+available in the [ASF Release Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the release manager
+can ask PMC to perform that final step of release.
+
+## Upload Public keys to id.apache.org
+
+Make sure your public key is on id.apache.org and in KEYS. You will need to sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
+
+- Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of that file. Upload your GPG public key to https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache credentials, paste your fingerprint into the pgp fingerprint field and hit save).
+
+```shell script
+# Create PGP Key
+gpg --gen-key
+
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/airflow
+cd airflow
+
+
+# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
+(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS
+
+
+# Commit the changes
+svn commit -m "Add PGP keys of Airflow developers"
+```
+
+See this for more detail on creating keys and what is required for signing releases.
+
+http://www.apache.org/dev/release-signing.html#basic-facts
+
+## Configure PyPI uploads
+
+In order to not reveal your password in plain text, it's best if you create and configure API Upload tokens.
+You can add and copy the tokens here:
+
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
+
+
+Create a `~/.pypirc` file:
+
+```ini
+[distutils]
+index-servers =
+ pypi
+ pypitest
+
+[pypi]
+username=__token__
+password=<API Upload Token>
+
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
+```
+
+Set proper permissions for the pypirc file:
+
+```shell script
+$ chmod 600 ~/.pypirc
+```
+
+- Install [twine](https://pypi.org/project/twine/) if you do not have it already (it can be done
+ in a separate virtual environment).
+
+```shell script
+pip install twine
+```
+
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
+
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
+
+- Confirm that `airflow/version.py` is set properly.
+
+
+## Hardware used to prepare and verify the packages
+
+The best way to prepare and verify the releases is to prepare them on a hardware owned and controlled
+by the committer acting as release manager. While strictly speaking, releases must only be verified
+on hardware owned and controlled by the committer, for practical reasons it's best if the packages are
+prepared using such hardware. More information can be found in this
+[FAQ](http://www.apache.org/legal/release-policy.html#owned-controlled-hardware)
+
+# Apache Airflow packages
+
+## Prepare the Apache Airflow Package RC
+
+### Build RC artifacts (both source packages and convenience packages)
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
+
+- Set environment variables
+
+ ```shell script
+ # Set Version
+ export VERSION=1.10.2rc3
+
+
+ # Set AIRFLOW_REPO_ROOT to the path of your git repo
+ export AIRFLOW_REPO_ROOT=$(pwd)
+
+
+ # Example after cloning
+ git clone https://github.com/apache/airflow.git airflow
+ cd airflow
+ export AIRFLOW_REPO_ROOT=$(pwd)
+ ```
+
+- Set your version to 1.10.2 in `airflow/version.py` (without the RC tag)
+- Commit the version change.
+
+- Tag your release
+
+ ```shell script
+ git tag -s ${VERSION}
+ ```
+
+- Clean the checkout: the sdist step below will
+
+ ```shell script
+ git clean -fxd
+ ```
+
+- Tarball the repo
+
+ ```shell script
+ git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz`
+ ```
+
+
+- Generate sdist
+
+ NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included
+ in the file produced.
+
+ ```shell script
+ python setup.py compile_assets sdist bdist_wheel
+ ```
+
+- Rename the sdist
+
+ ```shell script
+ mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
+ mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
+ ```
+
+- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
+
+ ```shell script
+ ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
+ ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
+ ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl
+ ```
+
+- Push Tags
+
+ ```shell script
+ git push origin ${VERSION}
+ ```
+
+- Push the artifacts to ASF dev dist repo
+```
+# First clone the repo
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# Create new folder for the release
+cd airflow-dev
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder & commit
+mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* ${VERSION}/
+cd ${VERSION}
+svn add *
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+```
+
+### Prepare PyPI convenience "snapshot" packages
+
+At this point we have the artefact that we vote on, but as a convenience to developers we also want to
+publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to
+
+- Edit the `airflow/version.py` to include the RC suffix.
+
+- Build the package:
+
+ ```shell script
+ python setup.py compile_assets sdist bdist_wheel
+ ```
+
+- Verify the artifacts that would be uploaded:
+
+ ```shell script
+ twine check dist/*
+ ```
+
+- Upload the package to PyPi's test environment:
+
+ ```shell script
+ twine upload -r pypitest dist/*
+ ```
+
+- Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
+https://test.pypi.org/project/apache-airflow/#files
+
+- Upload the package to PyPi's production environment:
+`twine upload -r pypi dist/*`
+
+- Again, confirm that the package is available here:
+https://pypi.python.org/pypi/apache-airflow
+
+- Throw away the change - we don't want to commit this: `git checkout airflow/version.py`
+
+It is important to stress that this snapshot should not be named "release", and it
+is not supposed to be used by and advertised to the end-users who do not read the devlist.
+
+## Vote and verify the Apache Airflow release candidate
+
+### Prepare Vote email on the Apache Airflow release candidate
+
+- Use the dev/airflow-jira script to generate a list of Airflow JIRAs that were closed in the release.
+
+- Send out a vote to the dev@airflow.apache.org mailing list:
+
+Subject:
+```
+[VOTE] Airflow 1.10.2rc3
+```
+
+Body:
+
+```
+Hey all,
+
+I have cut Airflow 1.10.2 RC3. This email is calling a vote on the release,
+which will last for 72 hours. Consider this my (binding) +1.
+
+Airflow 1.10.2 RC3 is available at:
+https://dist.apache.org/repos/dist/dev/airflow/1.10.2rc3/
+
+*apache-airflow-1.10.2rc3-source.tar.gz* is a source release that comes
+with INSTALL instructions.
+*apache-airflow-1.10.2rc3-bin.tar.gz* is the binary Python "sdist" release.
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Only votes from PMC members are binding, but the release manager should encourage members of the community
+to test the release and vote with "(non-binding)".
+
+The test procedure for PMCs and Contributors who would like to test this RC are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-apache-airflow-release-candidate
+
+Please note that the version number excludes the `rcX` string, so it's now
+simply 1.10.2. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+
+Changes since 1.10.2rc2:
+*Bugs*:
+[AIRFLOW-3732] Fix issue when trying to edit connection in RBAC UI
+[AIRFLOW-2866] Fix missing CSRF token head when using RBAC UI (#3804)
+...
+
+
+*Improvements*:
+[AIRFLOW-3302] Small CSS fixes (#4140)
+[Airflow-2766] Respect shared datetime across tabs
+...
+
+
+*New features*:
+[AIRFLOW-2874] Enables FAB's theme support (#3719)
+[AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)
+...
+
+
+*Doc-only Change*:
+[AIRFLOW-XXX] Fix BashOperator Docstring (#4052)
+[AIRFLOW-3018] Fix Minor issues in Documentation
+...
+
+Cheers,
+<your name>
+```
+
+### Verify the release candidate by PMCs (legal)
+
+#### PMC responsibilities
+
+The PMCs should verify the releases in order to make sure the release is following the
+[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+
+At least 3 (+1) votes should be recorded in accordance to
+[Votes on Package Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+
+The legal checks include:
+
+* checking if the packages are present in the right dist folder on svn
+* verifying if all the sources have correct licences
+* verifying if release manager signed the releases with the right key
+* verifying if all the checksums are valid for the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/)
+
+The following files should be present (9 files):
+
+* -bin-tar.gz + .asc + .sha512
+* -source.tar.gz + .asc + .sha512
+* -.whl + .asc + .sha512
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the release candidate "works" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+```shell script
+pip install apache-airflow==<VERSION>rc<X>
+```
+Optionally it can be followed with constraints
+
+```shell script
+pip install apache-airflow==<VERSION>rc<X> \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<VERSION>/constraints-3.6.txt"`
+```
+
+Note that the constraints contain python version that you are installing it with.
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheel
+downloaded from the SVN).
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Running the following command will use tmux inside breeze, create `admin` user and run Webserver & Scheduler:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> --python 3.7 --backend postgres --no-rbac-ui
+```
+
+Once you install and run Airflow, you should perform any verification you see as necessary to check
+that the Airflow works as you expected.
+
+## Publish the final Apache Airflow release
+
+### Summarize the voting for the Apache Airflow release
+
+Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:
+
+Subject:
+```
+[RESULT][VOTE] Airflow 1.10.2rc3
+```
+
+Message:
+
+```
+Hello,
+
+Apache Airflow 1.10.2 (based on RC3) has been accepted.
+
+4 “+1” binding votes received:
+- Kaxil Naik (binding)
+- Bolke de Bruin (binding)
+- Ash Berlin-Taylor (binding)
+- Tao Feng (binding)
+
+
+4 "+1" non-binding votes received:
+
+- Deng Xiaodong (non-binding)
+- Stefan Seelmann (non-binding)
+- Joshua Patchus (non-binding)
+- Felix Uellendall (non-binding)
+
+Vote thread:
+https://lists.apache.org/thread.html/736404ca3d2b2143b296d0910630b9bd0f8b56a0c54e3a05f4c8b5fe@%3Cdev.airflow.apache.org%3E
+
+I'll continue with the release process, and the release announcement will follow shortly.
+
+Cheers,
+<your name>
+```
+
+
+### Publish release to SVN
+
+You need to migrate the RC artifacts that passed to this repository:
+https://dist.apache.org/repos/dist/release/airflow/
+(The migration should include renaming the files so that they no longer have the RC number in their filenames.)
+
+The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries again, and gives a clearer history in the svn commit logs):
+
+```shell script
+# First clone the repo
+export RC=1.10.4rc5
+export VERSION=${RC/rc?/}
+svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
+
+# Create new folder for the release
+cd airflow-release
+svn mkdir ${VERSION}
+cd ${VERSION}
+
+# Move the artifacts to svn folder & commit
+for f in ../../airflow-dev/$RC/*; do svn cp $f ${$(basename $f)/rc?/}; done
+svn commit -m "Release Airflow ${VERSION} from ${RC}"
+
+# Remove old release
+# http://www.apache.org/legal/release-policy.html#when-to-archive
+cd ..
+export PREVIOUS_VERSION=1.10.1
+svn rm ${PREVIOUS_VERSION}
+svn commit -m "Remove old release: ${PREVIOUS_VERSION}"
+```
+
+Verify that the packages appear in [airflow](https://dist.apache.org/repos/dist/release/airflow/)
+
+### Prepare PyPI "release" packages
+
+At this point we release an official package:
+
+- Build the package:
+
+ ```shell script
+ python setup.py compile_assets sdist bdist_wheel`
+ ```
+
+- Verify the artifacts that would be uploaded:
+
+ ```shell script
+ twine check dist/*`
+ ```
+
+- Upload the package to PyPi's test environment:
+
+ ```shell script
+ twine upload -r pypitest dist/*
+ ```
+
+- Verify that the test package looks good by downloading it and installing it into a virtual environment.
+ The package download link is available at: https://test.pypi.org/project/apache-airflow/#files
+
+- Upload the package to PyPi's production environment:
+
+ ```shell script
+ twine upload -r pypi dist/*
+ ```
+
+- Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow
+
+### Update CHANGELOG.md
+
+- Get a diff between the last version and the current version:
+
+ ```shell script
+ $ git log 1.8.0..1.9.0 --pretty=oneline
+ ```
+- Update CHANGELOG.md with the details, and commit it.
+
+### Notify developers of release
+
+- Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that
+the artifacts have been published:
+
+Subject:
+```shell script
+cat <<EOF
+Airflow ${VERSION} is released
+EOF
+```
+
+Body:
+```shell script
+cat <<EOF
+Dear Airflow community,
+
+I'm happy to announce that Airflow ${VERSION} was just released.
+
+The source release, as well as the binary "sdist" release, are available
+here:
+
+https://dist.apache.org/repos/dist/release/airflow/${VERSION}/
+
+We also made this version available on PyPi for convenience (`pip install apache-airflow`):
+
+https://pypi.python.org/pypi/apache-airflow
+
+The documentation is available on:
+https://airflow.apache.org/
+https://airflow.apache.org/1.10.2/
+https://airflow.readthedocs.io/en/1.10.2/
+https://airflow.readthedocs.io/en/stable/
+
+Find the CHANGELOG here for more details:
+
+https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
+
+Cheers,
+<your name>
+EOF
+```
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+
+-----------------------------------------------------------------------------------------------------------
+
+
+# Provider Packages
+
+You can read more about the command line tools used to generate the packages and the two types of
+packages we have (Backport and Regular Provider Packages) in [Provider packages](PROVIDER_PACKAGES.md).
+
+## Decide when to release
+
+You can release provider packages separately from the main Airflow on an ad-hoc basis, whenever we find that
+a given provider needs to be released - due to new features or due to bug fixes.
+You can release each provider package separately, but due to voting and release overhead we try to group
+releases of provider packages together.
+
+### Backport provider packages versioning
+
+We are using the [CALVER](https://calver.org/) versioning scheme for the backport packages. We also have an
+automated way to prepare and build the packages, so it should be very easy to release the packages often and
+separately. Backport packages will be maintained for three months after 2.0.0 version of Airflow, and it is
+really a bridge, allowing people to migrate to Airflow 2.0 in stages, so the overhead of maintaining
+semver versioning does not apply there - subsequent releases might be backward-incompatible, and it is
+not indicated by the version of the packages.
+
+### Regular provider packages versioning
+
+We are using the [SEMVER](https://semver.org/) versioning scheme for the regular packages. This is in order
+to give the users confidence about maintaining backwards compatibility in the new releases of those
+packages.
+
+Details about maintaining the SEMVER version are going to be discussed and implemented in
+[the related issue](https://github.com/apache/airflow/issues/11425)
+
+## Prepare the Backport Provider Packages RC
+
+### Generate release notes
+
+Prepare release notes for all the packages you plan to release. Where YYYY.MM.DD is the CALVER
+date for the packages.
+
+```shell script
+./breeze --backports prepare-provider-readme YYYY.MM.DD [packages]
+```
+
+If you iterate with merges and release candidates you can update the release date without providing
+the date (to update the existing release notes)
+
+```shell script
+./breeze --backports prepare-provider-readme google
+```
+
+Generated readme files should be eventually committed to the repository.
+
+### Build an RC release for SVN apache upload
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote against, without any
+modification than renaming i.e. the contents of the files must be the same between voted
+release candidate and final release. Because of this the version in the built artifacts
+that will become the official Apache releases must not include the rcN suffix. They also need
+to be signed and have checksum files. You can generate the checksum/signature files by running
+the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
+generates corresponding .asc and .sha512 files for each file to sign.
+
+#### Build and sign the source and convenience packages
+
+* Set environment variables (version and root of airflow repo)
+
+```shell script
+export VERSION=2020.5.20rc2
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+```
+
+* Build the source package:
+
+```shell script
+./provider_packages/build_source_package.sh --backports
+```
+
+It will generate `apache-airflow-backport-providers-${VERSION}-source.tar.gz`
+
+* Generate the packages - since we are preparing packages for SVN repo, we should use the right switch. Note
+ that this will clean up dist folder before generating the packages, so it will only contain the packages
+ you intended to build.
+
+```shell script
+./breeze --backports prepare-provider-packages --version-suffix-for-svn rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze --backports prepare-provider-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
+```
+
+* Move the source tarball to dist folder
+
+```shell script
+mv apache-airflow-backport-providers-${VERSION}-source.tar.gz dist
+```
+
+* Sign all your packages
+
+```shell script
+pushd dist
+../dev/sign.sh *
+popd
+```
+
+* Push tags to Apache repository (assuming that you have apache remote pointing to apache/airflow repo)]
+
+```shell script
+git push apache backport-providers-${VERSION}
+```
+
+#### Commit the source packages to Apache SVN repo
+
+* Push the artifacts to ASF dev dist repo
+
+```shell script
+# First clone the repo if you do not have it
+svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
+
+# update the repo in case you have it already
+cd airflow-dev
+svn update
+
+# Create a new folder for the release.
+cd airflow-dev/backport-providers
+svn mkdir ${VERSION}
+
+# Move the artifacts to svn folder
+mv ${AIRFLOW_REPO_ROOT}/dist/* ${VERSION}/
+
+# Add and commit
+svn add ${VERSION}/*
+svn commit -m "Add artifacts for Airflow ${VERSION}"
+
+cd ${AIRFLOW_REPO_ROOT}
+```
+
+Verify that the files are available at
+[backport-providers](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+### Publish the RC convenience package to PyPI
-<!-- START doctoc generated TOC please keep comment here to allow auto update -->
-<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-**Table of contents**
+In order to publish to PyPI you just need to build and release packages. The packages should however
+contain the rcN suffix in the version name as well, so you need to use `--version-suffix-for-pypi` switch
+to prepare those packages. Note that these are different packages than the ones used for SVN upload
+though they should be generated from the same sources.
-- [Airflow Jira utility](#airflow-jira-utility)
-- [Airflow Pull Request Tool](#airflow-pull-request-tool)
-- [Airflow release signing tool](#airflow-release-signing-tool)
+* Generate the packages with the right RC version (specify the version suffix with PyPI switch). Note that
+this will clean up dist folder before generating the packages, so you will only have the right packages there.
-<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+```shell script
+./breeze --backports prepare-provider-packages --version-suffix-for-pypi rc1
+```
+
+if you ony build few packages, run:
+
+```shell script
+./breeze --backports prepare-provider-packages --version-suffix-for-pypi rc1 PACKAGE PACKAGE ....
+```
+
+* Verify the artifacts that would be uploaded:
+
+```shell script
+twine check dist/*
+```
+
+* Upload the package to PyPi's test environment:
+
+```shell script
+twine upload -r pypitest dist/*
+```
+
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
+
+```shell script
+twine upload -r pypi dist/*
+```
+
+* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
+
+* Again, confirm that the packages are available under the links printed.
+
+## Vote and verify the Backport Providers release candidate
+
+### Prepare voting email for Backport Providers release candidate
+
+Make sure the packages are in https://dist.apache.org/repos/dist/dev/airflow/backport-providers/
+
+Send out a vote to the dev@airflow.apache.org mailing list. Here you can prepare text of the
+email using the ${VERSION} variable you already set in the command line.
+
+subject:
+
+
+```shell script
+cat <<EOF
+[VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+```shell script
+cat <<EOF
+Hey all,
+
+I have cut Airflow Backport Providers ${VERSION}. This email is calling a vote on the release,
+which will last for 72 hours - which means that it will end on $(date -d '+3 days').
+
+Consider this my (binding) +1.
+
+Airflow Backport Providers ${VERSION} are available at:
+https://dist.apache.org/repos/dist/dev/airflow/backport-providers/${VERSION}/
+
+*apache-airflow-backport-providers-${VERSION}-source.tar.gz* is a source release that comes
+ with INSTALL instructions.
+
+*apache-airflow-backport-providers-<PROVIDER>-${VERSION}-bin.tar.gz* are the binary
+ Python "sdist" release.
+
+The test procedure for PMCs and Contributors who would like to test the RC candidates are described in
+https://github.com/apache/airflow/blob/master/dev/README.md#vote-and-verify-the-backport-providers-release-candidate
+
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/airflow/KEYS
+
+Please vote accordingly:
+
+[ ] +1 approve
+[ ] +0 no opinion
+[ ] -1 disapprove with the reason
+
+
+Only votes from PMC members are binding, but members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+Please note that the version number excludes the 'rcX' string, so it's now
+simply ${VERSION%rc?}. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+Each of the packages contains detailed changelog. Here is the list of links to
+the released packages and changelogs:
+
+<PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!>
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+```
+
+Due to the nature of backport packages, not all packages have to be released as convenience
+packages in the final release. During the voting process
+the voting PMCs might decide to exclude certain packages from the release if some critical
+problems have been found in some packages.
+
+Please modify the message above accordingly to clearly exclude those packages.
+
+### Verify the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Airflow dist](https://dist.apache.org/repos/dist/dev/airflow/backport-providers/)
+
+The following files should be present (9 files):
+
+* -source.tar.gz + .asc + .sha512 (one set of files)
+* -bin-tar.gz + .asc + .sha512 (one set of files per provider)
+* -.whl + .asc + .sha512 (one set of files per provider)
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+svn co https://dist.apache.org/repos/dist/dev/airflow/
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the sources,
+ the jar is inside)
+* Unpack the -source.tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from ..." is indication
+that the signatures are correct. Do not worry about the "not certified with a trusted signature"
+warning. Most of the certificates used by release managers are self signed, that's why you get this
+warning. By importing the server in the previous step and importing it via ID from
+[KEYS](https://dist.apache.org/repos/dist/release/airflow/KEYS) page, you know that
+this is a valid Key already.
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_airflow-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-airflow-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-airflow-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <ka...@gmail.com>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff - $i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache-airflow-1.10.12rc4-bin.tar.gz.sha512
+Checking apache_airflow-1.10.12rc4-py2.py3-none-any.whl.sha512
+Checking apache-airflow-1.10.12rc4-source.tar.gz.sha512
+```
+
+### Verify if the Backport Packages release candidates "work" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
+actual users of Apache Airflow test it in their own staging/test installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to install
+the release candidate version of Airflow via simply (<VERSION> is 1.10.12 for example, and <X> is
+release candidate number 1,2,3,....).
+
+You can use any of the installation methods you prefer (you can even install it via the binary wheels
+downloaded from the SVN).
+
+
+#### Installing in your local virtualenv
+
+You have to make sure you have Airilow 1.10.* installed in your PIP virtualenv
+(the version you want to install providers with).
+
+```shell script
+pip install apache-airflow-backport-providers-<provider>==<VERSION>rc<X>
+```
+
+#### Installing with Breeze
+
+There is also an easy way of installation with Breeze if you have the latest sources of Apache Airflow.
+Here is a typical scenario.
+
+First copy all the provider packages .whl files to the `dist` folder.
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels
+```
+
+For 1.10 releases you can also use `--no-rbac-ui` flag disable RBAC UI of Airflow:
+
+```shell script
+./breeze start-airflow --install-airflow-version <VERSION>rc<X> \
+ --python 3.7 --backend postgres --install-wheels --no-rbac-ui
+```
+
+#### Building your own docker image
+
+If you prefer to build your own image, you can also use the official image and PyPI packages to test
+backport packages. This is especially helpful when you want to test integrations, but you need to install
+additional tools. Below is an example Dockerfile, which installs backport providers for Google and
+an additional third-party tools:
+
+```dockerfile
+FROM apache/airflow:1.10.12
+
+RUN pip install --user apache-airflow-backport-providers-google==2020.10.5.rc1
+
+RUN curl https://sdk.cloud.google.com | bash \
+ && echo "source /home/airflow/google-cloud-sdk/path.bash.inc" >> /home/airflow/.bashrc \
+ && echo "source /home/airflow/google-cloud-sdk/completion.bash.inc" >> /home/airflow/.bashrc
+
+USER 0
+RUN KUBECTL_VERSION="$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)" \
+ && KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
+ && curl -L "${KUBECTL_URL}" --output /usr/local/bin/kubectl \
+ && chmod +x /usr/local/bin/kubectl
+
+USER ${AIRFLOW_UID}
+```
+
+To build an image build and run a shell, run:
+
+```shell script
+docker build . -t my-airflow
+docker run -ti \
+ --rm \
+ -v "$PWD/data:/opt/airflow/" \
+ -v "$PWD/keys/:/keys/" \
+ -p 8080:8080 \
+ -e GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json \
+ -e AIRFLOW__CORE__LOAD_EXAMPLES=True \
+ my-airflow bash
+```
+
+#### Verification
+
+Once you install and run Airflow, you can perform any verification you see as necessary to check
+that the Airflow works as you expected.
+
+## Publish the final releases of backport packages
+
+### Summarize the voting for the Backport Providers Release
+
+Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:
+
+Subject:
+```shell script
+cat <<EOF
+[RESULT][VOTE] Airflow Backport Providers ${VERSION}
+EOF
+```
+
+Body:
+
+```shell script
+cat <<EOF
+
+Hey all,
+
+Airflow Backport Providers ${VERSION} (based on the ${VERSION_RC} candidate) has been accepted.
+
+N "+1" binding votes received:
+- PMC Member (binding)
+...
+
+N "+1" non-binding votes received:
+
+- COMMITER (non-binding)
+
+Vote thread:
+https://lists.apache.org/thread.html/<TODO:REPLACE_ME_WITH_THE_VOTING_THREAD>@%3Cdev.airflow.apache.org%3E
+
+I'll continue with the release process and the release announcement will follow shortly.
+
+Cheers,
+<TODO: Your Name>
+
+EOF
+
+```
+
+### Publish release to SVN
+
+The best way of doing this is to svn cp between the two repos (this avoids having to upload the binaries
+again, and gives a clearer history in the svn commit logs.
+
+We also need to archive older releases before copying the new ones
+[Release policy](http://www.apache.org/legal/release-policy.html#when-to-archive)
+
+```shell script
+# Set the variables
+export VERSION_RC=2020.5.20rc2
+export VERSION=${VERSION_RC/rc?/}
+
+# Set AIRFLOW_REPO_ROOT to the path of your git repo
+export AIRFLOW_REPO_ROOT=$(pwd)
+
+# Go to the directory where you have checked out the dev svn release
+# And go to the sub-folder with RC candidates
+cd "<ROOT_OF_YOUR_DEV_REPO>/backport-providers/${VERSION_RC}"
+export SOURCE_DIR=$(pwd)
+
+# Go the folder where you have checked out the release repo
+# Clone it if it's not done yet
+svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
+
+# Update to latest version
+svn update
+
+# Create backport-providers folder if it does not exist
+# All latest releases are kept in this one folder without version sub-folder
+mkdir -pv backport-providers
+cd backport-providers
+
+# Move the artifacts to svn folder & remove the rc postfix
+for file in ${SOURCE_DIR}/*${VERSION_RC}*
+do
+ base_file=$(basename ${file})
+ svn cp "${file}" "${base_file/${VERSION_RC}/${VERSION}}"
+done
+
+
+# If some packages have been excluded, remove them now
+# Check the packages
+ls *<provider>*
+# Remove them
+svn rm *<provider>*
+
+# Check which old packages will be removed (you need python 3.6+)
+python ${AIRFLOW_REPO_ROOT}/provider_packages/remove_old_releases.py \
+ --directory .
+
+# Remove those packages
+python ${AIRFLOW_REPO_ROOT}/provider_packages/remove_old_releases.py \
+ --directory . --execute
+
+
+# Commit to SVN
+svn commit -m "Release Airflow Backport Providers ${VERSION} from ${VERSION_RC}"
+```
+
+Verify that the packages appear in
+[backport-providers](https://dist.apache.org/repos/dist/release/airflow/backport-providers)
-## Airflow Jira utility
+### Publish the final version convenience package to PyPI
-The `airflow-jira` script interact with the Airflow project in <https://issues.apache.org/jira/>. There are two modes of operation
+Checkout the RC Version:
+
+```shell script
+git checkout backport-providers-${VERSION_RC}
+```
+Tag and push the final version (providing that your apache remote is named 'apache'):
-- `compare` will examine issues in Jira based on the "Fix Version" field.
+```shell script
+git tag backport-providers-${VERSION}
+git push apache backport-providers-${VERSION}
+```
- This is useful for preparing releases, and also has an `--unmerged` flag to
- only show issues that aren't detected in the current branch.
+In order to publish to PyPI you just need to build and release packages.
- To run this check out the release branch (for instance `v1-10-test`) and run:
+* Generate the packages.
- ```
- ./dev/airflow-jira compare --unmerged --previous-version 1.10.6 1.10.7
- ```
+```shell script
+./breeze --backports prepare-provider-packages
+```
- The `--previous-version` is optional, but might speed up operation. That
- should be a tag reachable from the current HEAD, and will limit the script to
- look for cherry-picks in the commit range `$PREV_VERSION..HEAD`
+if you ony build few packages, run:
-- `changelog` will create a _rough_ output for creating the changelog file for a release
+```shell script
+./breeze --backports prepare-provider-packages <PACKAGE> ...
+```
- This output will not be perfect and will need manual processing to make sure
- the descriptions make sense, and that the items are in the right section (for
- instance you might want to create 'Doc-only' and 'Misc/Internal' section.)
+In case you decided to remove some of the packages. remove them from dist folder now:
-## Airflow Pull Request Tool
+```shell script
+ls dist/*<provider>*
+rm dist/*<provider>*
+```
-The `airflow-pr` tool interactively guides committers through the process of merging GitHub PRs into Airflow and closing associated JIRA issues.
-It is very important that PRs reference a JIRA issue. The preferred way to do that is for the PR title to begin with [AIRFLOW-XXX]. However, the PR tool can recognize and parse many other JIRA issue formats in the title and will offer to correct them if possible.
+* Verify the artifacts that would be uploaded:
-__Please note:__ this tool will restore your current branch when it finishes, but you will lose any uncommitted changes. Make sure you commit any changes you wish to keep before proceeding.
+```shell script
+twine check dist/*
+```
+* Upload the package to PyPi's test environment:
-### Execution
-Simply execute the `airflow-pr` tool:
+```shell script
+twine upload -r pypitest dist/*
```
-$ ./airflow-pr
-Usage: airflow-pr [OPTIONS] COMMAND [ARGS]...
- This tool should be used by Airflow committers to test PRs, merge them
- into the master branch, and close related JIRA issues.
+* Verify that the test packages look good by downloading it and installing them into a virtual environment.
+Twine prints the package links as output - separately for each package.
+
+* Upload the package to PyPi's production environment:
- Before you begin, make sure you have created the 'apache' and 'github' git
- remotes. You can use the "setup_git_remotes" command to do this
- automatically. If you do not want to use these remote names, you can tell
- the PR tool by setting the appropriate environment variables. For more
- information, run:
+```shell script
+twine upload -r pypi dist/*
+```
- airflow-pr merge --help
+### Notify developers of release
-Options:
- --help Show this message and exit.
+- Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that
+the artifacts have been published:
-Commands:
- close_jira Close a JIRA issue (without merging a PR)
- merge Merge a GitHub PR into Airflow master
- setup_git_remotes Set up default git remotes
- work_local Clone a GitHub PR locally for testing (no push)
+Subject:
+```shell script
+cat <<EOF
+Airflow Backport Providers ${VERSION} are released
+EOF
```
-#### Commands
+Body:
+```shell script
+cat <<EOF
+Dear Airflow community,
+
+I'm happy to announce that Airflow Backport Providers packages ${VERSION} were just released.
+
+The source release, as well as the binary releases, are available here:
-Execute `airflow-pr merge` to be interactively guided through the process of merging a PR, pushing changes to master, and closing JIRA issues.
+https://dist.apache.org/repos/dist/release/airflow/backport-providers/
-Execute `airflow-pr work_local` to only merge the PR locally. The tool will pause once the merge is complete, allowing the user to explore the PR, and then will delete the merge and restore the original development environment.
+We also made those versions available on PyPi for convenience ('pip install apache-airflow-backport-providers-*'):
-Execute `airflow-pr close_jira` to close a JIRA issue without needing to merge a PR. You will be prompted for an issue number and close comment.
+https://pypi.org/search/?q=apache-airflow-backport-providers
-Execute `airflow-pr setup_git_remotes` to configure the default (expected) git remotes. See below for details.
+The documentation and changelogs are available in the PyPI packages:
-### Configuration
+<PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!>
-#### Python Libraries
-The merge tool requires the `click` and `jira` libraries to be installed. If the libraries are not found, the user will be prompted to install them:
-```bash
-pip install click jira
+
+Cheers,
+<your name>
+EOF
```
-#### git Remotes
-tl;dr run `airflow-pr setup_git_remotes` before using the tool for the first time.
-Before using the merge tool, users need to make sure their git remotes are configured. By default, the tool assumes a setup like the one below, where the github repo remote is named `github`. If users have other remote names, they can be supplied by setting environment variables `GITHUB_REMOTE_NAME`.
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
+
+-----------------------------------------------------------------------------------------------------------
+
+## Prepare the Regular Provider Packages Alpha
-Users can configure this automatically by running `airflow-pr setup_git_remotes`.
+### Generate release notes
-```bash
-$ git remote -v
-github https://github.com/apache/airflow.git (fetch)
-github https://github.com/apache/airflow.git (push)
-origin https://github.com/<USER>/airflow (fetch)
-origin https://github.com/<USER>/airflow (push)
+Prepare release notes for all the packages you plan to release. Note that for now version number is
+hard-coded to 0.0.1 for all packages. Later on we are going to update the versions according
+to SEMVER versioning.
+
+Details about maintaining the SEMVER version are going to be discussed and implemented in
+[the related issue](https://github.com/apache/airflow/issues/11425)
+
+
+```shell script
+./breeze prepare-provider-readme [packages]
```
You can iterate and re-generate the same readme content as many times as you want.
@@ -138,11 +1468,11 @@ They also need to be signed and have checksum files. You can generate the checks
the "dev/sign.sh" script (assuming you have the right PGP key set-up for signing). The script
generates corresponding .asc and .sha512 files for each file to sign.
-## Airflow release signing tool
-The release signing tool can be used to create the SHA512/MD5 and ASC files that required for Apache releases.
+#### Build and sign the source and convenience packages
-### Execution
-To create a release tar ball execute following command from Airflow's root.
+Currently, we are releasing alpha provider packages together with the main sources of Airflow. In the future
+we are going to add procedure to release the sources of released provider packages separately.
+Details are in [the related issue](https://github.com/apache/airflow/issues/11425)
For alpha/beta releases you need to specify both - svn and pyp i - suffixes, and they have to match. This is
verified by the breeze script. Note that the script will clean up dist folder before generating the
@@ -165,14 +1495,13 @@ if you ony build few packages, run:
* Release candidate packages:
-*Note: `compile_assets` command build the frontend assets (JS and CSS) files for the
-Web UI using webpack and yarn. Please make sure you have `yarn` installed on your local machine globally.
-Details on how to install `yarn` can be found in CONTRIBUTING.rst file.*
+```shell script
+export VERSION=0.0.1alpha1
./breeze prepare-provider-packages --version-suffix-for-svn rc1
```
-`../dev/sign.sh <the_created_tar_ball.tar.gz`
+if you ony build few packages, run:
```shell script
./breeze prepare-provider-packages --version-suffix-for-svn rc1 PACKAGE PACKAGE ....
@@ -262,6 +1591,45 @@ Twine prints the package links as output - separately for each package.
twine upload -r pypi dist/*
```
-* Copy the list of links to the uploaded packages - they will be useful in preparing VOTE email.
-
* Again, confirm that the packages are available under the links printed.
+
+### Notify developers of release
+
+- Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that
+the artifacts have been published:
+
+Subject:
+```shell script
+cat <<EOF
+Airflow Providers are released
+EOF
+```
+
+Body:
+```shell script
+cat <<EOF
+Dear Airflow community,
+
+I'm happy to announce that new version of Airflow Providers packages were just released.
+
+The source release, as well as the binary releases, are available here:
+
+https://dist.apache.org/repos/dist/release/airflow/providers/
+
+We also made those versions available on PyPi for convenience ('pip install apache-airflow-providers-*'):
+
+https://pypi.org/search/?q=apache-airflow-providers
+
+The documentation and changelogs are available in the PyPI packages:
+
+<PASTE TWINE UPLOAD LINKS HERE. SORT THEM BEFORE!>
+
+Cheers,
+<your name>
+EOF
+```
+
+
+### Update Announcements page
+
+Update "Announcements" page at the [Official Airflow website](https://airflow.apache.org/announcements/)
[airflow] 18/44: Adds a forgotten word in a README.md (#12066)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 26d001fad24bcb2cbd5ab8d695f52ce713389691
Author: Damian Kula <He...@gmail.com>
AuthorDate: Tue Nov 3 14:31:19 2020 +0100
Adds a forgotten word in a README.md (#12066)
(cherry picked from commit d8aa24b729de4e80cd2d71f99d04cbabf5f3b36c)
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 70bf2d8..ad83639 100644
--- a/README.md
+++ b/README.md
@@ -80,11 +80,11 @@ Apache Airflow is tested with:
| ------------ | ------------------------- | ------------------------ |
| Python | 3.6, 3.7, 3.8 | 2.7, 3.5, 3.6, 3.7, 3.8 |
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
-| MySQL | 5.7 | 5.6, 5.7 |
+| MySQL | 5.7, 8 | 5.6, 5.7 |
| SQLite | latest stable | latest stable |
| Kubernetes | 1.16.2, 1.17.0 | 1.16.2, 1.17.0 |
-> Note: SQLite is used primarily for development purpose.
+**Note:** SQLite is used primarily for development purpose.
### Additional notes on Python version requirements
[airflow] 29/44: Uses always the same Python base image as used for
CI image (#12177)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a27701a396f803e6c5434f2c2062393efa55f62a
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 8 11:20:31 2020 +0100
Uses always the same Python base image as used for CI image (#12177)
When new Python version is released (bugfixes), we rebuild the CI image
and replace it with the new one, however releasing of the python
image and CI image is often hours or even days apart (we only
release the CI image when tests pass in master with the new python
image). We already use a better approach for Github - we simply
push the new python image to our registry together with the CI
image and the CI jobs are always pulling them from our registry
knowing that the two - python and CI image are in sync.
This PR introduces the same approach. We not only push CI image
but also the corresponding Python image to our registry. This has
no ill effect - DockerHub handles it automatically and reuses
the layers of the image directly from the Python one so it is
merely a label that is stored in our registry that points to the
exact Python image that was used by the last pushed CI image.
(cherry picked from commit 75bdfaeb9b2d7f47599e951ccaccc515a180ca19)
---
IMAGES.rst | 11 ++++++++
scripts/ci/libraries/_build_images.sh | 11 +++-----
scripts/ci/libraries/_push_pull_remove_images.sh | 35 +++---------------------
3 files changed, 19 insertions(+), 38 deletions(-)
diff --git a/IMAGES.rst b/IMAGES.rst
index 45aa09ab..8c913db 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -46,6 +46,17 @@ where:
* The ``-ci`` suffix is added for CI images
* The ``-manifest`` is added for manifest images (see below for explanation of manifest images)
+We also store (to increase speed of local build/pulls) python images that were used to build
+the CI images. Each CI image, when built uses current python version of the base images. Those
+python images are regularly updated (with bugfixes/security fixes), so for example python3.8 from
+last week might be a different image than python3.8 today. Therefore whenever we push CI image
+to airflow repository, we also push the python image that was used to build it this image is stored
+as ``apache/airflow:python-3.8-<BRANCH_OR_TAG>``.
+
+Since those are simply snapshots of the existing python images, DockerHub does not create a separate
+copy of those images - all layers are mounted from the original python images and those are merely
+labels pointing to those.
+
Building docker images
======================
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index cbd01c1..a2577cb 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -330,9 +330,11 @@ function build_images::get_docker_image_names() {
export AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-
# Default CI image
- export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
+ export AIRFLOW_CI_PYTHON_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:python${PYTHON_MAJOR_MINOR_VERSION}-${BRANCH_NAME}"
+ # CI image to build
+ export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
+
# Base production image tag - used to build kubernetes tag as well
if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
@@ -374,11 +376,6 @@ function build_images::prepare_ci_build() {
export GITHUB_REGISTRY_AIRFLOW_CI_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/${AIRFLOW_CI_BASE_TAG}"
export GITHUB_REGISTRY_PYTHON_BASE_IMAGE="${GITHUB_REGISTRY}/${github_repository_lowercase}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
fi
- if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == "${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
- export DEFAULT_PROD_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
- else
- export DEFAULT_PROD_IMAGE=""
- fi
export THE_IMAGE_TYPE="CI"
export IMAGE_DESCRIPTION="Airflow CI"
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index fae0807..e853f42 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -104,7 +104,6 @@ function push_pull_remove_images::pull_image_github_dockerhub() {
# Pulls CI image in case caching strategy is "pulled" and the image needs to be pulled
function push_pull_remove_images::pull_ci_images_if_needed() {
-
if [[ "${DOCKER_CACHE}" == "pulled" ]]; then
if [[ "${FORCE_PULL_IMAGES}" == "true" ]]; then
echo
@@ -122,7 +121,7 @@ Docker pulling ${PYTHON_BASE_IMAGE}.
fi
push_pull_remove_images::pull_image_github_dockerhub "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
- docker pull "${PYTHON_BASE_IMAGE}"
+ docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
fi
echo
fi
@@ -174,6 +173,9 @@ function push_pull_remove_images::push_ci_images_to_dockerhub() {
# Only push default image to DockerHub registry if it is defined
push_pull_remove_images::push_image_with_retries "${DEFAULT_CI_IMAGE}"
fi
+ # Also push python image so that we use the same image as the CI image it was built with
+ docker tag "${PYTHON_BASE_IMAGE}" "${AIRFLOW_CI_PYTHON_IMAGE}"
+ push_pull_remove_images::push_image_with_retries "${AIRFLOW_CI_PYTHON_IMAGE}"
}
# Pushes Ci images and their tags to registry in GitHub
@@ -196,21 +198,7 @@ function push_pull_remove_images::push_ci_images_to_github() {
PYTHON_TAG_SUFFIX="-${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
fi
docker tag "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
- set +e
push_pull_remove_images::push_image_with_retries "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
- local result=$?
- set -e
- if [[ ${result} != "0" ]]; then
- >&2 echo
- >&2 echo "There was an unexpected error when pushing images to the GitHub Registry"
- >&2 echo
- >&2 echo "If you see 'unknown blob' or similar kind of error it means that it was a transient error"
- >&2 echo "And it will likely be gone next time"
- >&2 echo
- >&2 echo "Please rebase your change or 'git commit --amend; git push --force' and try again"
- >&2 echo
- exit "${result}"
- fi
}
@@ -253,22 +241,7 @@ function push_pull_remove_images::push_prod_images_to_github () {
# Also push prod build image
AIRFLOW_PROD_BUILD_TAGGED_IMAGE="${GITHUB_REGISTRY_AIRFLOW_PROD_BUILD_IMAGE}:${GITHUB_REGISTRY_PUSH_IMAGE_TAG}"
docker tag "${AIRFLOW_PROD_BUILD_IMAGE}" "${AIRFLOW_PROD_BUILD_TAGGED_IMAGE}"
- set +e
push_pull_remove_images::push_image_with_retries "${AIRFLOW_PROD_BUILD_TAGGED_IMAGE}"
- local result=$?
- set -e
- if [[ ${result} != "0" ]]; then
- >&2 echo
- >&2 echo "There was an unexpected error when pushing images to the GitHub Registry"
- >&2 echo
- >&2 echo "If you see 'unknown blob' or similar kind of error it means that it was a transient error"
- >&2 echo "And it will likely be gone next time"
- >&2 echo
- >&2 echo "Please rebase your change or 'git commit --amend; git push --force' and try again"
- >&2 echo
- exit "${result}"
- fi
-
}
[airflow] 28/44: Fixes "--force-clean-images" flag in Breeze
(#12156)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 815f18b06c8711208971d9766350856d892b695c
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Nov 7 13:51:01 2020 +0100
Fixes "--force-clean-images" flag in Breeze (#12156)
The flag was broken - bad cache parameter value was passed.
This PR fixes it.
(cherry picked from commit 5c601578194739286097a75f082af2d83539581f)
---
breeze | 2 +-
scripts/ci/libraries/_build_images.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/breeze b/breeze
index 7838c88..1d8ffea 100755
--- a/breeze
+++ b/breeze
@@ -884,7 +884,7 @@ function breeze::parse_arguments() {
-C | --force-clean-images)
echo "Clean build of images without cache"
echo
- export DOCKER_CACHE="no-cache"
+ export DOCKER_CACHE="disabled"
# if not set here, docker cached is determined later, depending on type of image to be build
readonly DOCKER_CACHE
export FORCE_BUILD_IMAGES="true"
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index a6dbae0..cbd01c1 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -536,7 +536,7 @@ function build_images::build_ci_image() {
)
else
echo >&2
- echo >&2 "Error - thee ${DOCKER_CACHE} cache is unknown!"
+ echo >&2 "Error - the ${DOCKER_CACHE} cache is unknown!"
echo >&2
exit 1
fi
[airflow] 13/44: Fix canceling of CodeQL workflow (#12024)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df68fb7cd84626abe5675443f5254649147b446b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 1 22:05:45 2020 +0100
Fix canceling of CodeQL workflow (#12024)
The previous update for 4.3 version of the action also broke
CodeQL cancelling. This PR fixes it.
(cherry picked from commit 5f9792cee26c729f319e57fd09538fa6703929b4)
---
.github/workflows/build-images-workflow-run.yml | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 752e577..6099727 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -82,7 +82,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately
# and then we try to find and cancel all the jobs with the same Event + Repo + Branch as the
# current Event/Repo/Branch combination.
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -98,7 +98,7 @@ jobs:
# We also produce list of canceled "CI Build' runs as output, so that we
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -131,14 +131,14 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.cancel-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -167,7 +167,7 @@ jobs:
# trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then we try to match
# all the jobs with the same Event + Repo + Branch match and cancel all the duplicates for those
# This might cancel own run, so this is the last step in the job
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
cancelMode: allDuplicatedNamedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -382,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -397,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@f4a33154219b13dbb1e171695d6f03810f3a7b47 # v4_6
+ uses: potiuk/cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738 # v4_7
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 25/44: Work properly if some variables are not defined
(#12135)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 38d55820057aa4ae9f3750a819435b2193406e6e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Nov 6 16:56:43 2020 +0100
Work properly if some variables are not defined (#12135)
Those variables are defined in GitHub environment so when they
were recently addded it was not obvious that they will fail when
running kubernetes tests locally.
This PR fixes that.
(cherry picked from commit 5351f0d99605738b5efa0134bb51eb2dbb6eba46)
---
scripts/ci/libraries/_initialization.sh | 4 +++-
scripts/ci/libraries/_kind.sh | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index 316a92c..e0b4a8a 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -771,7 +771,9 @@ function initialization::ga_output() {
}
function initialization::ga_env() {
- echo "${1}=${2}" >> "${GITHUB_ENV}"
+ if [[ ${GITHUB_ENV=} != "" ]]; then
+ echo "${1}=${2}" >> "${GITHUB_ENV}"
+ fi
}
function initialization::set_mysql_encoding() {
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 451c370..41048fc 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -302,7 +302,7 @@ function kind::forward_port_to_kind_webserver() {
echo
echo "Trying to establish port forwarding to 'airflow webserver'"
echo
- if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES} == "true" ]] ; then
+ if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES=} == "true" ]] ; then
forwarded_port_number=$(( forwarded_port_number + 1 ))
fi
if [[ ${num_tries} == "${MAX_NUM_TRIES_FOR_PORT_FORWARD}" ]]; then
[airflow] 34/44: Fix permissions of mounted /tmp directory for
Breeze (#12157)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 40ebe6eb41a935cb83bff4ef80106347aa929f8c
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 9 22:01:58 2020 +0100
Fix permissions of mounted /tmp directory for Breeze (#12157)
The "tmp" directory is mounted from the host (from tmp folder
in the source airflow directory). This is needed to get some
of our docker-in-docker tools (such as gcloud/aws/java) and
get them working on demand. Thanks to that we do not have
to increase the size of CI image unnecessarily.
Those tools were introduced and made to work in #9376
However this causes some of the standard tools (such as apt-get)
to not work inside the container unless the mounted /tmp
folder has write permission for groups/other.
This PR fixes it.
(cherry picked from commit a42bbe21c899f4048b0861df2c84eb2ba9c5eeb5)
---
scripts/in_container/entrypoint_ci.sh | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index 37d3199..057d630 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -31,6 +31,11 @@ function disable_rbac_if_requested() {
# shellcheck source=scripts/in_container/_in_container_script_init.sh
. /opt/airflow/scripts/in_container/_in_container_script_init.sh
+# Add "other" and "group" write permission to the tmp folder
+# Note that it will also change permissions in the /tmp folder on the host
+# but this is necessary to enable some of our CLI tools to work without errors
+chmod 1777 /tmp
+
AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:=3.6}
[airflow] 15/44: Checks if all the libraries in setup.py are listed
in installation.rst file (#12023)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 43c72f2f4c7d0b72d9fcbbfb17135d9be59eb4c9
Author: SZN <sz...@nieradka.net>
AuthorDate: Mon Nov 2 14:17:41 2020 +0100
Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
(cherry picked from commit 2354bd2be381bcfe6db132990af1ac34df52b9b4)
---
.pre-commit-config.yaml | 6 +
BREEZE.rst | 9 +-
CONTRIBUTING.rst | 16 +-
INSTALL | 16 +-
STATIC_CODE_CHECKS.rst | 2 +
breeze-complete | 1 +
docs/installation.rst | 216 +++++++++++++++------
.../pre_commit_check_setup_installation.py | 108 +++++++++++
setup.py | 16 ++
9 files changed, 314 insertions(+), 76 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 6763164..cb5c753 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -213,6 +213,12 @@ repos:
files: ^setup.py$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
+ - id: setup-installation
+ name: Checks if all the libraries in setup.py are listed in installation.rst file
+ language: python
+ files: ^setup.py$|^docs/installation.rst$
+ pass_filenames: false
+ entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py
- id: update-breeze-file
name: Update output of breeze command in BREEZE.rst
entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh"
diff --git a/BREEZE.rst b/BREEZE.rst
index cf119ba..5e481f7 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -395,7 +395,7 @@ run ``airflow webserver``, ``airflow scheduler``, ``airflow worker`` in separate
This can be achieved either via ``tmux`` or via exec-ing into the running container from the host. Tmux
is installed inside the container and you can launch it with ``tmux`` command. Tmux provides you with the
capability of creating multiple virtual terminals and multiplex between them. More about ``tmux`` can be
-found at `tmux github wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts
+found at `tmux GitHub wiki page <https://github.com/tmux/tmux/wiki>`_ . Tmux has several useful shortcuts
that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.
.. raw:: html
@@ -1001,7 +1001,7 @@ Managing Dependencies
---------------------
If you need to change apt dependencies in the ``Dockerfile.ci``, add Python packages in ``setup.py`` or
-add javascript dependencies in ``package.json``, you can either add dependencies temporarily for a single
+add JavaScript dependencies in ``package.json``, you can either add dependencies temporarily for a single
Breeze session or permanently in ``setup.py``, ``Dockerfile.ci``, or ``package.json`` files.
Installing Dependencies for a Single Breeze Session
@@ -1865,8 +1865,9 @@ This is the current syntax for `./breeze <./breeze>`_:
flake8 forbid-tabs helm-lint incorrect-use-of-LoggingMixin insert-license
language-matters lint-dockerfile lint-openapi mixed-line-ending mypy mypy-helm
no-relative-imports pre-commit-descriptions pydevd python2-compile python2-fastcheck
- python-no-log-warn rst-backticks setup-order shellcheck sort-in-the-wild
- trailing-whitespace update-breeze-file update-extras update-local-yml-file yamllint
+ python-no-log-warn rst-backticks setup-order setup-installation shellcheck
+ sort-in-the-wild trailing-whitespace update-breeze-file update-extras
+ update-local-yml-file yamllint
You can pass extra arguments including options to to the pre-commit framework as
<EXTRA_ARGS> passed after --. For example:
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 6d3aa91..6d34026 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -540,13 +540,15 @@ This is the full list of those extras:
.. START EXTRAS HERE
-all, all_dbs, async, atlas, aws, azure, azure_blob_storage, azure_container_instances, azure_cosmos,
-azure_data_lake, azure_secrets, cassandra, celery, cgroups, cloudant, crypto, dask, databricks,
-datadog, devel, devel_azure, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp,
-gcp_api, github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos,
-kubernetes, ldap, mongo, mssql, mysql, oracle, papermill, password, pinot, postgres, presto, qds,
-rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry, slack, snowflake, ssh, statsd,
-vertica, virtualenv, webhdfs, winrm
+all, all_dbs, amazon, apache.atlas, apache.cassandra, apache.druid, apache.hdfs, apache.hive,
+apache.pinot, apache.presto, apache.webhdfs, async, atlas, aws, azure, azure_blob_storage,
+azure_container_instances, azure_cosmos, azure_data_lake, azure_secrets, cassandra, celery, cgroups,
+cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, devel, devel_all, devel_azure,
+devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp, gcp_api, github_enterprise,
+google, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap,
+microsoft.azure, microsoft.mssql, microsoft.winrm, mongo, mssql, mysql, oracle, papermill, password,
+pinot, postgres, presto, qds, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
+slack, snowflake, ssh, statsd, vertica, virtualenv, webhdfs, winrm
.. END EXTRAS HERE
diff --git a/INSTALL b/INSTALL
index c0582f3..0e2f582 100644
--- a/INSTALL
+++ b/INSTALL
@@ -45,13 +45,15 @@ pip install . \
# You can also install Airflow with extras specified. The list of available extras:
# START EXTRAS HERE
-all, all_dbs, async, atlas, aws, azure, azure_blob_storage, azure_container_instances, azure_cosmos,
-azure_data_lake, azure_secrets, cassandra, celery, cgroups, cloudant, crypto, dask, databricks,
-datadog, devel, devel_azure, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp,
-gcp_api, github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos,
-kubernetes, ldap, mongo, mssql, mysql, oracle, papermill, password, pinot, postgres, presto, qds,
-rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry, slack, snowflake, ssh, statsd,
-vertica, virtualenv, webhdfs, winrm
+all, all_dbs, amazon, apache.atlas, apache.cassandra, apache.druid, apache.hdfs, apache.hive,
+apache.pinot, apache.presto, apache.webhdfs, async, atlas, aws, azure, azure_blob_storage,
+azure_container_instances, azure_cosmos, azure_data_lake, azure_secrets, cassandra, celery, cgroups,
+cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, devel, devel_all, devel_azure,
+devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, emr, gcp, gcp_api, github_enterprise,
+google, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap,
+microsoft.azure, microsoft.mssql, microsoft.winrm, mongo, mssql, mysql, oracle, papermill, password,
+pinot, postgres, presto, qds, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
+slack, snowflake, ssh, statsd, vertica, virtualenv, webhdfs, winrm
# END EXTRAS HERE
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 1439417..ce68e0a 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -134,6 +134,8 @@ require Breeze Docker images to be installed locally:
----------------------------------- ---------------------------------------------------------------- ------------
``setup-order`` Checks for an order of dependencies in setup.py
----------------------------------- ---------------------------------------------------------------- ------------
+``setup-installation`` Checks if all the libraries in setup.py are listed in docs
+----------------------------------- ---------------------------------------------------------------- ------------
``shellcheck`` Checks shell files with shellcheck.
----------------------------------- ---------------------------------------------------------------- ------------
``sort-in-the-wild`` Sort INTHEWILD.md alphabetically.
diff --git a/breeze-complete b/breeze-complete
index 6ec739c..1b89b70 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -102,6 +102,7 @@ python2-fastcheck
python-no-log-warn
rst-backticks
setup-order
+setup-installation
shellcheck
sort-in-the-wild
trailing-whitespace
diff --git a/docs/installation.rst b/docs/installation.rst
index ed82157..de1985c 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -37,7 +37,6 @@ Those "known-to-be-working" constraints are per major/minor python version. You
files when installing Airflow from PyPI. Note that you have to specify correct Airflow version
and python versions in the URL.
-
**Prerequisites**
On Debian based Linux OS:
@@ -52,18 +51,21 @@ and python versions in the URL.
.. code-block:: bash
- pip install \
- apache-airflow==1.10.12 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
-
+ AIRFLOW_VERSION=1.10.12
+ PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
+ # For example: 3.6
+ CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
+ # For example: https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt
+ pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
-2. Installing with extras (for example postgres, gcp)
+2. Installing with extras (for example postgres, google)
.. code-block:: bash
- pip install \
- apache-airflow[postgres,gcp]==1.10.12 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
+ AIRFLOW_VERSION=1.10.12
+ PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
+ CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
+ pip install "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
You need certain system level requirements in order to install Airflow. Those are requirements that are known
@@ -108,6 +110,9 @@ these extra dependencies.
Here's the list of the subpackages and what they enable:
+
+**Fundamentals:**
+
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| subpackage | install command | enables |
+=====================+=====================================================+======================================================================+
@@ -115,76 +120,171 @@ Here's the list of the subpackages and what they enable:
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` | All databases integrations |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| async | ``pip install 'apache-airflow[async]'`` | Async worker classes for Gunicorn |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| aws | ``pip install 'apache-airflow[aws]'`` | Amazon Web Services |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| azure | ``pip install 'apache-airflow[azure]'`` | Microsoft Azure |
+| devel | ``pip install 'apache-airflow[devel]'`` | Minimum dev tools requirements |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| celery | ``pip install 'apache-airflow[celery]'`` | CeleryExecutor |
+| devel_all | ``pip install 'apache-airflow[devel_all]'`` | All dev tools requirements |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| cloudant | ``pip install 'apache-airflow[cloudant]'`` | Cloudant hook |
+| devel_azure | ``pip install 'apache-airflow[devel_azure]'`` | Azure development requirements |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| crypto | ``pip install 'apache-airflow[crypto]'`` | Encrypt connection passwords in metadata db |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` | Minimum dev tools requirements |
+| devel_ci | ``pip install 'apache-airflow[devel_ci]'`` | Development requirements used in CI |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` | Airflow + dependencies on the Hadoop stack |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| druid | ``pip install 'apache-airflow[druid]'`` | Druid related operators & hooks |
+| doc | ``pip install 'apache-airflow[doc]'`` | Packages needed to build docs |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| gcp | ``pip install 'apache-airflow[gcp]'`` | Google Cloud Platform |
+| password | ``pip install 'apache-airflow[password]'`` | Password authentication for users |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| github_enterprise | ``pip install 'apache-airflow[github_enterprise]'`` | GitHub Enterprise auth backend |
+
+
+**Apache Software:**
+
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
+| subpackage | install command | enables |
++=====================+=====================================================+======================================================================+
+| atlas | ``pip install 'apache-airflow[apache.atlas]'`` | Apache Atlas to use Data Lineage feature |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| cassandra | ``pip install 'apache-airflow[apache.cassandra]'`` | Cassandra related operators & hooks |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| druid | ``pip install 'apache-airflow[apache.druid]'`` | Druid related operators & hooks |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| hdfs | ``pip install 'apache-airflow[apache.hdfs]'`` | HDFS hooks and operators |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| hive | ``pip install 'apache-airflow[apache.hive]'`` | All Hive related operators |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| presto | ``pip install 'apache-airflow[apache.presto]'`` | All Presto related operators & hooks |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+| webhdfs | ``pip install 'apache-airflow[webhdfs]'`` | HDFS hooks and operators |
++---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
+
+
+**Services:**
+
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| subpackage | install command | enables |
++=============================+===============================================================+======================================================================+
+| aws | ``pip install 'apache-airflow[amazon]'`` | Amazon Web Services |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure | ``pip install 'apache-airflow[microsoft.azure]'`` | Microsoft Azure |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure_blob_storage | ``pip install 'apache-airflow[azure_blob_storage]'`` | Microsoft Azure (blob storage) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure_cosmos | ``pip install 'apache-airflow[azure_cosmos]'`` | Microsoft Azure (cosmos) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure_container_instances | ``pip install 'apache-airflow[azure_container_instances]'`` | Microsoft Azure (container instances) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure_data_lake | ``pip install 'apache-airflow[azure_data_lake]'`` | Microsoft Azure (data lake) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure_secrets | ``pip install 'apache-airflow[azure_secrets]'`` | Microsoft Azure (secrets) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| azure | ``pip install 'apache-airflow[microsoft.azure]'`` | Microsoft Azure |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| cloudant | ``pip install 'apache-airflow[cloudant]'`` | Cloudant hook |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| databricks | ``pip install 'apache-airflow[databricks]'`` | Databricks hooks and operators |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| datadog | ``pip install 'apache-airflow[datadog]'`` | Datadog hooks and sensors |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| gcp | ``pip install 'apache-airflow[gcp]'`` | Google Cloud |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| github_enterprise | ``pip install 'apache-airflow[github_enterprise]'`` | GitHub Enterprise auth backend |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| google | ``pip install 'apache-airflow[google]'`` | Google Cloud (same as gcp) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| google_auth | ``pip install 'apache-airflow[google_auth]'`` | Google auth backend |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| hashicorp | ``pip install 'apache-airflow[hashicorp]'`` | Hashicorp Services (Vault) |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| jira | ``pip install 'apache-airflow[jira]'`` | Jira hooks and operators |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| qds | ``pip install 'apache-airflow[qds]'`` | Enable QDS (Qubole Data Service) support |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| salesforce | ``pip install 'apache-airflow[salesforce]'`` | Salesforce hook |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| sendgrid | ``pip install 'apache-airflow[sendgrid]'`` | Send email using sendgrid |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| segment | ``pip install 'apache-airflow[segment]'`` | Segment hooks and sensors |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| sentry | ``pip install 'apache-airflow[sentry]'`` | |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| slack | ``pip install 'apache-airflow[slack]'`` | :class:`airflow.providers.slack.operators.slack.SlackAPIOperator` |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| snowflake | ``pip install 'apache-airflow[snowflake]'`` | Snowflake hooks and operators |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+| vertica | ``pip install 'apache-airflow[vertica]'`` | Vertica hook support as an Airflow backend |
++-----------------------------+---------------------------------------------------------------+----------------------------------------------------------------------+
+
+
+**Software:**
+
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| subpackage | install command | enables |
++=====================+=====================================================+====================================================================================+
+| async | ``pip install 'apache-airflow[async]'`` | Async worker classes for Gunicorn |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| celery | ``pip install 'apache-airflow[celery]'`` | CeleryExecutor |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| dask | ``pip install 'apache-airflow[dask]'`` | DaskExecutor |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| docker | ``pip install 'apache-airflow[docker]'`` | Docker hooks and operators |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| elasticsearch | ``pip install 'apache-airflow[elasticsearch]'`` | Elasticsearch hooks and Log Handler |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| kubernetes | ``pip install 'apache-airflow[cncf.kubernetes]'`` | Kubernetes Executor and operator |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| mongo | ``pip install 'apache-airflow[mongo]'`` | Mongo hooks and operators |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| mssql (deprecated) | ``pip install 'apache-airflow[microsoft.mssql]'`` | Microsoft SQL Server operators and hook, |
+| | | support as an Airflow backend. Uses pymssql. |
+| | | Will be replaced by subpackage ``odbc``. |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| mysql | ``pip install 'apache-airflow[mysql]'`` | MySQL operators and hook, support as an Airflow |
+| | | backend. The version of MySQL server has to be |
+| | | 5.6.4+. The exact version upper bound depends |
+| | | on version of ``mysqlclient`` package. For |
+| | | example, ``mysqlclient`` 1.3.12 can only be |
+| | | used with MySQL server 5.6.4 through 5.7. |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| oracle | ``pip install 'apache-airflow[oracle]'`` | Oracle hooks and operators |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| pinot | ``pip install 'apache-airflow[pinot]'`` | Pinot DB hook |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| postgres | ``pip install 'apache-airflow[postgres]'`` | PostgreSQL operators and hook, support as an |
+| | | Airflow backend |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| rabbitmq | ``pip install 'apache-airflow[rabbitmq]'`` | RabbitMQ support as a Celery backend |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| redis | ``pip install 'apache-airflow[redis]'`` | Redis hooks and sensors |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| samba | ``pip install 'apache-airflow[samba]'`` | :class:`airflow.providers.apache.hive.transfers.hive_to_samba.HiveToSambaOperator` |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| statsd | ``pip install 'apache-airflow[statsd]'`` | Needed by StatsD metrics |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+| virtualenv | ``pip install 'apache-airflow[virtualenv]'`` | |
++---------------------+-----------------------------------------------------+------------------------------------------------------------------------------------+
+
+
+**Other:**
+
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| hashicorp | ``pip install 'apache-airflow[hashicorp]'`` | Hashicorp Services (Vault) |
+| subpackage | install command | enables |
++=====================+=====================================================+======================================================================+
+| cgroups | ``pip install 'apache-airflow[cgroups]'`` | Needed To use CgroupTaskRunner |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| hdfs | ``pip install 'apache-airflow[hdfs]'`` | HDFS hooks and operators |
+| crypto | ``pip install 'apache-airflow[crypto]'`` | Cryptography libraries |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| hive | ``pip install 'apache-airflow[hive]'`` | All Hive related operators |
+| grpc | ``pip install 'apache-airflow[grpc]'`` | Grpc hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| jdbc | ``pip install 'apache-airflow[jdbc]'`` | JDBC hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| kerberos | ``pip install 'apache-airflow[kerberos]'`` | Kerberos integration for Kerberized Hadoop |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| kubernetes | ``pip install 'apache-airflow[kubernetes]'`` | Kubernetes Executor and operator |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| ldap | ``pip install 'apache-airflow[ldap]'`` | LDAP authentication for users |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| mssql | ``pip install 'apache-airflow[mssql]'`` | Microsoft SQL Server operators and hook, |
-| | | support as an Airflow backend |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| mysql | ``pip install 'apache-airflow[mysql]'`` | MySQL operators and hook, support as an Airflow |
-| | | backend. The version of MySQL server has to be |
-| | | 5.6.4+. The exact version upper bound depends |
-| | | on version of ``mysqlclient`` package. For |
-| | | example, ``mysqlclient`` 1.3.12 can only be |
-| | | used with MySQL server 5.6.4 through 5.7. |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| oracle | ``pip install 'apache-airflow[oracle]'`` | Oracle hooks and operators |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| password | ``pip install 'apache-airflow[password]'`` | Password authentication for users |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| postgres | ``pip install 'apache-airflow[postgres]'`` | PostgreSQL operators and hook, support as an |
-| | | Airflow backend |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| presto | ``pip install 'apache-airflow[presto]'`` | All Presto related operators & hooks |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| qds | ``pip install 'apache-airflow[qds]'`` | Enable QDS (Qubole Data Service) support |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| rabbitmq | ``pip install 'apache-airflow[rabbitmq]'`` | RabbitMQ support as a Celery backend |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| redis | ``pip install 'apache-airflow[redis]'`` | Redis hooks and sensors |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| samba | ``pip install apache-airflow[samba]'`` | :class:`airflow.operators.hive_to_samba_operator.Hive2SambaOperator` |
-+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| slack | ``pip install 'apache-airflow[slack']`` | :class:`airflow.operators.slack_operator.SlackAPIOperator` |
+| papermill | ``pip install 'apache-airflow[papermill]'`` | Papermill hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
| ssh | ``pip install 'apache-airflow[ssh]'`` | SSH hooks and Operator |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
-| vertica | ``pip install 'apache-airflow[vertica]'`` | Vertica hook support as an Airflow backend |
+| winrm | ``pip install 'apache-airflow[microsoft.winrm]'`` | WinRM hooks and operators |
+---------------------+-----------------------------------------------------+----------------------------------------------------------------------+
Initializing Airflow Database
@@ -200,4 +300,4 @@ run tasks:
.. code-block:: bash
- airflow initdb
+ airflow db init
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
new file mode 100755
index 0000000..2fdeca6
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Checks if all the libraries in setup.py are listed in installation.rst file
+"""
+
+import os
+import re
+from os.path import dirname
+from typing import Dict, List
+
+AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir, os.pardir)
+SETUP_PY_FILE = 'setup.py'
+DOCS_FILE = 'installation.rst'
+PY_IDENTIFIER = r'[a-zA-Z_][a-zA-Z0-9_\.]*'
+
+
+def get_file_content(*path_elements: str) -> str:
+ file_path = os.path.join(AIRFLOW_SOURCES_DIR, *path_elements)
+ with open(file_path) as file_to_read:
+ return file_to_read.read()
+
+
+def get_extras_from_setup() -> Dict[str, List[str]]:
+ """
+ Returns an array EXTRAS_REQUIREMENTS with aliases from setup.py file in format:
+ {'package name': ['alias1', 'alias2], ...}
+ """
+ setup_content = get_file_content(SETUP_PY_FILE)
+
+ extras_section_regex = re.compile(
+ r'^EXTRAS_REQUIREMENTS[^{]+{([^}]+)}', re.MULTILINE)
+ extras_section = extras_section_regex.findall(setup_content)[0]
+
+ extras_regex = re.compile(
+ rf'^\s+[\"\']({PY_IDENTIFIER})[\"\']:\s*({PY_IDENTIFIER})[^#\n]*(#\s*TODO.*)?$', re.MULTILINE)
+
+ extras_dict: Dict[str, List[str]] = {}
+ for extras in extras_regex.findall(extras_section):
+ package = extras[1]
+ alias = extras[0]
+ if not extras_dict.get(package):
+ extras_dict[package] = []
+ extras_dict[package].append(alias)
+ return extras_dict
+
+
+def get_extras_from_docs() -> List[str]:
+ """
+ Returns an array of install packages names from installation.rst.
+ """
+ docs_content = get_file_content('docs', DOCS_FILE)
+
+ extras_section_regex = re.compile(rf'^\|[^|]+\|.*pip install .apache-airflow\[({PY_IDENTIFIER})\].',
+ re.MULTILINE)
+ extras = extras_section_regex.findall(docs_content)
+
+ extras = list(filter(lambda entry: entry != 'all', extras))
+ return extras
+
+
+if __name__ == '__main__':
+ setup_packages = get_extras_from_setup()
+ docs_packages = get_extras_from_docs()
+
+ output_table = ""
+
+ for extras in sorted(setup_packages.keys()):
+ if not set(setup_packages[extras]).intersection(docs_packages):
+ output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "V", "")
+
+ setup_packages_str = str(setup_packages)
+ for extras in sorted(docs_packages):
+ if f"'{extras}'" not in setup_packages_str:
+ output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "", "V")
+
+ if(output_table == ""):
+ exit(0)
+
+ print(f"""
+ERROR
+
+"EXTRAS_REQUIREMENTS" section in {SETUP_PY_FILE} should be synchronized
+with "Extra Packages" section in documentation file doc/{DOCS_FILE}.
+
+here is a list of packages that are used but are not documented, or
+documented although not used.
+ """)
+ print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
+ print(output_table)
+
+ exit(1)
diff --git a/setup.py b/setup.py
index 3ace091..9a7e732 100644
--- a/setup.py
+++ b/setup.py
@@ -484,6 +484,15 @@ else:
EXTRAS_REQUIREMENTS = {
'all': devel_all,
'all_dbs': all_dbs,
+ 'amazon': aws,
+ 'apache.atlas': atlas,
+ "apache.cassandra": cassandra,
+ "apache.druid": druid,
+ "apache.hdfs": hdfs,
+ "apache.hive": hive,
+ "apache.pinot": pinot,
+ "apache.presto": presto,
+ "apache.webhdfs": webhdfs,
'async': async_packages,
'atlas': atlas,
'aws': aws,
@@ -497,11 +506,13 @@ EXTRAS_REQUIREMENTS = {
'celery': celery,
'cgroups': cgroups,
'cloudant': cloudant,
+ 'cncf.kubernetes': kubernetes,
'crypto': crypto,
'dask': dask,
'databricks': databricks,
'datadog': datadog,
'devel': devel_minreq,
+ 'devel_all': devel_all,
'devel_azure': devel_azure,
'devel_ci': devel_ci,
'devel_hadoop': devel_hadoop,
@@ -513,6 +524,7 @@ EXTRAS_REQUIREMENTS = {
'gcp': gcp,
'gcp_api': gcp,
'github_enterprise': flask_oauth,
+ 'google': gcp,
'google_auth': flask_oauth,
'grpc': grpc,
'hashicorp': hashicorp,
@@ -525,6 +537,10 @@ EXTRAS_REQUIREMENTS = {
'ldap': ldap,
'mongo': mongo,
'mssql': mssql,
+ 'microsoft.azure':
+ azure_blob_storage + azure_container_instances + azure_cosmos + azure_data_lake + azure_secrets,
+ 'microsoft.mssql': mssql,
+ 'microsoft.winrm': winrm,
'mysql': mysql,
'oracle': oracle,
'papermill': papermill,
[airflow] 24/44: Add Kubernetes files to selective checks (#12114)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 49430a0b6478b80b720a2ea278cfd428a23d1f95
Author: Daniel Imberman <da...@gmail.com>
AuthorDate: Thu Nov 5 11:54:49 2020 -0800
Add Kubernetes files to selective checks (#12114)
* Add Kubernetes files to selective checks
There are multiple kubernetes-related files that require
running the k8s integration tests. This PR adds those to the
run_selective_tests
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Kaxil Naik <ka...@gmail.com>
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
* Update scripts/ci/selective_ci_checks.sh
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
Co-authored-by: Kaxil Naik <ka...@gmail.com>
Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
(cherry picked from commit 054de0703aa99fc425028d4cbe1f7b363e0cbeca)
---
scripts/ci/selective_ci_checks.sh | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/scripts/ci/selective_ci_checks.sh b/scripts/ci/selective_ci_checks.sh
index b9a6d74..3c7132d 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -179,7 +179,6 @@ ALL_TESTS="Core Integration Heisentests"
readonly ALL_TESTS
function set_outputs_run_everything_and_exit() {
- needs_api_tests "true"
needs_helm_tests "true"
needs_javascript_scans "true"
needs_python_scans "true"
@@ -201,7 +200,6 @@ function set_outputs_run_all_tests() {
}
function set_output_skip_all_tests_and_docs_and_exit() {
- needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -215,7 +213,6 @@ function set_output_skip_all_tests_and_docs_and_exit() {
}
function set_output_skip_tests_but_build_images_and_exit() {
- needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
needs_python_scans "false"
@@ -378,9 +375,8 @@ function get_count_kubernetes_files() {
echo "Count Kubernetes files"
echo
local pattern_array=(
- "^airflow/kubernetes"
"^chart"
- "^tests/kubernetes_tests"
+ "^kubernetes_tests"
)
show_changed_files
COUNT_KUBERNETES_CHANGED_FILES=$(count_changed_files)
[airflow] 10/44: Adds more aggressive cancelling of duplicate Build
Image jobs (#12018)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 99aa8839732795038288adcfb031a726700ff930
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 1 17:54:37 2020 +0100
Adds more aggressive cancelling of duplicate Build Image jobs (#12018)
This change adds even more aggressive cancelling of duplicates of
'Build Image' jobs. it's not an obvious task to know which
Build Image jobs are duplicates, we are matching those duplicates
based on specially crafted "build-info" job names. We add
Event, Branch, Repo to the job names and assume that two
runs with the same event + branch + repo are duplicates.
It also disables self-preservation for this step because
it is perfectly ok to cancel itself in case there is a newer
in-progress Build Image job.
Unfortunately even this will not work perfectly well. Those job
names are resolved only for the jobs that are runnning rather than
the queued ones, so in case we have several duplicates of the
same build image job in the queue, they will not be found/cancelled.
The cancelling will only happen if both duplicates are already
running.
It's good enough for now and we cannot do much more until there
is a missing feature added to GitHub API that allows to link
the workflow_run with the run that triggered it. This issue has
been raised to GitHub Support and internal engineering ticket
has been apparently opened to add this feature.
More detailed status for the missing feature is kept at #11294
(cherry picked from commit 1d14e74e33efbcb17e5553dba92baf9c8b0fc7c8)
---
.github/workflows/build-images-workflow-run.yml | 40 +++++++++++++++----------
.github/workflows/ci.yml | 2 +-
2 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 972195a..a3880e7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,7 +65,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: allDuplicates
@@ -77,12 +77,12 @@ jobs:
# https://github.community/t/how-to-set-and-access-a-workflow-variable/17335/16
echo "::set-output name=buildImages::${BUILD_IMAGES}"
- name: "Cancel duplicated 'Build Image' runs"
-
# We find duplicates of our own "Build Image" runs - due to a missing feature
- # in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
- # build-info step there to find the duplicates ¯\_(ツ)_/¯.
-
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ # in GitHub Actions, we have to use Job names to match Event/Repo/Branch matching
+ # trick ¯\_(ツ)_/¯. We name the build-info job appropriately
+ # and then we try to find and cancel all the jobs with the same Event + Repo + Branch as the
+ # current Event/Repo/Branch combination.
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -93,14 +93,12 @@ jobs:
Branch: ${{ steps.source-run-info.outputs.sourceHeadBranch }}.*"]
if: env.BUILD_IMAGES == 'true'
- name: "Cancel all 'CI Build' runs where some jobs failed"
-
# We find any of the "CI Build" workflow runs, where any of the important jobs
# failed. The important jobs are selected by the regexp array below.
# We also produce list of canceled "CI Build' runs as output, so that we
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
-
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -111,7 +109,6 @@ jobs:
["^Static checks.*", "^Build docs$", "^Spell check docs$", "^Backport packages$",
"^Checks: Helm tests$", "^Test OpenAPI*"]
- name: "Extract canceled failed runs"
-
# We use this step to build regexp that will be used to match the Source Run id in
# the build-info job below. If we cancelled some "CI Build" runs in the "cancel-failed' step
# above - we want to cancel also the corresponding "Build Images" runs. Again we have
@@ -133,15 +130,15 @@ jobs:
# We take the extracted regexp array prepared in the previous step and we use
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
- if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ if: env.BUILD_IMAGES == 'true' && steps.cancel-failed.outputs.cancelledRuns != '[]'
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -164,6 +161,19 @@ jobs:
else
echo "::set-output name=upgradeToLatestConstraints::false"
fi
+ - name: "Cancel all duplicated 'Build Image' runs"
+ # We find duplicates of all "Build Image" runs - due to a missing feature
+ # in GitHub Actions, we have to use Job names to match Event/Repo/Branch matching
+ # trick ¯\_(ツ)_/¯. We name the build-info job appropriately and then we try to match
+ # all the jobs with the same Event + Repo + Branch match and cancel all the duplicates for those
+ # This might cancel own run, so this is the last step in the job
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
+ with:
+ cancelMode: allDuplicatedNamedJobs
+ token: ${{ secrets.GITHUB_TOKEN }}
+ notifyPRCancel: true
+ selfPreservation: false
+ jobNameRegexps: '["Event: \\S* Repo: \\S* Branch: \\S* "]'
build-info:
# The name is such long because we are using it to cancel duplicated 'Build Images' runs
@@ -372,7 +382,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -387,7 +397,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
+ uses: potiuk/cancel-workflow-runs@f06d03cd576a179ea5169d048dbd8c8d73757b52 # v4_4
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 630b5d6..a5457c0 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -117,7 +117,7 @@ jobs:
uses: actions/checkout@v2
- name: >
Event: ${{ github.event_name }}
- Repo: ${{ github.repository }}
+ Repo: ${{ steps.source-run-info.outputs.sourceHeadRepo }}
Branch: ${{ github.head_ref }}
Run id: ${{ github.run_id }}
Sha: ${{ github.sha }}
[airflow] 21/44: Use sys.exit() instead of exit() (#12084)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2be4d56a78b2b49b08866ac9114a89ab5db7eef3
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Wed Nov 4 11:50:52 2020 +0000
Use sys.exit() instead of exit() (#12084)
The `exit` and `quit` functions are actually `site.Quitter` objects and are loaded, at interpreter start up, from site.py. However, if the interpreter is started with the `-S` flag, or a custom site.py is used then exit and quit may not be present. It is recommended to use `sys.exit()` which is built into the interpreter and is guaranteed to be present.
(cherry picked from commit bec9f3b29fd42ecd1beae3db75784b9a726caf15)
---
scripts/ci/pre_commit/pre_commit_check_setup_installation.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index 2fdeca6..b4f3281 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -22,6 +22,7 @@ Checks if all the libraries in setup.py are listed in installation.rst file
import os
import re
+import sys
from os.path import dirname
from typing import Dict, List
@@ -90,8 +91,8 @@ if __name__ == '__main__':
if f"'{extras}'" not in setup_packages_str:
output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "", "V")
- if(output_table == ""):
- exit(0)
+ if output_table == "":
+ sys.exit(0)
print(f"""
ERROR
@@ -105,4 +106,4 @@ documented although not used.
print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
print(output_table)
- exit(1)
+ sys.exit(1)
[airflow] 06/44: Switches to "cancel-all-duplicates' mode of
cancelling. (#12004)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d3ab3df2eaad211f2ce24f73e354de4ec904716d
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Oct 31 23:39:54 2020 +0100
Switches to "cancel-all-duplicates' mode of cancelling. (#12004)
The cancel-workflow-runs action in version 4.1 got the capability
of cancelling not only duplicates of own run (including future,
queued duplicates) but also cancelling all duplicates from all
running worklfows - regardless if they were triggered by my own
PR or some other PRs. This will be even more helpful with handling
the queues and optimising our builds, because in case ANY of
the build image workflows starts to run, it will cancel ALL
duplicates immediately.
(cherry picked from commit 21350aa3cf8952b605713257ae94e1ed648dd00b)
---
.github/workflows/build-images-workflow-run.yml | 19 +++++++++----------
1 file changed, 9 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 81c4fb4..972195a 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -65,12 +65,11 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
sourceRunId: ${{ github.event.workflow_run.id }}
- notifyPRCancel: true
- name: "Output BUILD_IMAGES"
id: build-images
run: |
@@ -83,7 +82,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -101,7 +100,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -135,18 +134,18 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
notifyPRCancel: true
jobNameRegexps: ${{ steps.extract-cancelled-failed-runs.outputs.matching-regexp }}
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
- cancelMode: duplicates
+ cancelMode: allDuplicates
workflowFileName: 'codeql-analysis.yml'
- name: "Set Docker Cache Directive"
id: cache-directive
@@ -373,7 +372,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -388,7 +387,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@c8448eb1e435664b3731ea1ead2efa0d1bb83b5b # v4_0
+ uses: potiuk/cancel-workflow-runs@99869d37d982384d18c79539b67df94f17557cbe # v4_1
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
[airflow] 08/44: Adds documentation about the optimized PR workflow
(#12006)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 80845847d5971e64a0b19a589a9998cf330abe07
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 1 00:20:38 2020 +0100
Adds documentation about the optimized PR workflow (#12006)
We had a lot of problems recently about the queues in Github
Actions. This documentations explains the motivation and approach
we have taken for optimizing our PR workflow.
(cherry picked from commit d85a31f2d88b92b480438d86aa8e3e79e6c3614d)
---
CI.rst | 102 ++----------
PULL_REQUEST_WORKFLOW.rst | 260 +++++++++++++++++++++++++++++++
images/pr/pr-full-tests-needed.png | Bin 0 -> 88512 bytes
images/pr/pr-likely-ok-to-merge.png | Bin 0 -> 98362 bytes
images/pr/pr-no-tests-needed-comment.png | Bin 0 -> 80852 bytes
images/pr/selective_checks.md5 | 1 +
images/pr/selective_checks.mermaid | 35 +++++
images/pr/selective_checks.png | Bin 0 -> 64501 bytes
8 files changed, 309 insertions(+), 89 deletions(-)
diff --git a/CI.rst b/CI.rst
index dc1cdf2..f4b5294 100644
--- a/CI.rst
+++ b/CI.rst
@@ -35,7 +35,7 @@ the CI jobs. And we have a number of variables determine build behaviour.
-Github Actions runs
+GitHub Actions runs
-------------------
Our builds on CI are highly optimized. They utilise some of the latest features provided by GitHub Actions
@@ -65,7 +65,7 @@ utilise the WRITE access to Apache Airflow repository via an external Pull Reque
Thanks to the WRITE access and fact that the 'workflow_run' by default uses the 'master' version of the
sources, we can safely run some logic there will checkout the incoming Pull Request, build the container
-image from the sources from the incoming PR and push such image to an Github Docker Registry - so that
+image from the sources from the incoming PR and push such image to an GitHub Docker Registry - so that
this image can be built only once and used by all the jobs running tests. The image is tagged with unique
``RUN_ID`` of the incoming Pull Request and the tests run in the Pull Request can simply pull such image
rather than build it from the scratch. Pulling such image takes ~ 1 minute, thanks to that we are saving
@@ -92,7 +92,7 @@ connected with the run.
You can read more about it in `BREEZE.rst <BREEZE.rst>`_ and `TESTING.rst <TESTING.rst>`_
-Difference between local runs and Github Action workflows
+Difference between local runs and GitHub Action workflows
---------------------------------------------------------
Depending whether the scripts are run locally (most often via `Breeze <BREEZE.rst>`_) or whether they
@@ -470,7 +470,13 @@ The main purpose of those jobs is to check if PR builds cleanly, if the test run
the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry -
CI, Production Images as well as base Python images that are also cached in the Private GitHub registry.
Also for those builds we only execute Python tests if important files changed (so for example if it is
-doc-only change, no tests will be executed.
+"no-code" change, no tests will be executed.
+
+The workflow involved in Pull Requests review and approval is a bit more complex than simple workflows
+in most of other projects because we've implemented some optimizations related to efficient use
+of queue slots we share with other Apache Software Foundation projects. More details about it
+can be found in `PULL_REQUEST_WORKFLOW.rst <PULL_REQUEST_WORKFLOW.rst>`_.
+
Direct Push/Merge Run
---------------------
@@ -641,7 +647,7 @@ Comments:
(1) CRON jobs builds images from scratch - to test if everything works properly for clean builds
(2) The tests are run when the Trigger Tests job determine that important files change (this allows
- for example doc-only changes to build much faster)
+ for example "no-code" changes to build much faster)
(3) The jobs wait for CI images if ``GITHUB_REGISTRY_WAIT_FOR_IMAGE`` variable is set to "true".
You can set it to "false" to disable using shared images - this is slower though as the images
are rebuilt in every job that needs them. You can also set your own fork's secret
@@ -685,7 +691,7 @@ way to sync your fork master to the Apache Airflow's one.
Delete old artifacts
--------------------
-This workflow is introduced, to delete old artifacts from the Github Actions build. We set it to
+This workflow is introduced, to delete old artifacts from the GitHub Actions build. We set it to
delete old artifacts that are > 7 days old. It only runs for the 'apache/airflow' repository.
We also have a script that can help to clean-up the old artifacts:
@@ -695,89 +701,7 @@ CodeQL scan
-----------
The CodeQL security scan uses GitHub security scan framework to scan our code for security violations.
-It is run for javascript and python code.
-
-
-Selective CI Checks
-===================
-
-In order to optimise our CI builds, we've implemented optimisations to only run selected checks for some
-kind of changes. The logic implemented reflects the internal architecture of Airflow 2.0 packages
-and it helps to keep down both the usage of jobs in GitHub Actions as well as CI feedback time to
-contributors in case of simpler changes.
-
-We have the following test types (separated by packages in which they are):
-
-* Core - for the core Airflow functionality (core folder)
-
-We also have several special kinds of tests that are not separated by packages but they are marked with
-pytest markers. They can be found in any of those packages and they can be selected by the appropriate
-pylint custom command line options. See `TESTING.rst <TESTING.rst>`_ for details but those are:
-
-* Integration - tests that require external integration images running in docker-compose
-* Heisentests - tests that are vulnerable to some side effects and are better to be run on their own
-* Quarantined - tests that are flaky and need to be fixed
-* Postgres - tests that require Postgres database. They are only run when backend is Postgres
-* MySQL - tests that require MySQL database. They are only run when backend is MySQL
-
-Even if the types are separated, In case they share the same backend version/python version, they are
-run sequentially in the same job, on the same CI machine. Each of them in a separate ``docker run`` command
-and with additional docker cleaning between the steps to not fall into the trap of exceeding resource
-usage in one big test run, but also not to increase the number of jobs per each Pull Request.
-
-The logic implemented for the changes works as follows:
-
-1) In case of direct push (so when PR gets merged) or scheduled run, we always run all tests and checks.
- This is in order to make sure that the merge did not miss anything important. The remainder of the logic
- is executed only in case of Pull Requests.
-
-2) We retrieve which files have changed in the incoming Merge Commit (github.sha is a merge commit
- automatically prepared by GitHub in case of Pull Request, so we can retrieve the list of changed
- files from that commit directly).
-
-3) If any of the important, environment files changed (Dockerfile, ci scripts, setup.py, GitHub workflow
- files), then we again run all tests and checks. Those are cases where the logic of the checks changed
- or the environment for the checks changed so we want to make sure to check everything.
-
-4) If any of docs changed: we need to have CI image so we enable image building
-
-5) If any of chart files changed, we need to run helm tests so we enable helm unit tests
-
-6) If any of API files changed, we need to run API tests so we enable them
-
-7) If any of the relevant source files that trigger the tests have changed at all. Those are airflow
- sources, chart, tests and kubernetes_tests. If any of those files changed, we enable tests and we
- enable image building, because the CI images are needed to run tests.
-
-8) Then we determine which types of the tests should be run. We count all the changed files in the
- relevant airflow sources (airflow, chart, tests, kubernetes_tests) first and then we count how many
- files changed in different packages:
-
- a) if any of the Kubernetes files changed we enable ``Kubernetes`` test type
- b) Then we subtract count of all the ``specific`` above per-type changed files from the count of
- all changed files. In case there are any files changed, then we assume that some unknown files
- changed (likely from the core of airflow) and in this case we enable all test types above and the
- Core test types - simply because we do not want to risk to miss anything.
- g) In all cases where tests are enabled we also add Heisentests, Integration and - depending on
- the backend used = Postgres or MySQL types of tests.
-
-9) Quarantined tests are always run when tests are run - we need to run them often to observe how
- often they fail so that we can decide to move them out of quarantine. Details about the
- Quarantined tests are described in `TESTING.rst <TESTING.rst>`_
-
-10) There is a special case of static checks. In case the above logic determines that the CI image
- needs to be build, we run long and more comprehensive version of static checks - including Pylint,
- MyPy, Flake8. And those tests are run on all files, no matter how many files changed.
- In case the image is not built, we run only simpler set of changes - the longer static checks
- that require CI image are skipped, and we only run the tests on the files that changed in the incoming
- commit - unlike pylint/flake8/mypy, those static checks are per-file based and they should not miss any
- important change.
-
-Similarly to selective tests we also run selective security scans. In Pull requests,
-the Python scan will only run when there is a python code change and javascript scan will only run if
-there is a javascript or yarn.lock file change. For master builds, all scans are always executed.
-
-
+It is run for JavaScript and python code.
Naming conventions for stored images
====================================
diff --git a/PULL_REQUEST_WORKFLOW.rst b/PULL_REQUEST_WORKFLOW.rst
new file mode 100644
index 0000000..c9cc6bf
--- /dev/null
+++ b/PULL_REQUEST_WORKFLOW.rst
@@ -0,0 +1,260 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. contents:: :local:
+
+Why non-standard pull request workflow?
+---------------------------------------
+
+This document describes the Pull Request Workflow we've implemented in Airflow. The workflow is slightly
+more complex than regular workflow you might encounter in most of the projects because after experiencing
+some huge delays in processing queues in October 2020 with GitHub Actions, we've decided to optimize the
+workflow to minimize the use of Github Actions build time by utilising selective approach on which tests
+and checks in the CI system are run depending on analysis of which files changed in the incoming PR and
+allowing the Committers to control the scope of the tests during the approval/review process.
+
+Just to give a bit of context, we started off with the approach that we always run all tests for all the
+incoming PRs, however due to our matrix of tests growing, this approach did not scale with the increasing
+number of PRs and when we had to compete with other Apache Software Foundation projects for the 180
+slots that are available for the whole organization. More Apache Software Foundation projects started
+to use GitHub Actions and we've started to experience long queues when our jobs waited for free slots.
+
+We approached the problem by:
+
+1) Improving mechanism of cancelling duplicate workflow runs more efficiently in case of queue conditions
+ (duplicate workflow runs are generated when someone pushes a fixup quickly - leading to running both
+ out-dated and current run to completion, taking precious slots. This has been implemented by improving
+ `cancel-workflow-run <https://github.com/potiuk/cancel-workflow-runs/>`_ action we are using. In version
+ 4.1 it got a new feature of cancelling all duplicates even if there is a long queue of builds.
+
+2) Heavily decreasing strain on the Github Actions jobs by introducing selective checks - mechanism
+ to control which parts of the tests are run during the tests. This is implemented by the
+ ``scripts/ci/selective_ci_checks.sh`` script in our repository. This script analyses which part of the
+ code has changed and based on that it sets the right outputs that control which tests are executed in
+ the CI build, and whether we need to build CI images necessary to run those steps. This allowed to
+ heavily decrease the strain especially for the Pull Requests that were not touching code (in which case
+ the builds can complete in < 2 minutes) but also by limiting the number of tests executed in PRs that do
+ not touch the "core" of Airflow, or only touching some - standalone - parts of Airflow such as
+ "Providers", "WWW" or "CLI". This solution is not yet perfect as there are likely some edge cases but
+ it is easy to maintain and we have an escape-hatch - all the tests are always executed in master pushes,
+ so contributors can easily spot if there is a "missed" case and fix it - both by fixing the problem and
+ adding those exceptions to the code. More about it can be found in the
+ `Selective CI checks <#selective-ci-checks>`_ chapter.
+
+3) Even more optimisation came from limiting the scope of tests to only "default" matrix parameters. So far
+ in Airflow we always run all tests for all matrix combinations. The primary matrix components are:
+
+ * Python versions (currently 3.6, 3.7, 3.8)
+ * Backend types (currently MySQL/Postgres)
+ * Backed version (currently MySQL 5.7, MySQL 8, Postgres 9.6, Postgres 13
+
+ We've decided that instead of running all the combinations of parameters for all matrix component we will
+ only run default values (Python 3.6, Mysql 5.7, Postgres 9.6) for all PRs which are not approved yet by
+ the committers. This has a nice effect, that full set of tests (though with limited combinations of
+ the matrix) are still run in the CI for every Pull Request that needs tests at all - allowing the
+ contributors to make sure that their PR is "good enough" to be reviewed.
+
+ Even after approval, the automated workflows we've implemented, check if the PR seems to need
+ "full test matrix" and provide helpful information to both contributors and committers in the form of
+ explanatory comments and labels set automatically showing the status of the PR. Committers have still
+ control whether they want to merge such requests automatically or ask for rebase or re-run the tests
+ and run "full tests" by applying the "full tests needed" label and re-running such request.
+ The "full tests needed" label is also applied automatically after approval when the change touches
+ the "core" of Airflow - also a separate check is added to the PR so that the "merge" button status
+ will indicate to the committer that full tests are still needed. The committer might still decide,
+ whether to merge such PR without the "full matrix". The "escape hatch" we have - i.e. running the full
+ matrix of tests in the "merge push" will enable committers to catch and fix such problems quickly.
+ More about it can be found in `Approval workflow and Matrix tests <#approval-workflow-and-matrix-tests>`_
+ chapter.
+
+4) We've also applied (and received) funds to run self-hosted runners. This is not yet implemented, due to
+ discussions about security of self-hosted runners for public repositories. Running self-hosted runners by
+ public repositories is currently (as of end of October 2020)
+ `Discouraged by GitHub <https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories>`_
+ and we are working on solving the problem - also involving Apache Software Foundation infrastructure team.
+ This document does not describe this part of the approach. Most likely we will add soon a document
+ describing details of the approach taken there.
+
+Selective CI Checks
+-------------------
+
+In order to optimise our CI builds, we've implemented optimisations to only run selected checks for some
+kind of changes. The logic implemented reflects the internal architecture of Airflow 2.0 packages
+and it helps to keep down both the usage of jobs in GitHub Actions as well as CI feedback time to
+contributors in case of simpler changes.
+
+We have the following test types (separated by packages in which they are):
+
+* Always - those are tests that should be always executed (always folder)
+* Core - for the core Airflow functionality (core folder)
+* API - Tests for the Airflow API (api and api_connexion folders)
+* CLI - Tests for the Airflow CLI (cli folder)
+* WWW - Tests for the Airflow webserver (www and www_rbac in 1.10 folders)
+* Providers - Tests for all Providers of Airflow (providers folder)
+* Other - all other tests (all other folders that are not part of any of the above)
+
+We also have several special kinds of tests that are not separated by packages but they are marked with
+pytest markers. They can be found in any of those packages and they can be selected by the appropriate
+pylint custom command line options. See `TESTING.rst <TESTING.rst>`_ for details but those are:
+
+* Integration - tests that require external integration images running in docker-compose
+* Heisentests - tests that are vulnerable to some side effects and are better to be run on their own
+* Quarantined - tests that are flaky and need to be fixed
+* Postgres - tests that require Postgres database. They are only run when backend is Postgres
+* MySQL - tests that require MySQL database. They are only run when backend is MySQL
+
+Even if the types are separated, In case they share the same backend version/python version, they are
+run sequentially in the same job, on the same CI machine. Each of them in a separate ``docker run`` command
+and with additional docker cleaning between the steps to not fall into the trap of exceeding resource
+usage in one big test run, but also not to increase the number of jobs per each Pull Request.
+
+The logic implemented for the changes works as follows:
+
+1) In case of direct push (so when PR gets merged) or scheduled run, we always run all tests and checks.
+ This is in order to make sure that the merge did not miss anything important. The remainder of the logic
+ is executed only in case of Pull Requests.
+
+2) We retrieve which files have changed in the incoming Merge Commit (github.sha is a merge commit
+ automatically prepared by GitHub in case of Pull Request, so we can retrieve the list of changed
+ files from that commit directly).
+
+3) If any of the important, environment files changed (Dockerfile, ci scripts, setup.py, GitHub workflow
+ files), then we again run all tests and checks. Those are cases where the logic of the checks changed
+ or the environment for the checks changed so we want to make sure to check everything.
+
+4) If any of docs changed: we need to have CI image so we enable image building
+
+5) If any of chart files changed, we need to run helm tests so we enable helm unit tests
+
+6) If any of API files changed, we need to run API tests so we enable them
+
+7) If any of the relevant source files that trigger the tests have changed at all. Those are airflow
+ sources, chart, tests and kubernetes_tests. If any of those files changed, we enable tests and we
+ enable image building, because the CI images are needed to run tests.
+
+8) Then we determine which types of the tests should be run. We count all the changed files in the
+ relevant airflow sources (airflow, chart, tests, kubernetes_tests) first and then we count how many
+ files changed in different packages:
+
+ a) in any case tests in ``Always`` folder are run. Those are special tests that should be run any time
+ modifications to any Python code occurs. Example test of this type is verifying proper structure of
+ the project including proper naming of all files.
+ b) if any of the Airflow API files changed we enable ``API`` test type
+ c) if any of the Airflow CLI files changed we enable ``CLI`` test type
+ d) if any of the Provider files changed we enable ``Providers`` test type
+ e) if any of the WWW files changed we enable ``WWW`` test type
+ f) if any of the Kubernetes files changed we enable ``Kubernetes`` test type
+ g) Then we subtract count of all the ``specific`` above per-type changed files from the count of
+ all changed files. In case there are any files changed, then we assume that some unknown files
+ changed (likely from the core of airflow) and in this case we enable all test types above and the
+ Core test types - simply because we do not want to risk to miss anything.
+ h) In all cases where tests are enabled we also add Heisentests, Integration and - depending on
+ the backend used = Postgres or MySQL types of tests.
+
+9) Quarantined tests are always run when tests are run - we need to run them often to observe how
+ often they fail so that we can decide to move them out of quarantine. Details about the
+ Quarantined tests are described in `TESTING.rst <TESTING.rst>`_
+
+10) There is a special case of static checks. In case the above logic determines that the CI image
+ needs to be build, we run long and more comprehensive version of static checks - including Pylint,
+ MyPy, Flake8. And those tests are run on all files, no matter how many files changed.
+ In case the image is not built, we run only simpler set of changes - the longer static checks
+ that require CI image are skipped, and we only run the tests on the files that changed in the incoming
+ commit - unlike pylint/flake8/mypy, those static checks are per-file based and they should not miss any
+ important change.
+
+Similarly to selective tests we also run selective security scans. In Pull requests,
+the Python scan will only run when there is a python code change and JavaScript scan will only run if
+there is a JavaScript or yarn.lock file change. For master builds, all scans are always executed.
+
+The selective check algorithm is shown here:
+
+.. image:: images/pr/selective_checks.png
+ :align: center
+ :alt: Selective check algorithm
+
+Approval Workflow and Matrix tests
+----------------------------------
+
+As explained above the approval and matrix tests workflow works according to the algorithm below:
+
+1) In case of "no-code" changes - so changes that do not change any of the code or environment of
+ the application, no test are run (this is done via selective checks above). Also no CI/PROD images are
+ build saving extra minutes. Such build takes less than 2 minutes currently and only few jobs are run
+ which is a very small fraction of the "full build" time.
+
+2) When new PR is created, only a "default set" of matrix test are running. Only default
+ values for each of the parameters are used effectively limiting it to running matrix builds for only
+ one python version and one version of each of the backends. In this case only one CI and one PROD
+ image is built, saving precious job slots. This build takes around 50% less time than the "full matrix"
+ build.
+
+3) When such PR gets approved, the system further analyses the files changed in this PR and further
+ decision is made that should be communicated to both Committer and Reviewer.
+
+3a) In case of "no-code" builds, a message is communicated that the PR is ready to be merged and
+ no tests are needed.
+
+.. image:: images/pr/pr-no-tests-needed-comment.png
+ :align: center
+ :alt: No tests needed for "no-code" builds
+
+3b) In case of "non-core" builds a message is communicated that such PR is likely OK to be merged as is with
+ limited set of tests, but that the committer might decide to re-run the PR after applying
+ "full tests needed" label, which will trigger full matrix build for tests for this PR. The committer
+ might make further decision on what to do with this PR.
+
+.. image:: images/pr/pr-likely-ok-to-merge.png
+ :align: center
+ :alt: Likely ok to merge the PR with only small set of tests
+
+3c) In case of "core" builds (i. e. when the PR touches some "core" part of Airflow) a message is
+ communicated that this PR needs "full test matrix", the "full tests needed" label is applied
+ automatically and either the contributor might rebase the request to trigger full test build or the
+ committer might re-run the build manually to trigger such full test rebuild. Also a check "in-progress"
+ is added, so that the committer realises that the PR is not yet "green to merge". Pull requests with
+ "full tests needed" label always trigger the full matrix build when rebased or re-run so if the
+ PR gets rebased, it will continue triggering full matrix build.
+
+.. image:: images/pr/pr-full-tests-needed.png
+ :align: center
+ :alt: Full tests are needed for the PR
+
+4) If this or another committer "request changes" in in a previously approved PR with "full tests needed"
+ label, the bot automatically removes the label, moving it back to "run only default set of parameters"
+ mode. For PRs touching core of airflow once the PR gets approved back, the label will be restored.
+ If it was manually set by the committer, it has to be restored manually.
+
+.. note:: Note that setting the labels and adding comments might be delayed, due to limitation of Github Actions,
+ in case of queues, processing of Pull Request reviews might take some time, so it is advised not to merge
+ PR immediately after approval. Luckily, the comments describing the status of the PR trigger notifications
+ for the PRs and they provide good "notification" for the committer to act on a PR that was recently
+ approved.
+
+The PR approval workflow is possible thanks two two custom Github Actions we've developed:
+
+* `Get workflow origin <https://github.com/potiuk/get-workflow-origin/>`_
+* `Label when approved <https://github.com/TobKed/label-when-approved-action>`_
+
+
+Next steps
+----------
+
+We are planning to also propose the approach to other projects from Apache Software Foundation to
+make it a common approach, so that our effort is not limited only to one project.
+
+Discussion about it in `this discussion <https://lists.apache.org/thread.html/r1708881f52adbdae722afb8fea16b23325b739b254b60890e72375e1%40%3Cbuilds.apache.org%3E>`_
diff --git a/images/pr/pr-full-tests-needed.png b/images/pr/pr-full-tests-needed.png
new file mode 100644
index 0000000..c863153
Binary files /dev/null and b/images/pr/pr-full-tests-needed.png differ
diff --git a/images/pr/pr-likely-ok-to-merge.png b/images/pr/pr-likely-ok-to-merge.png
new file mode 100644
index 0000000..9c04dee
Binary files /dev/null and b/images/pr/pr-likely-ok-to-merge.png differ
diff --git a/images/pr/pr-no-tests-needed-comment.png b/images/pr/pr-no-tests-needed-comment.png
new file mode 100644
index 0000000..78a1181
Binary files /dev/null and b/images/pr/pr-no-tests-needed-comment.png differ
diff --git a/images/pr/selective_checks.md5 b/images/pr/selective_checks.md5
new file mode 100644
index 0000000..d4a57c1
--- /dev/null
+++ b/images/pr/selective_checks.md5
@@ -0,0 +1 @@
+2ae1b3fadb26317f4a3531c40b7702f2 images/pr/selective_checks.mermaid
diff --git a/images/pr/selective_checks.mermaid b/images/pr/selective_checks.mermaid
new file mode 100644
index 0000000..e8e8c2a
--- /dev/null
+++ b/images/pr/selective_checks.mermaid
@@ -0,0 +1,35 @@
+%% Licensed to the Apache Software Foundation (ASF) under one
+%% or more contributor license agreements. See the NOTICE file
+%% distributed with this work for additional information
+%% regarding copyright ownership. The ASF licenses this file
+%% to you under the Apache License, Version 2.0 (the
+%% "License"); you may not use this file except in compliance
+%% with the License. You may obtain a copy of the License at
+%%
+%% http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing,
+%% software distributed under the License is distributed on an
+%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+%% KIND, either express or implied. See the License for the
+%% specific language governing permissions and limitations
+%% under the License.
+
+graph LR
+A[PR arrives] --> B[Selective check]
+B --> C{direct push merge?}
+C -->|Yes: enable images| D[Run Full tests<br>+Quarantined<br>run full static checks]
+C -->|No| E[Retrieve changed files]
+E --> F{environment files changed?}
+F -->|Yes: enable images| D
+F -->|No| G{docs changed}
+G -->|Yes: enable image building| H{Chart files changed?}
+G -->|No| H
+H -->|Yes: enable helm tests| I{API files changed?}
+H -->|No| I
+I -->|Yes: enable API tests| J{sources changed?}
+I -->|No| J
+J -->|Yes: enable Pytests| K{determine test type}
+J -->|No| L[skip running test<br/>Run subset of static checks]
+K -->|Core files changed: enable images| D
+K -->|No core files changed: enable images| M[Run selected tests +<br/> Heisentest, Integration, Quarantined<br/>Full static checks]
diff --git a/images/pr/selective_checks.png b/images/pr/selective_checks.png
new file mode 100644
index 0000000..afc2384
Binary files /dev/null and b/images/pr/selective_checks.png differ
[airflow] 22/44: Fix proper SHA in check preventing accidentally
merging PR (#12083)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64393ad508119197907f8a7aa61afa82dd506095
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Nov 4 13:51:04 2020 +0100
Fix proper SHA in check preventing accidentally merging PR (#12083)
The SHA in check was not working for PRs from forks.
(cherry picked from commit d559da14b0a99b369ba027b4b6261321b6b8e03c)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml b/.github/workflows/label_when_reviewed_workflow_run.yml
index 56bc835..cdcbca5 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -93,7 +93,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }}
name: "Please rebase or re-run to run full tests"
status: "in_progress"
- sha: ${{ github.event.pull_request.head.sha }}
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
details_url: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
output: >
{"summary":
[airflow] 19/44: If we build a new image,
we should run more than basic checks (#12070)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f64e98543f230f23b87d3adef72d2a29d5a9b40
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Tue Nov 3 17:42:01 2020 +0000
If we build a new image, we should run more than basic checks (#12070)
This lead to bases such as in #11699 where despite there being changes,
and an image being build, the pre-commit tests were not being run.
(cherry picked from commit 8000ab7375d319e389b6ea973acf8b1085c46bb1)
---
scripts/ci/selective_ci_checks.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/selective_ci_checks.sh b/scripts/ci/selective_ci_checks.sh
index 0faa541..b9a6d74 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -450,7 +450,6 @@ if [[ ${image_build_needed} == "true" ]]; then
else
set_basic_checks_only "true"
fi
-set_basic_checks_only "${image_build_needed}"
set_docs_build "${docs_build_needed}"
run_tests "${tests_needed}"
run_kubernetes_tests "${kubernetes_tests_needed}"
[airflow] 27/44: Fixes undefined variables (#12155)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit fc19025960f659b86d8d989b8b2db22010c9ebfd
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sat Nov 7 12:18:14 2020 +0100
Fixes undefined variables (#12155)
There are few more variables that (if not defined) prevent
from using the CI image directly without breeze or the
CI scripts.
With this change you can run:
`docker run -it apache/airflow:master-python3.6-ci`
and enter the image without errors.
(cherry picked from commit c7f34104516095c487cd6729f830df373ab78fbd)
---
scripts/ci/libraries/_initialization.sh | 4 ++--
scripts/in_container/entrypoint_ci.sh | 2 +-
scripts/in_container/run_init_script.sh | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index e0b4a8a..b58fa2e 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -566,10 +566,10 @@ Detected CI build environment:
Initialization variables:
- INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE}
+ INIT_SCRIPT_FILE: ${INIT_SCRIPT_FILE=}
LOAD_DEFAULT_CONNECTIONS: ${LOAD_DEFAULT_CONNECTIONS}
LOAD_EXAMPLES: ${LOAD_EXAMPLES}
- INSTALL_WHEELS: ${INSTALL_WHEELS}
+ INSTALL_WHEELS: ${INSTALL_WHEELS=}
DISABLE_RBAC: ${DISABLE_RBAC}
Test variables:
diff --git a/scripts/in_container/entrypoint_ci.sh b/scripts/in_container/entrypoint_ci.sh
index 8a7b683..37d3199 100755
--- a/scripts/in_container/entrypoint_ci.sh
+++ b/scripts/in_container/entrypoint_ci.sh
@@ -97,7 +97,7 @@ else
install_released_airflow_version "${INSTALL_AIRFLOW_VERSION}"
fi
-if [[ ${INSTALL_WHEELS} == "true" ]]; then
+if [[ ${INSTALL_WHEELS=} == "true" ]]; then
pip install /dist/*.whl || true
fi
diff --git a/scripts/in_container/run_init_script.sh b/scripts/in_container/run_init_script.sh
index a4b030a..0bd685c 100755
--- a/scripts/in_container/run_init_script.sh
+++ b/scripts/in_container/run_init_script.sh
@@ -23,7 +23,7 @@ if [ -z "${AIRFLOW_BREEZE_CONFIG_DIR+x}" ]; then
export AIRFLOW_BREEZE_CONFIG_DIR="${FILES_DIR}/airflow-breeze-config"
fi
-if [ -z "${INIT_SCRIPT_FILE}" ]; then
+if [ -z "${INIT_SCRIPT_FILE=}" ]; then
export INIT_SCRIPT_FILE="init.sh"
fi
[airflow] 23/44: Update install_mysql.sh (#12101)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b062b9259b81ba9c042a523f5454beb366e134ce
Author: J. Daniel Medeiros <37...@users.noreply.github.com>
AuthorDate: Thu Nov 5 15:32:39 2020 -0100
Update install_mysql.sh (#12101)
After Debian 9 and according to the manual https://manpages.debian.org/stretch/apt/apt-key.8.en.html, after Debian 9 instead of using "apt-key add" a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or "asc" as file extension. Also added better redirection on the apt-key list command.
(cherry picked from commit ded3dbbff0c8e4fabcee62a677394dec0db1aa45)
---
scripts/docker/install_mysql.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/docker/install_mysql.sh b/scripts/docker/install_mysql.sh
index 1ddc76b..0fbb1da 100755
--- a/scripts/docker/install_mysql.sh
+++ b/scripts/docker/install_mysql.sh
@@ -46,10 +46,10 @@ if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
gpg --keyserver "${keyserver}" --recv-keys "${KEY}" && break
done
set -e
- gpg --export "${KEY}" | apt-key add -
+ gpg --export "${KEY}" > /etc/apt/trusted.gpg.d/mysql.gpg
gpgconf --kill all
rm -rf "${GNUPGHOME}"
- apt-key list > /dev/null
+ apt-key list > /dev/null 2>&1
echo "deb http://repo.mysql.com/apt/debian/ buster mysql-5.7" | tee -a /etc/apt/sources.list.d/mysql.list
apt-get update
apt-get install --no-install-recommends -y "${packages[@]}"
[airflow] 30/44: Adds extra check while the selective checks are
run (#12178)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 20512aff626a04a418ccef5861b6856ac0c3e215
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Nov 8 13:25:32 2020 +0100
Adds extra check while the selective checks are run (#12178)
The selective checks are run in "workflow_run" because
they need to be able to set label and make comments, however
status of those checks are not displayed in GitHub and in
cases of small PRs the "merge" button might be green before
the status complete.
This PR adds additional check that is always completed after
the "worfklow_run" finishes it's job. This will prevent
accidental merges before the check completes.
(cherry picked from commit 63ac07d9c735b1ccc8aa5fa974a260cc944cc539)
---
.../workflows/label_when_reviewed_workflow_run.yml | 29 ++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml b/.github/workflows/label_when_reviewed_workflow_run.yml
index cdcbca5..c2359c8 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -35,6 +35,20 @@ jobs:
with:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
+ - name: Initiate Selective Build check
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+ id: selective-build-check
+ with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ name: "Selective build check"
+ status: "in_progress"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ details_url: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ output: >
+ {"summary":
+ "Checking selective status of the build in
+ [the run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
+ "}
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: >
@@ -125,3 +139,18 @@ jobs:
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
require_committers_approval: 'true'
comment: "The PR is ready to be merged. No tests are needed!"
+ - name: Update Selective Build check
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
+ if: always()
+ with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+ check_id: ${{ steps.selective-build-check.outputs.check_id }}
+ status: "completed"
+ sha: ${{ steps.source-run-info.outputs.sourceHeadSha }}
+ conclusion: ${{ job.status }}
+ details_url: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ output: >
+ {"summary":
+ "Checking selective status of the build completed in
+ [the run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
+ "}
[airflow] 20/44: Fixes problem with building a PROD image (#12080)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f4323092e6544cf1c28e97c3f461ef0ed282e3ef
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Nov 4 09:31:00 2020 +0100
Fixes problem with building a PROD image (#12080)
The change #12050 that aimed at automation of Docker images
building in DockerHub had an undesired effect of overriding the
production image tag with the CI one.
This is fixed by this PR.
(cherry picked from commit d971c1c0e141f285a3cdd3f029028ab62df6f81a)
---
scripts/ci/images/ci_build_dockerhub.sh | 6 +++++-
scripts/ci/libraries/_build_images.sh | 9 ++++++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/scripts/ci/images/ci_build_dockerhub.sh b/scripts/ci/images/ci_build_dockerhub.sh
index 7d92abc..ac7764f 100755
--- a/scripts/ci/images/ci_build_dockerhub.sh
+++ b/scripts/ci/images/ci_build_dockerhub.sh
@@ -39,7 +39,6 @@ echo
echo "DOCKER_TAG=${DOCKER_TAG}"
-
[[ ${DOCKER_TAG:=} =~ ${DEFAULT_BRANCH}-python([0-9.]*)(.*) ]] && export PYTHON_MAJOR_MINOR_VERSION=${BASH_REMATCH[1]}
: "${PYTHON_MAJOR_MINOR_VERSION:?"The tag '${DOCKER_TAG}' should follow the pattern .*-pythonX.Y[-ci]"}"
@@ -47,6 +46,11 @@ echo "DOCKER_TAG=${DOCKER_TAG}"
echo "Detected PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}"
echo
+FORCE_AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG}"
+export FORCE_AIRFLOW_PROD_BASE_TAG
+
+readonly FORCE_AIRFLOW_PROD_BASE_TAG
+
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index 387372a..a6dbae0 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -327,8 +327,7 @@ function build_images::get_docker_image_names() {
export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
# CI image base tag
- export AIRFLOW_CI_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci}"
-
+ export AIRFLOW_CI_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
# CI image to build
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
@@ -336,7 +335,11 @@ function build_images::get_docker_image_names() {
export AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
# Base production image tag - used to build kubernetes tag as well
- export AIRFLOW_PROD_BASE_TAG="${DOCKER_TAG=${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}}"
+ if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
+ export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
+ else
+ export AIRFLOW_PROD_BASE_TAG="${FORCE_AIRFLOW_PROD_BASE_TAG}"
+ fi
# PROD image to build
export AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
[airflow] 43/44: Simplifies check whether the CI image should be
rebuilt (#12181)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 23c62063e11b0cb0f1a9ab763fd0e3c106711904
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Fri Nov 13 22:21:39 2020 +0100
Simplifies check whether the CI image should be rebuilt (#12181)
Rather than counting changed layers in the image (which was
enigmatic, difficult and prone to some magic number) we rely now
on random file generated while building the image.
We are using the docker image caching mechanism here. The random
file will be regenerated only when the previous layer (which is
about installling Airflow dependencies for the first time) gets
rebuild. And for us this is the indication, that the building
the image will take quite some time. This layer should be
relatively static - even if setup.py changes the CI image is
designed in the way that the first time installation of Airflow
dependencies is not invalidated.
This should lead to faster and less frequent rebuild for people
using Breeze and static checks.
(cherry picked from commit 167b9b9889ac5481b21cb35c6cdef5869b8ab713)
---
Dockerfile.ci | 9 +-
breeze | 6 +-
manifests/.gitignore | 2 +-
scripts/ci/libraries/_build_images.sh | 163 ++++++++++++++++----------------
scripts/ci/libraries/_initialization.sh | 19 +++-
5 files changed, 106 insertions(+), 93 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 7699706..f06087b 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -132,7 +132,9 @@ ARG RUNTIME_APT_DEPS="\
sqlite3 \
tmux \
unzip \
- vim"
+ vim \
+ xxd"
+ENV RUNTIME_APT_DEP=${RUNTIME_APT_DEPS}
ARG ADDITIONAL_RUNTIME_APT_DEPS=""
ENV ADDITIONAL_RUNTIME_APT_DEPS=${ADDITIONAL_RUNTIME_APT_DEPS}
@@ -275,6 +277,11 @@ RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" ]]; then \
fi
+# Generate random hex dump file so that we can determine whether it's faster to rebuild the image
+# using current cache (when our dump is the same as the remote onb) or better to pull
+# the new image (when it is different)
+RUN head -c 30 /dev/urandom | xxd -ps >/build-cache-hash
+
# Link dumb-init for backwards compatibility (so that older images also work)
RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
diff --git a/breeze b/breeze
index 55412d0..175a4ab 100755
--- a/breeze
+++ b/breeze
@@ -2873,11 +2873,9 @@ function breeze::run_build_command() {
build_images::prepare_prod_build
build_images::build_prod_images
else
+
build_images::prepare_ci_build
- md5sum::calculate_md5sum_for_all_files
- build_images::build_ci_image
- md5sum::update_all_md5
- build_images::build_ci_image_manifest
+ build_images::rebuild_ci_image_if_needed
fi
;;
cleanup_image | run_exec)
diff --git a/manifests/.gitignore b/manifests/.gitignore
index a6c57f5..72e8ffc 100644
--- a/manifests/.gitignore
+++ b/manifests/.gitignore
@@ -1 +1 @@
-*.json
+*
diff --git a/scripts/ci/libraries/_build_images.sh b/scripts/ci/libraries/_build_images.sh
index a2577cb..52b8a84 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -136,10 +136,18 @@ function build_images::confirm_image_rebuild() {
;;
esac
elif [[ -t 0 ]]; then
+ echo
+ echo
+ echo "Make sure that you rebased to latest master before rebuilding!"
+ echo
# Check if this script is run interactively with stdin open and terminal attached
"${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}"
RES=$?
elif [[ ${DETECTED_TERMINAL:=$(tty)} != "not a tty" ]]; then
+ echo > "${DETECTED_TERMINAL}"
+ echo > "${DETECTED_TERMINAL}"
+ echo "Make sure that you rebased to latest master before rebuilding!" > "${DETECTED_TERMINAL}"
+ echo > "${DETECTED_TERMINAL}"
# Make sure to use output of tty rather than stdin/stdout when available - this way confirm
# will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
# shellcheck disable=SC2094
@@ -151,6 +159,10 @@ function build_images::confirm_image_rebuild() {
export DETECTED_TERMINAL=/dev/tty
# Make sure to use /dev/tty first rather than stdin/stdout when available - this way confirm
# will works also in case of pre-commits (git does not pass stdin/stdout to pre-commit hooks)
+ echo > "${DETECTED_TERMINAL}"
+ echo > "${DETECTED_TERMINAL}"
+ echo "Make sure that you rebased to latest master before rebuilding!" > "${DETECTED_TERMINAL}"
+ echo > "${DETECTED_TERMINAL}"
# shellcheck disable=SC2094
"${AIRFLOW_SOURCES}/confirm" "${ACTION} image ${THE_IMAGE_TYPE}-python${PYTHON_MAJOR_MINOR_VERSION}" \
<"${DETECTED_TERMINAL}" >"${DETECTED_TERMINAL}"
@@ -193,113 +205,95 @@ function build_images::confirm_image_rebuild() {
# We cannot use docker registry APIs as they are available only with authorisation
# But this image can be pulled without authentication
function build_images::build_ci_image_manifest() {
- docker inspect "${AIRFLOW_CI_IMAGE}" >"manifests/${AIRFLOW_CI_BASE_TAG}.json"
docker build \
- --build-arg AIRFLOW_CI_BASE_TAG="${AIRFLOW_CI_BASE_TAG}" \
--tag="${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" \
-f- . <<EOF
-ARG AIRFLOW_CI_BASE_TAG
FROM scratch
-COPY "manifests/${AIRFLOW_CI_BASE_TAG}.json" .
+COPY "manifests/local-build-cache-hash" /build-cache-hash
CMD ""
EOF
}
#
-# Retrieves information about layers in the local IMAGE
-# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_LOCAL_SHA
+# Retrieves information about build cache hash random file from the local image
#
-function build_images::get_local_image_info() {
- TMP_MANIFEST_LOCAL_JSON=$(mktemp)
- TMP_MANIFEST_LOCAL_SHA=$(mktemp)
+function build_images::get_local_build_cache_hash() {
+
set +e
# Remove the container just in case
- docker rm --force "local-airflow-manifest" 2>/dev/null >/dev/null
- # Create manifest from the local manifest image
- if ! docker create --name "local-airflow-manifest" "${AIRFLOW_CI_LOCAL_MANIFEST_IMAGE}" 2>/dev/null; then
- echo
- echo "Local manifest image not available"
- echo
+ docker rm --force "local-airflow-ci-container" 2>/dev/null >/dev/null
+ if ! docker create --name "local-airflow-ci-container" "${AIRFLOW_CI_IMAGE}" 2>/dev/null; then
+ >&2 echo
+ >&2 echo "Local airflow CI image not available"
+ >&2 echo
LOCAL_MANIFEST_IMAGE_UNAVAILABLE="true"
+ export LOCAL_MANIFEST_IMAGE_UNAVAILABLE
+ touch "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}"
return
fi
- # Create manifest from the local manifest image
- docker cp "local-airflow-manifest:${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_LOCAL_JSON}"
- sed 's/ *//g' "${TMP_MANIFEST_LOCAL_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_LOCAL_SHA}"
- docker rm --force "local-airflow-manifest" 2>/dev/null >/dev/null
+ docker cp "local-airflow-ci-container:/build-cache-hash" \
+ "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}" 2> /dev/null \
+ || touch "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}"
set -e
+ echo
+ echo "Local build cache hash: '$(cat "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}")'"
+ echo
}
-#
-# Retrieves information about layers in the remote IMAGE
-# it stores list of SHAs of image layers in the file pointed at by TMP_MANIFEST_REMOTE_SHA
-# This cannot be done easily with existing APIs of Dockerhub because they require additional authentication
-# even for public images. Therefore instead we are downloading a specially prepared manifest image
-# which is built together with the main image. This special manifest image is prepared during
-# building of the main image and contains single JSON file being result of docker inspect on that image
-# This image is from scratch so it is very tiny
-function build_images::get_remote_image_info() {
+# Retrieves information about the build cache hash random file from the remote image.
+# We actually use manifest image for that, which is a really, really small image to pull!
+# The problem is that inspecting information about remote image cannot be done easily with existing APIs
+# of Dockerhub because they require additional authentication even for public images.
+# Therefore instead we are downloading a specially prepared manifest image
+# which is built together with the main image and pushed with it. This special manifest image is prepared
+# during building of the main image and contains single file which is randomly built during the docker
+# build in the right place in the image (right after installing all dependencies of Apache Airflow
+# for the first time). When this random file gets regenerated it means that either base image has
+# changed or some of the earlier layers was modified - which means that it is usually faster to pull
+# that image first and then rebuild it - because this will likely be faster
+function build_images::get_remote_image_build_cache_hash() {
set +e
# Pull remote manifest image
if ! docker pull "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" 2>/dev/null >/dev/null; then
- echo >&2
- echo >&2 "Remote docker registry unreachable"
- echo >&2
+ >&2 echo
+ >&2 echo "Remote docker registry unreachable"
+ >&2 echo
REMOTE_DOCKER_REGISTRY_UNREACHABLE="true"
+ export REMOTE_DOCKER_REGISTRY_UNREACHABLE
+ touch "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}"
return
fi
set -e
-
- # Docker needs the file passed to --cidfile to not exist, so we can't use mktemp
- TMP_CONTAINER_DIR="$(mktemp -d)"
- TMP_CONTAINER_ID="${TMP_CONTAINER_DIR}/remote-airflow-manifest-$$.container_id"
- FILES_TO_CLEANUP_ON_EXIT+=("$TMP_CONTAINER_ID")
-
- TMP_MANIFEST_REMOTE_JSON=$(mktemp)
- TMP_MANIFEST_REMOTE_SHA=$(mktemp)
- # Create container out of the manifest image without running it
- docker create --cidfile "${TMP_CONTAINER_ID}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" 2>/dev/null >/dev/null
+ # Create container dump out of the manifest image without actually running it
+ docker create --cidfile "${REMOTE_IMAGE_CONTAINER_ID_FILE}" "${AIRFLOW_CI_REMOTE_MANIFEST_IMAGE}" \
+ 2>/dev/null >/dev/null || true
# Extract manifest and store it in local file
- docker cp "$(cat "${TMP_CONTAINER_ID}"):${AIRFLOW_CI_BASE_TAG}.json" "${TMP_MANIFEST_REMOTE_JSON}"
- # Filter everything except SHAs of image layers
- sed 's/ *//g' "${TMP_MANIFEST_REMOTE_JSON}" | grep '^"sha256:' >"${TMP_MANIFEST_REMOTE_SHA}"
- docker rm --force "$(cat "${TMP_CONTAINER_ID}")" 2>/dev/null >/dev/null
+ docker cp "$(cat "${REMOTE_IMAGE_CONTAINER_ID_FILE}"):/build-cache-hash" \
+ "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}" 2> /dev/null \
+ || touch "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}"
+ docker rm --force "$(cat "${REMOTE_IMAGE_CONTAINER_ID_FILE}")" 2>/dev/null || true
+ echo
+ echo "Remote build cache hash: '$(cat "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}")'"
+ echo
}
-# The Number determines the cut-off between local building time and pull + build time.
-# It is a bit experimental and it will have to be kept
-# updated as we keep on changing layers. The cut-off point is at the moment when we do first
-# pip install "https://github.com/apache/airflow/archive/${AIRFLOW_BRANCH}.tar...
-# you can get it via this command:
-# docker history --no-trunc apache/airflow:master-python3.6-ci | \
-# grep ^sha256 | grep -n "pip uninstall" | awk 'BEGIN {FS=":"} {print $1 }'
-#
-# This command returns the number of layer in docker history where pip uninstall is called. This is the
-# line that will take a lot of time to run and at this point it's worth to pull the image from repo
-# if there are at least NN changed layers in your docker file, you should pull the image.
-#
-# Note that this only matters if you have any of the important files changed since the last build
-# of your image such as Dockerfile.ci, setup.py etc.
-#
-MAGIC_CUT_OFF_NUMBER_OF_LAYERS=36
-
# Compares layers from both remote and local image and set FORCE_PULL_IMAGES to true in case
# More than the last NN layers are different.
-function build_images::compare_layers() {
- NUM_DIFF=$(diff -y --suppress-common-lines "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_SHA}" |
- wc -l || true)
- rm -f "${TMP_MANIFEST_REMOTE_JSON}" "${TMP_MANIFEST_REMOTE_SHA}" "${TMP_MANIFEST_LOCAL_JSON}" "${TMP_MANIFEST_LOCAL_SHA}"
- echo
- echo "Number of layers different between the local and remote image: ${NUM_DIFF}"
- echo
- # This is where setup py is rebuilt - it will usually take a looooot of time to build it, so it is
- # Better to pull here
- if ((NUM_DIFF >= MAGIC_CUT_OFF_NUMBER_OF_LAYERS)); then
+function build_images::compare_local_and_remote_build_cache_hash() {
+ set +e
+ local remote_hash
+ remote_hash=$(cat "${REMOTE_IMAGE_BUILD_CACHE_HASH_FILE}")
+ local local_hash
+ local_hash=$(cat "${LOCAL_IMAGE_BUILD_CACHE_HASH_FILE}")
+
+ if [[ ${remote_hash} != "${local_hash}" ||
+ ${local_hash} == "" ]]; then
echo
echo
- echo "WARNING! Your image and the dockerhub image differ significantly"
+ echo "Your image and the dockerhub have different or missing build cache hashes."
+ echo "Local hash: '${local_hash}'. Remote hash: '${remote_hash}'."
echo
echo "Forcing pulling the images. It will be faster than rebuilding usually."
echo "You can avoid it by setting SKIP_CHECK_REMOTE_IMAGE to true"
@@ -307,9 +301,10 @@ function build_images::compare_layers() {
export FORCE_PULL_IMAGES="true"
else
echo
- echo "No need to pull the image. Local rebuild will be faster"
+ echo "No need to pull the image. Yours and remote cache hashes are the same!"
echo
fi
+ set -e
}
# Prints summary of the build parameters
@@ -335,7 +330,6 @@ function build_images::get_docker_image_names() {
# CI image to build
export AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_CI_BASE_TAG}"
-
# Base production image tag - used to build kubernetes tag as well
if [[ ${FORCE_AIRFLOW_PROD_BASE_TAG=} == "" ]]; then
export AIRFLOW_PROD_BASE_TAG="${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
@@ -396,6 +390,9 @@ function build_images::prepare_ci_build() {
# In case rebuild is needed, it determines (by comparing layers in local and remote image)
# Whether pull is needed before rebuild.
function build_images::rebuild_ci_image_if_needed() {
+ verbosity::print_info
+ verbosity::print_info "Checking if pull or just build for ${THE_IMAGE_TYPE} is needed."
+ verbosity::print_info
if [[ -f "${BUILT_CI_IMAGE_FLAG_FILE}" ]]; then
verbosity::print_info
verbosity::print_info "${THE_IMAGE_TYPE} image already built locally."
@@ -418,6 +415,7 @@ function build_images::rebuild_ci_image_if_needed() {
local needs_docker_build="false"
md5sum::check_if_docker_build_is_needed
+ build_images::get_local_build_cache_hash
if [[ ${needs_docker_build} == "true" ]]; then
if [[ ${SKIP_CHECK_REMOTE_IMAGE:=} != "true" && ${DOCKER_CACHE} == "pulled" ]]; then
# Check if remote image is different enough to force pull
@@ -427,14 +425,12 @@ function build_images::rebuild_ci_image_if_needed() {
echo
echo "Checking if the remote image needs to be pulled"
echo
- build_images::get_remote_image_info
- if [[ ${REMOTE_DOCKER_REGISTRY_UNREACHABLE:=} != "true" ]]; then
- build_images::get_local_image_info
- if [[ ${LOCAL_MANIFEST_IMAGE_UNAVAILABLE:=} != "true" ]]; then
- build_images::compare_layers
- else
- FORCE_PULL_IMAGES="true"
- fi
+ build_images::get_remote_image_build_cache_hash
+ if [[ ${REMOTE_DOCKER_REGISTRY_UNREACHABLE:=} != "true" && \
+ ${LOCAL_MANIFEST_IMAGE_UNAVAILABLE:=} != "true" ]]; then
+ build_images::compare_local_and_remote_build_cache_hash
+ else
+ FORCE_PULL_IMAGES="true"
fi
fi
SKIP_REBUILD="false"
@@ -453,6 +449,7 @@ function build_images::rebuild_ci_image_if_needed() {
verbosity::print_info "Build start: ${THE_IMAGE_TYPE} image."
verbosity::print_info
build_images::build_ci_image
+ build_images::get_local_build_cache_hash
md5sum::update_all_md5
build_images::build_ci_image_manifest
verbosity::print_info
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index b58fa2e..396bded 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -186,9 +186,9 @@ function initialization::initialize_files_for_rebuild_check() {
"Dockerfile.ci"
".dockerignore"
"airflow/version.py"
- "airflow/www/package.json"
- "airflow/www/yarn.lock"
- "airflow/www/webpack.config.js"
+ "airflow/www_rbac/package.json"
+ "airflow/www_rbac/yarn.lock"
+ "airflow/www_rbac/webpack.config.js"
)
}
@@ -452,6 +452,12 @@ function initialization::initialize_test_variables() {
export TEST_TYPE=${TEST_TYPE:=""}
}
+function initialization::initialize_build_image_variables() {
+ REMOTE_IMAGE_CONTAINER_ID_FILE="${AIRFLOW_SOURCES}/manifests/remote-airflow-manifest-image"
+ LOCAL_IMAGE_BUILD_CACHE_HASH_FILE="${AIRFLOW_SOURCES}/manifests/local-build-cache-hash"
+ REMOTE_IMAGE_BUILD_CACHE_HASH_FILE="${AIRFLOW_SOURCES}/manifests/remote-build-cache-hash"
+}
+
# Common environment that is initialized by both Breeze and CI scripts
function initialization::initialize_common_environment() {
initialization::create_directories
@@ -469,6 +475,7 @@ function initialization::initialize_common_environment() {
initialization::initialize_git_variables
initialization::initialize_github_variables
initialization::initialize_test_variables
+ initialization::initialize_build_image_variables
}
function initialization::set_default_python_version_if_empty() {
@@ -750,6 +757,10 @@ function initialization::make_constants_read_only() {
readonly BUILT_CI_IMAGE_FLAG_FILE
readonly INIT_SCRIPT_FILE
+ readonly REMOTE_IMAGE_CONTAINER_ID_FILE
+ readonly LOCAL_IMAGE_BUILD_CACHE_HASH_FILE
+ readonly REMOTE_IMAGE_BUILD_CACHE_HASH_FILE
+
}
# converts parameters to json array
@@ -772,7 +783,7 @@ function initialization::ga_output() {
function initialization::ga_env() {
if [[ ${GITHUB_ENV=} != "" ]]; then
- echo "${1}=${2}" >> "${GITHUB_ENV}"
+ echo "${1}=${2}" >>"${GITHUB_ENV}"
fi
}
[airflow] 39/44: Added k9s as integrated tool to help with
kubernetes testing (#12163)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d8db36a1b1bee478af4b4ca3e8ac6471da557a96
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Wed Nov 11 17:15:02 2020 +0100
Added k9s as integrated tool to help with kubernetes testing (#12163)
The K9s is fantastic tool that helps to debug a running k8s
instance. It is terminal-based windowed CLI that makes you
several times more productive comparing to using kubectl
commands. We've integrated k9s (it is run as a docker container
and downloaded on demand). We've also separated out KUBECONFIG
of the integrated kind cluster so that it does not mess with
kubernetes configuration you might already have.
Also - together with that the "surrounding" of the kubernetes
tests were simplified and improved so that the k9s integration
can be utilized well. Instead of kubectl port forwarding (which
caused multitude of problems) we are now utilizing kind's
portMapping feature + custom NodePort resource that maps
port 8080 to 30007 NodePort which in turn maps it to 8080
port of the Webserver. This way we do not have to establish
an external kubectl port forward which is prone to error and
management - everything is brought up when Airflow gets
deployed to the Kind Cluster and shuts down when the Kind
cluster is stopped.
Yet another problem fixed was killing of postgres by one of the
kubernetes tests ('test_integration_run_dag_with_scheduler_failure').
Instead of just killing the scheduler it killed all pods - including
the Postgres one (it was named 'airflow-postgres.*'). That caused
various problems, as the database could be left in a strange state.
I changed the tests to do what it claimed was doing - so killing only the
scheduler during the test. This seemed to improve the stability
of tests immensely in my local setup.
(cherry picked from commit 21999dd56e9dbe9f7f9e25961954c5677c3c7c58)
---
.github/workflows/ci.yml | 17 +-
BREEZE.rst | 11 +-
TESTING.rst | 151 ++++-
breeze | 9 +
breeze-complete | 5 +-
chart/requirements.lock | 4 +-
images/testing/k9s.png | Bin 0 -> 238713 bytes
images/testing/kubeconfig-env.png | Bin 0 -> 231280 bytes
images/testing/kubernetes-virtualenv.png | Bin 0 -> 110011 bytes
images/testing/pytest-runner.png | Bin 0 -> 131589 bytes
images/testing/run-test.png | Bin 0 -> 140728 bytes
kubernetes_tests/test_kubernetes_executor.py | 7 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 672 ++++++++-------------
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 7 +-
...up_cluster_and_deploy_airflow_to_kubernetes.sh} | 3 +-
scripts/ci/kubernetes/kind-cluster-conf.yaml | 5 +
.../{kind-cluster-conf.yaml => nodeport.yaml} | 30 +-
...oy_app_to_kubernetes.sh => redeploy_airflow.sh} | 6 +-
scripts/ci/libraries/_kind.sh | 126 ++--
19 files changed, 516 insertions(+), 537 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 6c854a1..81890a7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -636,23 +636,14 @@ jobs:
python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- - name: "Setup Kind Cluster ${{ env.KIND_VERSION }}"
- uses: engineerd/setup-kind@v0.4.0
- with:
- version: "${{ env.KIND_VERSION }}"
- name: airflow-python-${{matrix.python-version}}-${{matrix.kubernetes-version}}
- config: "scripts/ci/kubernetes/kind-cluster-conf.yaml"
- name: "Prepare PROD Image"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- - name: "Deploy airflow to cluster"
- id: deploy-app
- run: ./scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+ - name: "Setup cluster and deploy Airflow"
+ id: setp-cluster-deploy-app
+ run: ./scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
env:
# We have the right image pulled already by the previous step
SKIP_BUILDING_PROD_IMAGE: "true"
- # due to some instabilities, in CI we try to increase port numbers when trying to establish
- # port forwarding
- INCREASE_PORT_NUMBER_FOR_KUBERNETES: "true"
- name: "Cache virtualenv for kubernetes testing"
uses: actions/cache@v2
env:
@@ -669,7 +660,7 @@ jobs:
key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}\
-${{ needs.build-info.outputs.defaultKindVersion }}\
-${{ needs.build-info.outputs.defaultHelmVersion }}\
--$${{ matrix.kubernetes-version }}"
+-${{ matrix.kubernetes-version }}"
- name: "Kubernetes Tests"
run: ./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
- name: "Upload KinD logs"
diff --git a/BREEZE.rst b/BREEZE.rst
index c3c2d95..ce7dc6a 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1188,6 +1188,7 @@ This is the current syntax for `./breeze <./breeze>`_:
image building time in production image and at container entering time for CI image. One of:
1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2
+ wheel
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
@@ -1712,7 +1713,14 @@ This is the current syntax for `./breeze <./breeze>`_:
to the cluster so you can also pass appropriate build image flags that will influence
rebuilding the production image. Operation is one of:
- start stop restart status deploy test shell
+ start stop restart status deploy test shell k9s
+
+ The last two operations - shell and k9s allow you to perform interactive testing with
+ kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
+ another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
+ way to debug the kubernetes deployments.
+
+ You can read more about k9s at https://k9scli.io/
Flags:
@@ -2087,6 +2095,7 @@ This is the current syntax for `./breeze <./breeze>`_:
image building time in production image and at container entering time for CI image. One of:
1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3 1.10.2
+ wheel
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
diff --git a/TESTING.rst b/TESTING.rst
index c8b170b..f6c6c10 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -418,6 +418,14 @@ can also decide to only run tests with ``-m quarantined`` flag to run only those
Running Tests with Kubernetes
=============================
+Airflow has tests that are run against real kubernetes cluster. We are using
+`Kind <https://kind.sigs.k8s.io/>`_ to create and run the cluster. We integrated the tools to start/stop/
+deploy and run the cluster tests in our repository and into Breeze development environment.
+
+Configuration for the cluster is kept in ``./build/.kube/config`` file in your Airflow source repository and
+our scripts set the ``KUBECONFIG`` variable to it. If you want to interact with the Kind cluster created
+you can do it from outside of the scripts by exporting this variable and point it to this file.
+
Starting Kubernetes Cluster
---------------------------
@@ -425,7 +433,7 @@ For your testing you manage Kind cluster with ``kind-cluster`` breeze command:
.. code-block:: bash
- ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell ]
+ ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
The command allows you to start/stop/recreate/status Kind Kubernetes cluster, deploy Airflow via Helm
chart as well as interact with the cluster (via test and shell commands).
@@ -444,11 +452,11 @@ Deploying Airflow to Kubernetes Cluster
Deploying Airflow to the Kubernetes cluster created is also done via ``kind-cluster deploy`` breeze command:
-.. code-block:: bash`
+.. code-block:: bash
./breeze kind-cluster deploy
-The deploy commands performs tthose steps:
+The deploy commands performs those steps:
1. It rebuilds the latest ``apache/airflow:master-pythonX.Y`` production images using the
latest sources using local cachine. It also adds example DAGs to the image, so that they do not
@@ -465,20 +473,63 @@ Running tests with Kubernetes Cluster
You can either run all tests or you can select which tests to run. You can also enter interactive virtualenv
to run the tests manually one by one.
-.. code-block:: bash
+Running kubernetes tests via shell:
- Running kubernetes tests
+.. code-block:: bash
./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh - runs all kubernetes tests
./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh TEST [TEST ...] - runs selected kubernetes tests (from kubernetes_tests folder)
+
+
+Running kubernetes tests via breeze:
+
+.. code-block:: bash
+
+ ./breeze kind-cluster test
+ ./breeze kind-cluster test -- TEST TEST [TEST ...]
+
+
+Entering shell with Kubernetes Cluster
+--------------------------------------
+
+This shell is prepared to run kubernetes tests interactively. It has ``kubectl`` and ``kind`` cli tools
+available in the path, it has also activated virtualenv environment that allows you to run tests via pytest.
+
+You can enter the shell via those scripts
+
./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh [-i|--interactive] - Activates virtual environment ready to run tests and drops you in
./scripts/ci/kubernetes/ci_run_kubernetes_tests.sh [--help] - Prints this help message
-You can also run the same tests command with Breeze, using ``kind-cluster test`` command (to run all
-kubernetes tests) and with ``kind-cluster shell`` command you can enter interactive shell when you can
-run tests.
+.. code-block:: bash
+
+ ./breeze kind-cluster shell
+
+K9s CLI - debug kubernetes in style!
+------------------------------------
+
+Breeze has built-in integration with fantastic k9s CLI tool, that allows you to debug the kubernetes
+installation effortlessly and in style. K9S provides terminal (but windowed) CLI that allows you to
+easily observe what's going on in the kubernetes instance, observe the resources defined (pods, secrets,
+custom resource definitions), enter shell for the Pods/Containers running, see the log files and more.
+
+You can read more about k9s at `https://k9scli.io/ <https://k9scli.io/>`_
+
+Here is the screenshot of k9s tools in operation:
+
+.. image:: images/testing/k9s.png
+ :align: center
+ :alt: K9S tool
+
+
+You can enter the k9s tool via breeze (after you deployed Airflow):
+
+.. code-block:: bash
+
+ ./breeze kind-cluster k9s
+
+You can exit k9s by pressing Ctrl-C.
Typical testing pattern for Kubernetes tests
--------------------------------------------
@@ -578,7 +629,6 @@ This prepares and enters the virtualenv in ``.build/.kubernetes_venv`` folder:
./breeze kind-cluster shell
-
Once you enter the environment you receive this information:
@@ -595,12 +645,67 @@ Once you enter the environment you receive this information:
You are entering the virtualenv now. Type exit to exit back to the original shell
+In a separate terminal you can open the k9s CLI:
+
+.. code-block:: bash
+
+ ./breeze kind-cluster k9s
+
+Use it to observe what's going on in your cluster.
+
+6. Debugging in IntelliJ/PyCharm
+
+It is very easy to running/debug Kubernetes tests with IntelliJ/PyCharm. Unlike the regular tests they are
+in ``kubernetes_tests`` folder and if you followed the previous steps and entered the shell using
+``./breeze kind-cluster shell`` command, you can setup your IDE very easily to run (and debug) your
+tests using the standard IntelliJ Run/Debug feature. You just need a few steps:
+
+a) Add the virtualenv as interpreter for the project:
+
+.. image:: images/testing/kubernetes-virtualenv.png
+ :align: center
+ :alt: Kubernetes testing virtualenv
+
+The virtualenv is created in your "Airflow" source directory in ``.build/.kubernetes_venv/`` folder and you
+have to find ``python`` binary and choose it when selecting interpreter.
+
+b) Choose pytest as test runner:
+
+.. image:: images/testing/pytest-runner.png
+ :align: center
+ :alt: Pytest runner
+
+c) Run/Debug tests using standard "Run/Debug" feature of IntelliJ
+
+.. image:: images/testing/run-tests.png
+ :align: center
+ :alt: Run/Debug tests
+
+
+NOTE! The first time you run it, it will likely fail with
+``kubernetes.config.config_exception.ConfigException``:
+``Invalid kube-config file. Expected key current-context in kube-config``. You need to add KUBECONFIG
+environment variabl copying it from the result of "./breeze kind-cluster test":
+
+.. code-block:: bash
+
+ echo ${KUBECONFIG}
+
+ /home/jarek/code/airflow/.build/.kube/config
+
+
+.. image:: images/testing/kubeconfig-env.png
+ :align: center
+ :alt: Run/Debug tests
+
+
+The configuration for kubernetes is stored in your "Airflow" source directory in ".build/.kube/config" file
+and this is where KUBECONFIG env should point to.
You can iterate with tests while you are in the virtualenv. All the tests requiring kubernetes cluster
are in "kubernetes_tests" folder. You can add extra ``pytest`` parameters then (for example ``-s`` will
print output generated test logs and print statements to the terminal immediately.
-
.. code-block:: bash
pytest kubernetes_tests/test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure -s
@@ -609,6 +714,30 @@ print output generated test logs and print statements to the terminal immediatel
You can modify the tests or KubernetesPodOperator and re-run them without re-deploying
airflow to KinD cluster.
+
+Sometimes there are side effects from running tests. You can run ``redeploy_airflow.sh`` without
+recreating the whole cluster. This will delete the whole namespace, including the database data
+and start a new Airflow deployment in the cluster.
+
+.. code-block:: bash
+
+ ./scripts/ci/redeploy_airflow.sh
+
+If needed you can also delete the cluster manually:
+
+
+.. code-block:: bash
+
+ kind get clusters
+ kind delete clusters <NAME_OF_THE_CLUSTER>
+
+Kind has also useful commands to inspect your running cluster:
+
+.. code-block:: text
+
+ kind --help
+
+
However, when you change Airflow Kubernetes executor implementation you need to redeploy
Airflow to the cluster.
@@ -617,7 +746,7 @@ Airflow to the cluster.
./breeze kind-cluster deploy
-5. Stop KinD cluster when you are done
+7. Stop KinD cluster when you are done
.. code-block:: bash
diff --git a/breeze b/breeze
index 3498c64..55412d0 100755
--- a/breeze
+++ b/breeze
@@ -1684,6 +1684,13 @@ ${CMDNAME} kind-cluster [FLAGS] OPERATION
${FORMATTED_KIND_OPERATIONS}
+ The last two operations - shell and k9s allow you to perform interactive testing with
+ kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
+ another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
+ way to debug the kubernetes deployments.
+
+ You can read more about k9s at https://k9scli.io/
+
Flags:
$(breeze::flag_airflow_variants)
$(breeze::flag_build_docker_images)
@@ -2901,6 +2908,8 @@ function breeze::run_build_command() {
echo "Run Kubernetes tests with the KinD cluster "
elif [[ ${KIND_CLUSTER_OPERATION} == "shell" ]]; then
echo "Enter an interactive shell for kubernetes testing"
+ elif [[ ${KIND_CLUSTER_OPERATION} == "k9s" ]]; then
+ echo "Run k9s cli to debug in style"
elif [[ -z ${KIND_CLUSTER_OPERATION=} ]]; then
echo
echo "Please provide an operation to run"
diff --git a/breeze-complete b/breeze-complete
index 4855c86..94854ad 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -32,7 +32,7 @@ _breeze_allowed_helm_versions="v3.2.4"
_breeze_allowed_kind_versions="v0.8.0"
_breeze_allowed_mysql_versions="5.6 5.7"
_breeze_allowed_postgres_versions="9.6 10 11 12 13"
-_breeze_allowed_kind_operations="start stop restart status deploy test shell"
+_breeze_allowed_kind_operations="start stop restart status deploy test shell k9s"
_breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm"
# shellcheck disable=SC2034
@@ -60,6 +60,7 @@ _breeze_allowed_install_airflow_versions=$(cat <<-EOF
1.10.4
1.10.3
1.10.2
+wheel
EOF
)
@@ -134,7 +135,7 @@ _breeze_long_options="
help python: backend: integration:
kubernetes-mode: kubernetes-version: helm-version: kind-version:
skip-mounting-local-sources install-airflow-version: install-airflow-reference: db-reset
-verbose assume-yes assume-no assume-quit forward-credentials rbac-ui init-script:
+verbose assume-yes assume-no assume-quit forward-credentials init-script:
force-build-images force-pull-images production-image extras: force-clean-images skip-rebuild-check
build-cache-local build-cache-pulled build-cache-disabled disable-pip-cache
dockerhub-user: dockerhub-repo: github-registry github-repository: github-image-id:
diff --git a/chart/requirements.lock b/chart/requirements.lock
index 3f3c34a..e460e9f 100644
--- a/chart/requirements.lock
+++ b/chart/requirements.lock
@@ -2,5 +2,5 @@ dependencies:
- name: postgresql
repository: https://charts.helm.sh/stable/
version: 6.3.12
-digest: sha256:58d88cf56e78b2380091e9e16cc6ccf58b88b3abe4a1886dd47cd9faef5309af
-generated: "2020-11-04T15:59:36.967913-08:00"
+digest: sha256:1748aa702050d4e72ffba1b18960f49bfe5368757cf976116afeffbdedda1c98
+generated: "2020-11-07T17:40:45.418723358+01:00"
diff --git a/images/testing/k9s.png b/images/testing/k9s.png
new file mode 100644
index 0000000..a8eec97
Binary files /dev/null and b/images/testing/k9s.png differ
diff --git a/images/testing/kubeconfig-env.png b/images/testing/kubeconfig-env.png
new file mode 100644
index 0000000..b2ebfd5
Binary files /dev/null and b/images/testing/kubeconfig-env.png differ
diff --git a/images/testing/kubernetes-virtualenv.png b/images/testing/kubernetes-virtualenv.png
new file mode 100644
index 0000000..6e208d6
Binary files /dev/null and b/images/testing/kubernetes-virtualenv.png differ
diff --git a/images/testing/pytest-runner.png b/images/testing/pytest-runner.png
new file mode 100644
index 0000000..fdb48cc
Binary files /dev/null and b/images/testing/pytest-runner.png differ
diff --git a/images/testing/run-test.png b/images/testing/run-test.png
new file mode 100644
index 0000000..21a5c9d
Binary files /dev/null and b/images/testing/run-test.png differ
diff --git a/kubernetes_tests/test_kubernetes_executor.py b/kubernetes_tests/test_kubernetes_executor.py
index 694cf75..bb89cb7 100644
--- a/kubernetes_tests/test_kubernetes_executor.py
+++ b/kubernetes_tests/test_kubernetes_executor.py
@@ -64,10 +64,11 @@ class TestKubernetesExecutor(unittest.TestCase):
return len(names)
@staticmethod
- def _delete_airflow_pod():
+ def _delete_airflow_pod(name=''):
+ suffix = '-' + name if name else ''
air_pod = check_output(['kubectl', 'get', 'pods']).decode()
air_pod = air_pod.split('\n')
- names = [re.compile(r'\s+').split(x)[0] for x in air_pod if 'airflow' in x]
+ names = [re.compile(r'\s+').split(x)[0] for x in air_pod if 'airflow' + suffix in x]
if names:
check_call(['kubectl', 'delete', 'pod', names[0]])
@@ -233,7 +234,7 @@ class TestKubernetesExecutor(unittest.TestCase):
execution_date = self.start_job_in_kubernetes(dag_id, host)
- self._delete_airflow_pod()
+ self._delete_airflow_pod("scheduler")
time.sleep(10) # give time for pod to restart
diff --git a/kubernetes_tests/test_kubernetes_pod_operator.py b/kubernetes_tests/test_kubernetes_pod_operator.py
index 7a8674a..c2a0c62 100644
--- a/kubernetes_tests/test_kubernetes_pod_operator.py
+++ b/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -15,43 +15,38 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
import json
import logging
import os
+import random
import shutil
import sys
-import unittest
import textwrap
+import unittest
+from unittest import mock
+from unittest.mock import ANY
-import kubernetes.client.models as k8s
import pendulum
+from kubernetes.client import models as k8s
from kubernetes.client.api_client import ApiClient
from kubernetes.client.rest import ApiException
-from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.exceptions import AirflowException
from airflow.kubernetes import kube_client
-from airflow.kubernetes.pod import Port
from airflow.kubernetes.pod_generator import PodDefaults
from airflow.kubernetes.pod_launcher import PodLauncher
-from airflow.kubernetes.pod_runtime_info_env import PodRuntimeInfoEnv
from airflow.kubernetes.secret import Secret
-from airflow.kubernetes.volume import Volume
-from airflow.kubernetes.volume_mount import VolumeMount
from airflow.models import DAG, TaskInstance
+from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.utils import timezone
from airflow.version import version as airflow_version
-from tests.compat import mock, patch
-# noinspection DuplicatedCode
def create_context(task):
dag = DAG(dag_id="dag")
tzinfo = pendulum.timezone("Europe/Amsterdam")
execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
- task_instance = TaskInstance(task=task,
- execution_date=execution_date)
+ task_instance = TaskInstance(task=task, execution_date=execution_date)
return {
"dag": dag,
"ts": execution_date.isoformat(),
@@ -60,7 +55,11 @@ def create_context(task):
}
-# noinspection DuplicatedCode,PyUnusedLocal
+def get_kubeconfig_path():
+ kubeconfig_path = os.environ.get('KUBECONFIG')
+ return kubeconfig_path if kubeconfig_path else os.path.expanduser('~/.kube/config')
+
+
class TestKubernetesPodOperatorSystem(unittest.TestCase):
def get_current_task_name(self):
# reverse test name to make pod name unique (it has limited length)
@@ -74,29 +73,33 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
'kind': 'Pod',
'metadata': {
'namespace': 'default',
- 'name': mock.ANY,
+ 'name': ANY,
'annotations': {},
'labels': {
- 'foo': 'bar', 'kubernetes_pod_operator': 'True',
+ 'foo': 'bar',
+ 'kubernetes_pod_operator': 'True',
'airflow_version': airflow_version.replace('+', '-'),
'execution_date': '2016-01-01T0100000100-a2f50a31f',
'dag_id': 'dag',
- 'task_id': 'task',
- 'try_number': '1'},
+ 'task_id': ANY,
+ 'try_number': '1',
+ },
},
'spec': {
'affinity': {},
- 'containers': [{
- 'image': 'ubuntu:16.04',
- 'args': ["echo 10"],
- 'command': ["bash", "-cx"],
- 'env': [],
- 'imagePullPolicy': 'IfNotPresent',
- 'envFrom': [],
- 'name': 'base',
- 'ports': [],
- 'volumeMounts': [],
- }],
+ 'containers': [
+ {
+ 'image': 'ubuntu:16.04',
+ 'args': ["echo 10"],
+ 'command': ["bash", "-cx"],
+ 'env': [],
+ 'envFrom': [],
+ 'resources': {},
+ 'name': 'base',
+ 'ports': [],
+ 'volumeMounts': [],
+ }
+ ],
'hostNetwork': False,
'imagePullSecrets': [],
'initContainers': [],
@@ -106,29 +109,19 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
'serviceAccountName': 'default',
'tolerations': [],
'volumes': [],
- }
+ },
}
def tearDown(self):
client = kube_client.get_kube_client(in_cluster=False)
client.delete_collection_namespaced_pod(namespace="default")
+ import time
- def create_context(self, task):
- dag = DAG(dag_id="dag")
- tzinfo = pendulum.timezone("Europe/Amsterdam")
- execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
- task_instance = TaskInstance(task=task,
- execution_date=execution_date)
- return {
- "dag": dag,
- "ts": execution_date.isoformat(),
- "task": task,
- "ti": task_instance,
- }
+ time.sleep(1)
def test_do_xcom_push_defaults_false(self):
new_config_path = '/tmp/kube_config'
- old_config_path = os.path.expanduser('~/.kube/config')
+ old_config_path = get_kubeconfig_path()
shutil.copy(old_config_path, new_config_path)
k = KubernetesPodOperator(
@@ -137,8 +130,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
config_file=new_config_path,
@@ -147,7 +140,7 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
def test_config_path_move(self):
new_config_path = '/tmp/kube_config'
- old_config_path = os.path.expanduser('~/.kube/config')
+ old_config_path = get_kubeconfig_path()
shutil.copy(old_config_path, new_config_path)
k = KubernetesPodOperator(
@@ -157,103 +150,16 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
arguments=["echo 10"],
labels={"foo": "bar"},
name="test1",
- task_id="task",
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
config_file=new_config_path,
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.assertEqual(self.expected_pod, actual_pod)
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
- @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
- def test_config_path(self, client_mock, monitor_mock, start_mock): # pylint: disable=unused-argument
- from airflow.utils.state import State
-
- file_path = "/tmp/fake_file"
- k = KubernetesPodOperator(
- namespace='default',
- image="ubuntu:16.04",
- cmds=["bash", "-cx"],
- arguments=["echo 10"],
- labels={"foo": "bar"},
- name="test",
- task_id="task",
- in_cluster=False,
- do_xcom_push=False,
- config_file=file_path,
- cluster_context='default',
- )
- monitor_mock.return_value = (State.SUCCESS, None)
- client_mock.list_namespaced_pod.return_value = []
- context = self.create_context(k)
- k.execute(context=context)
- client_mock.assert_called_once_with(
- in_cluster=False,
- cluster_context='default',
- config_file=file_path,
- )
-
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
- @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
- def test_image_pull_secrets_correctly_set(self, mock_client, monitor_mock, start_mock):
- from airflow.utils.state import State
-
- fake_pull_secrets = "fakeSecret"
- k = KubernetesPodOperator(
- namespace='default',
- image="ubuntu:16.04",
- cmds=["bash", "-cx"],
- arguments=["echo 10"],
- labels={"foo": "bar"},
- name="test",
- task_id="task",
- in_cluster=False,
- do_xcom_push=False,
- image_pull_secrets=fake_pull_secrets,
- cluster_context='default',
- )
- monitor_mock.return_value = (State.SUCCESS, None)
- context = self.create_context(k)
- k.execute(context=context)
- self.assertEqual(
- start_mock.call_args[0][0].spec.image_pull_secrets,
- [k8s.V1LocalObjectReference(name=fake_pull_secrets)]
- )
-
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
- @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.delete_pod")
- @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
- def test_pod_delete_even_on_launcher_error(
- self,
- mock_client,
- delete_pod_mock,
- monitor_pod_mock,
- start_pod_mock): # pylint: disable=unused-argument
- k = KubernetesPodOperator(
- namespace='default',
- image="ubuntu:16.04",
- cmds=["bash", "-cx"],
- arguments=["echo 10"],
- labels={"foo": "bar"},
- name="test",
- task_id="task",
- in_cluster=False,
- do_xcom_push=False,
- cluster_context='default',
- is_delete_operator_pod=True,
- )
- monitor_pod_mock.side_effect = AirflowException('fake failure')
- with self.assertRaises(AirflowException):
- context = self.create_context(k)
- k.execute(context=context)
- assert delete_pod_mock.called
-
def test_working_pod(self):
k = KubernetesPodOperator(
namespace='default',
@@ -261,8 +167,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
@@ -279,49 +185,15 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
is_delete_operator_pod=True,
)
- context = self.create_context(k)
- k.execute(context)
- actual_pod = self.api_client.sanitize_for_serialization(k.pod)
- self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
- self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
-
- def test_pod_with_volume_secret(self):
- k = KubernetesPodOperator(
- namespace='default',
- image="ubuntu:16.04",
- cmds=["bash", "-cx"],
- in_cluster=False,
- labels={"foo": "bar"},
- arguments=["echo 10"],
- secrets=[Secret(
- deploy_type="volume",
- deploy_target="/var/location",
- secret="my-secret",
- key="content.json",
- )],
- name="airflow-test-pod",
- task_id="task",
- get_logs=True,
- is_delete_operator_pod=True,
- )
-
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
- self.expected_pod['spec']['containers'][0]['volumeMounts'] = [
- {'mountPath': '/var/location',
- 'name': mock.ANY,
- 'readOnly': True}]
- self.expected_pod['spec']['volumes'] = [
- {'name': mock.ANY,
- 'secret': {'secretName': 'my-secret'}}
- ]
self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
@@ -332,13 +204,13 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
hostnetwork=True,
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['hostNetwork'] = True
@@ -353,14 +225,14 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
hostnetwork=True,
- dnspolicy=dns_policy
+ dnspolicy=dns_policy,
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['hostNetwork'] = True
@@ -376,32 +248,28 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
- schedulername=scheduler_name
+ schedulername=scheduler_name,
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['schedulerName'] = scheduler_name
self.assertEqual(self.expected_pod, actual_pod)
- self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
- self.assertEqual(self.expected_pod['metadata']['labels'], actual_pod['metadata']['labels'])
def test_pod_node_selectors(self):
- node_selectors = {
- 'beta.kubernetes.io/os': 'linux'
- }
+ node_selectors = {'beta.kubernetes.io/os': 'linux'}
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
node_selectors=node_selectors,
@@ -413,40 +281,28 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
self.assertEqual(self.expected_pod, actual_pod)
def test_pod_resources(self):
- resources = {
- 'limit_cpu': 0.25,
- 'limit_memory': '64Mi',
- 'limit_ephemeral_storage': '2Gi',
- 'request_cpu': '250m',
- 'request_memory': '64Mi',
- 'request_ephemeral_storage': '1Gi',
- }
+ resources = k8s.V1ResourceRequirements(
+ requests={'memory': '64Mi', 'cpu': '250m', 'ephemeral-storage': '1Gi'},
+ limits={'memory': '64Mi', 'cpu': 0.25, 'nvidia.com/gpu': None, 'ephemeral-storage': '2Gi'},
+ )
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
resources=resources,
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['containers'][0]['resources'] = {
- 'requests': {
- 'memory': '64Mi',
- 'cpu': '250m',
- 'ephemeral-storage': '1Gi'
- },
- 'limits': {
- 'memory': '64Mi',
- 'cpu': 0.25,
- 'ephemeral-storage': '2Gi'
- }
+ 'requests': {'memory': '64Mi', 'cpu': '250m', 'ephemeral-storage': '1Gi'},
+ 'limits': {'memory': '64Mi', 'cpu': 0.25, 'nvidia.com/gpu': None, 'ephemeral-storage': '2Gi'},
}
self.assertEqual(self.expected_pod, actual_pod)
@@ -457,11 +313,7 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
'nodeSelectorTerms': [
{
'matchExpressions': [
- {
- 'key': 'beta.kubernetes.io/os',
- 'operator': 'In',
- 'values': ['linux']
- }
+ {'key': 'beta.kubernetes.io/os', 'operator': 'In', 'values': ['linux']}
]
}
]
@@ -474,8 +326,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
affinity=affinity,
@@ -487,7 +339,10 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
self.assertEqual(self.expected_pod, actual_pod)
def test_port(self):
- port = Port('http', 80)
+ port = k8s.V1ContainerPort(
+ name='http',
+ container_port=80,
+ )
k = KubernetesPodOperator(
namespace='default',
@@ -495,37 +350,33 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
ports=[port],
)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context=context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
- self.expected_pod['spec']['containers'][0]['ports'] = [{
- 'name': 'http',
- 'containerPort': 80
- }]
+ self.expected_pod['spec']['containers'][0]['ports'] = [{'name': 'http', 'containerPort': 80}]
self.assertEqual(self.expected_pod, actual_pod)
def test_volume_mount(self):
- with patch.object(PodLauncher, 'log') as mock_logger:
- volume_mount = VolumeMount('test-volume',
- mount_path='/tmp/test_volume',
- sub_path=None,
- read_only=False)
-
- volume_config = {
- 'persistentVolumeClaim':
- {
- 'claimName': 'test-volume'
- }
- }
- volume = Volume(name='test-volume', configs=volume_config)
- args = ["echo \"retrieved from mount\" > /tmp/test_volume/test.txt "
- "&& cat /tmp/test_volume/test.txt"]
+ with mock.patch.object(PodLauncher, 'log') as mock_logger:
+ volume_mount = k8s.V1VolumeMount(
+ name='test-volume', mount_path='/tmp/test_volume', sub_path=None, read_only=False
+ )
+
+ volume = k8s.V1Volume(
+ name='test-volume',
+ persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='test-volume'),
+ )
+
+ args = [
+ "echo \"retrieved from mount\" > /tmp/test_volume/test.txt "
+ "&& cat /tmp/test_volume/test.txt"
+ ]
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
@@ -534,27 +385,22 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
labels={"foo": "bar"},
volume_mounts=[volume_mount],
volumes=[volume],
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
context = create_context(k)
k.execute(context=context)
- mock_logger.info.assert_any_call(b"retrieved from mount\n")
+ mock_logger.info.assert_any_call('retrieved from mount')
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['containers'][0]['args'] = args
- self.expected_pod['spec']['containers'][0]['volumeMounts'] = [{
- 'name': 'test-volume',
- 'mountPath': '/tmp/test_volume',
- 'readOnly': False
- }]
- self.expected_pod['spec']['volumes'] = [{
- 'name': 'test-volume',
- 'persistentVolumeClaim': {
- 'claimName': 'test-volume'
- }
- }]
+ self.expected_pod['spec']['containers'][0]['volumeMounts'] = [
+ {'name': 'test-volume', 'mountPath': '/tmp/test_volume', 'readOnly': False}
+ ]
+ self.expected_pod['spec']['volumes'] = [
+ {'name': 'test-volume', 'persistentVolumeClaim': {'claimName': 'test-volume'}}
+ ]
self.assertEqual(self.expected_pod, actual_pod)
def test_run_as_user_root(self):
@@ -569,8 +415,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
security_context=security_context,
@@ -594,8 +440,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
security_context=security_context,
@@ -619,8 +465,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-fs-group",
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
security_context=security_context,
@@ -639,8 +485,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
startup_timeout_seconds=5,
@@ -660,8 +506,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
startup_timeout_seconds=5,
@@ -685,8 +531,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=bad_internal_command,
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
@@ -699,15 +545,15 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
def test_xcom_push(self):
return_value = '{"foo": "bar"\n, "buzz": 2}'
- args = ['echo \'{}\' > /airflow/xcom/return.json'.format(return_value)]
+ args = ['echo \'' + str(return_value) + '\' > /airflow/xcom/return.json']
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=args,
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=True,
)
@@ -730,7 +576,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
# GIVEN
from airflow.utils.state import State
- configmap = 'test-configmap'
+ configmap_name = "test-config-map"
+ env_from = [k8s.V1EnvFromSource(config_map_ref=k8s.V1ConfigMapEnvSource(name=configmap_name))]
# WHEN
k = KubernetesPodOperator(
namespace='default',
@@ -738,22 +585,17 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
- configmaps=[configmap],
+ env_from=env_from,
)
# THEN
mock_monitor.return_value = (State.SUCCESS, None)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
- self.assertEqual(
- mock_start.call_args[0][0].spec.containers[0].env_from,
- [k8s.V1EnvFromSource(config_map_ref=k8s.V1ConfigMapEnvSource(
- name=configmap
- ))]
- )
+ self.assertEqual(mock_start.call_args[0][0].spec.containers[0].env_from, env_from)
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
@@ -761,6 +603,7 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
def test_envs_from_secrets(self, mock_client, monitor_mock, start_mock):
# GIVEN
from airflow.utils.state import State
+
secret_ref = 'secret_name'
secrets = [Secret('env', None, secret_ref)]
# WHEN
@@ -771,34 +614,40 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
arguments=["echo 10"],
secrets=secrets,
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
# THEN
monitor_mock.return_value = (State.SUCCESS, None)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
self.assertEqual(
start_mock.call_args[0][0].spec.containers[0].env_from,
- [k8s.V1EnvFromSource(secret_ref=k8s.V1SecretEnvSource(
- name=secret_ref
- ))]
+ [k8s.V1EnvFromSource(secret_ref=k8s.V1SecretEnvSource(name=secret_ref))],
)
def test_env_vars(self):
# WHEN
+ env_vars = [
+ k8s.V1EnvVar(name="ENV1", value="val1"),
+ k8s.V1EnvVar(name="ENV2", value="val2"),
+ k8s.V1EnvVar(
+ name="ENV3",
+ value_from=k8s.V1EnvVarSource(field_ref=k8s.V1ObjectFieldSelector(field_path="status.podIP")),
+ ),
+ ]
+
k = KubernetesPodOperator(
namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
- env_vars={"ENV1": "val1", "ENV2": "val2", },
- pod_runtime_info_envs=[PodRuntimeInfoEnv("ENV3", "status.podIP")],
+ env_vars=env_vars,
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
@@ -811,14 +660,7 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
self.expected_pod['spec']['containers'][0]['env'] = [
{'name': 'ENV1', 'value': 'val1'},
{'name': 'ENV2', 'value': 'val2'},
- {
- 'name': 'ENV3',
- 'valueFrom': {
- 'fieldRef': {
- 'fieldPath': 'status.podIP'
- }
- }
- }
+ {'name': 'ENV3', 'valueFrom': {'fieldRef': {'fieldPath': 'status.podIP'}}},
]
self.assertEqual(self.expected_pod, actual_pod)
@@ -828,7 +670,7 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
task_id="task" + self.get_current_task_name(),
in_cluster=False,
pod_template_file=fixture,
- do_xcom_push=True
+ do_xcom_push=True,
)
context = create_context(k)
@@ -841,10 +683,10 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
k = KubernetesPodOperator(
task_id="task" + self.get_current_task_name(),
labels={"foo": "bar", "fizz": "buzz"},
- env_vars={"env_name": "value"},
+ env_vars=[k8s.V1EnvVar(name="env_name", value="value")],
in_cluster=False,
pod_template_file=fixture,
- do_xcom_push=True
+ do_xcom_push=True,
)
context = create_context(k)
@@ -856,20 +698,14 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
def test_init_container(self):
# GIVEN
- volume_mounts = [k8s.V1VolumeMount(
- mount_path='/etc/foo',
- name='test-volume',
- sub_path=None,
- read_only=True
- )]
-
- init_environments = [k8s.V1EnvVar(
- name='key1',
- value='value1'
- ), k8s.V1EnvVar(
- name='key2',
- value='value2'
- )]
+ volume_mounts = [
+ k8s.V1VolumeMount(mount_path='/etc/foo', name='test-volume', sub_path=None, read_only=True)
+ ]
+
+ init_environments = [
+ k8s.V1EnvVar(name='key1', value='value1'),
+ k8s.V1EnvVar(name='key2', value='value2'),
+ ]
init_container = k8s.V1Container(
name="init-container",
@@ -877,34 +713,20 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
env=init_environments,
volume_mounts=volume_mounts,
command=["bash", "-cx"],
- args=["echo 10"]
+ args=["echo 10"],
)
- volume_config = {
- 'persistentVolumeClaim':
- {
- 'claimName': 'test-volume'
- }
- }
- volume = Volume(name='test-volume', configs=volume_config)
-
+ volume = k8s.V1Volume(
+ name='test-volume',
+ persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='test-volume'),
+ )
expected_init_container = {
'name': 'init-container',
'image': 'ubuntu:16.04',
'command': ['bash', '-cx'],
'args': ['echo 10'],
- 'env': [{
- 'name': 'key1',
- 'value': 'value1'
- }, {
- 'name': 'key2',
- 'value': 'value2'
- }],
- 'volumeMounts': [{
- 'mountPath': '/etc/foo',
- 'name': 'test-volume',
- 'readOnly': True
- }],
+ 'env': [{'name': 'key1', 'value': 'value1'}, {'name': 'key2', 'value': 'value2'}],
+ 'volumeMounts': [{'mountPath': '/etc/foo', 'name': 'test-volume', 'readOnly': True}],
}
k = KubernetesPodOperator(
@@ -913,8 +735,8 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
volumes=[volume],
init_containers=[init_container],
in_cluster=False,
@@ -924,30 +746,30 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['initContainers'] = [expected_init_container]
- self.expected_pod['spec']['volumes'] = [{
- 'name': 'test-volume',
- 'persistentVolumeClaim': {
- 'claimName': 'test-volume'
- }
- }]
+ self.expected_pod['spec']['volumes'] = [
+ {'name': 'test-volume', 'persistentVolumeClaim': {'claimName': 'test-volume'}}
+ ]
self.assertEqual(self.expected_pod, actual_pod)
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
@mock.patch("airflow.kubernetes.kube_client.get_kube_client")
- def test_pod_template_file(self, mock_client, monitor_mock, start_mock):
+ def test_pod_template_file(
+ self, mock_client, monitor_mock, start_mock # pylint: disable=unused-argument
+ ):
from airflow.utils.state import State
- fixture = sys.path[0] + '/tests/kubernetes/pod.yaml'
+
+ path = sys.path[0] + '/tests/kubernetes/pod.yaml'
k = KubernetesPodOperator(
- task_id='task',
- pod_template_file=fixture,
- do_xcom_push=True
+ task_id="task" + self.get_current_task_name(), pod_template_file=path, do_xcom_push=True
)
+
monitor_mock.return_value = (State.SUCCESS, None)
context = create_context(k)
with self.assertLogs(k.log, level=logging.DEBUG) as cm:
k.execute(context)
- expected_line = textwrap.dedent("""\
+ expected_line = textwrap.dedent(
+ """\
DEBUG:airflow.task.operators:Starting pod:
api_version: v1
kind: Pod
@@ -956,65 +778,57 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cluster_name: null
creation_timestamp: null
deletion_grace_period_seconds: null\
- """).strip()
+ """
+ ).strip()
self.assertTrue(any(line.startswith(expected_line) for line in cm.output))
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
- expected_dict = {'apiVersion': 'v1',
- 'kind': 'Pod',
- 'metadata': {'annotations': {},
- 'labels': {},
- 'name': 'memory-demo',
- 'namespace': 'mem-example'},
- 'spec': {'affinity': {},
- 'containers': [{'args': ['--vm',
- '1',
- '--vm-bytes',
- '150M',
- '--vm-hang',
- '1'],
- 'command': ['stress'],
- 'env': [],
- 'envFrom': [],
- 'image': 'apache/airflow:stress-2020.07.10-1.0.4',
- 'imagePullPolicy': 'IfNotPresent',
- 'name': 'base',
- 'ports': [],
- 'resources': {'limits': {'memory': '200Mi'},
- 'requests': {'memory': '100Mi'}},
- 'volumeMounts': [{'mountPath': '/airflow/xcom',
- 'name': 'xcom'}]},
- {'command': ['sh',
- '-c',
- 'trap "exit 0" INT; while true; do sleep '
- '30; done;'],
- 'image': 'alpine',
- 'name': 'airflow-xcom-sidecar',
- 'resources': {'requests': {'cpu': '1m'}},
- 'volumeMounts': [{'mountPath': '/airflow/xcom',
- 'name': 'xcom'}]}],
- 'hostNetwork': False,
- 'imagePullSecrets': [],
- 'initContainers': [],
- 'nodeSelector': {},
- 'restartPolicy': 'Never',
- 'securityContext': {},
- 'serviceAccountName': 'default',
- 'tolerations': [],
- 'volumes': [{'emptyDir': {}, 'name': 'xcom'}]}}
+ expected_dict = {
+ 'apiVersion': 'v1',
+ 'kind': 'Pod',
+ 'metadata': {'annotations': {}, 'labels': {}, 'name': 'memory-demo', 'namespace': 'mem-example'},
+ 'spec': {
+ 'affinity': {},
+ 'containers': [
+ {
+ 'args': ['--vm', '1', '--vm-bytes', '150M', '--vm-hang', '1'],
+ 'command': ['stress'],
+ 'env': [],
+ 'envFrom': [],
+ 'image': 'apache/airflow:stress-2020.07.10-1.0.4',
+ 'name': 'base',
+ 'ports': [],
+ 'resources': {'limits': {'memory': '200Mi'}, 'requests': {'memory': '100Mi'}},
+ 'volumeMounts': [{'mountPath': '/airflow/xcom', 'name': 'xcom'}],
+ },
+ {
+ 'command': ['sh', '-c', 'trap "exit 0" INT; while true; do sleep 30; done;'],
+ 'image': 'alpine',
+ 'name': 'airflow-xcom-sidecar',
+ 'resources': {'requests': {'cpu': '1m'}},
+ 'volumeMounts': [{'mountPath': '/airflow/xcom', 'name': 'xcom'}],
+ },
+ ],
+ 'hostNetwork': False,
+ 'imagePullSecrets': [],
+ 'initContainers': [],
+ 'nodeSelector': {},
+ 'restartPolicy': 'Never',
+ 'securityContext': {},
+ 'serviceAccountName': 'default',
+ 'tolerations': [],
+ 'volumes': [{'emptyDir': {}, 'name': 'xcom'}],
+ },
+ }
self.assertEqual(expected_dict, actual_pod)
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
@mock.patch("airflow.kubernetes.kube_client.get_kube_client")
def test_pod_priority_class_name(
- self,
- mock_client,
- monitor_mock,
- start_mock): # pylint: disable=unused-argument
- """Test ability to assign priorityClassName to pod
-
- """
+ self, mock_client, monitor_mock, start_mock # pylint: disable=unused-argument
+ ):
+ """Test ability to assign priorityClassName to pod"""
from airflow.utils.state import State
priority_class_name = "medium-test"
@@ -1024,15 +838,15 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels={"foo": "bar"},
- name="test",
- task_id="task",
+ name="test-" + str(random.randint(0, 1000000)),
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
priority_class_name=priority_class_name,
)
monitor_mock.return_value = (State.SUCCESS, None)
- context = self.create_context(k)
+ context = create_context(k)
k.execute(context)
actual_pod = self.api_client.sanitize_for_serialization(k.pod)
self.expected_pod['spec']['priorityClassName'] = priority_class_name
@@ -1048,15 +862,15 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
arguments=["echo 10"],
labels={"foo": "bar"},
name=pod_name_too_long,
- task_id="task",
+ task_id="task" + self.get_current_task_name(),
in_cluster=False,
do_xcom_push=False,
)
@mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
- def test_on_kill(self,
- monitor_mock): # pylint: disable=unused-argument
+ def test_on_kill(self, monitor_mock): # pylint: disable=unused-argument
from airflow.utils.state import State
+
client = kube_client.get_kube_client(in_cluster=False)
name = "test"
namespace = "default"
@@ -1082,4 +896,46 @@ class TestKubernetesPodOperatorSystem(unittest.TestCase):
with self.assertRaises(ApiException):
pod = client.read_namespaced_pod(name=name, namespace=namespace)
+ def test_reattach_failing_pod_once(self):
+ from airflow.utils.state import State
+
+ client = kube_client.get_kube_client(in_cluster=False)
+ name = "test"
+ namespace = "default"
+ k = KubernetesPodOperator(
+ namespace='default',
+ image="ubuntu:16.04",
+ cmds=["bash", "-cx"],
+ arguments=["exit 1"],
+ labels={"foo": "bar"},
+ name="test",
+ task_id=name,
+ in_cluster=False,
+ do_xcom_push=False,
+ is_delete_operator_pod=False,
+ termination_grace_period=0,
+ )
+
+ context = create_context(k)
+
+ with mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod") as monitor_mock:
+ monitor_mock.return_value = (State.SUCCESS, None)
+ k.execute(context)
+ name = k.pod.metadata.name
+ pod = client.read_namespaced_pod(name=name, namespace=namespace)
+ while pod.status.phase != "Failed":
+ pod = client.read_namespaced_pod(name=name, namespace=namespace)
+ with self.assertRaises(AirflowException):
+ k.execute(context)
+ pod = client.read_namespaced_pod(name=name, namespace=namespace)
+ self.assertEqual(pod.metadata.labels["already_checked"], "True")
+ with mock.patch(
+ "airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator"
+ ".create_new_pod_for_operator"
+ ) as create_mock:
+ create_mock.return_value = ("success", {}, {})
+ k.execute(context)
+ create_mock.assert_called_once()
+
+
# pylint: enable=unused-argument
diff --git a/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh b/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
index 4f13335..bcb5cf4 100755
--- a/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
+++ b/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh
@@ -21,7 +21,6 @@
kind::make_sure_kubernetes_tools_are_installed
kind::get_kind_cluster_name
-traps::add_trap kind::stop_kubectl EXIT HUP INT TERM
traps::add_trap kind::dump_kind_logs EXIT HUP INT TERM
interactive="false"
@@ -82,8 +81,13 @@ if [[ ! -d ${virtualenv_path} ]]; then
python -m venv "${virtualenv_path}"
fi
+# In Python 3.5 activating virtualenv hits undefined variable
+set +u
+
. "${virtualenv_path}/bin/activate"
+set -u
+
pip install --upgrade pip==20.2.3
pip install pytest freezegun pytest-cov \
@@ -105,7 +109,6 @@ if [[ ${interactive} == "true" ]]; then
echo
echo "You are entering the virtualenv now. Type exit to exit back to the original shell"
echo
- kubectl config set-context --current --namespace=airflow
exec "${SHELL}"
else
pytest "${pytest_args[@]}" "${tests_to_run[@]}"
diff --git a/scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh b/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
similarity index 94%
copy from scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
copy to scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
index 2a7455a..e12f809 100755
--- a/scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+++ b/scripts/ci/kubernetes/ci_setup_cluster_and_deploy_airflow_to_kubernetes.sh
@@ -23,10 +23,11 @@ traps::add_trap "kind::dump_kind_logs" EXIT HUP INT TERM
kind::make_sure_kubernetes_tools_are_installed
kind::get_kind_cluster_name
+kind::perform_kind_cluster_operation "start"
build_images::prepare_prod_build
build_images::build_prod_images
kind::build_image_for_kubernetes_tests
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
-kind::forward_port_to_kind_webserver
kind::deploy_test_kubernetes_resources
+kind::wait_for_webserver_healthy
diff --git a/scripts/ci/kubernetes/kind-cluster-conf.yaml b/scripts/ci/kubernetes/kind-cluster-conf.yaml
index 348fb68..df60820 100644
--- a/scripts/ci/kubernetes/kind-cluster-conf.yaml
+++ b/scripts/ci/kubernetes/kind-cluster-conf.yaml
@@ -23,6 +23,11 @@ networking:
nodes:
- role: control-plane
- role: worker
+ extraPortMappings:
+ - containerPort: 30007
+ hostPort: 8080
+ listenAddress: "0.0.0.0"
+ protocol: TCP
kubeadmConfigPatchesJson6902:
- group: kubeadm.k8s.io
version: v1beta2
diff --git a/scripts/ci/kubernetes/kind-cluster-conf.yaml b/scripts/ci/kubernetes/nodeport.yaml
similarity index 68%
copy from scripts/ci/kubernetes/kind-cluster-conf.yaml
copy to scripts/ci/kubernetes/nodeport.yaml
index 348fb68..8438281 100644
--- a/scripts/ci/kubernetes/kind-cluster-conf.yaml
+++ b/scripts/ci/kubernetes/nodeport.yaml
@@ -15,19 +15,17 @@
# specific language governing permissions and limitations
# under the License.
---
-kind: Cluster
-apiVersion: kind.sigs.k8s.io/v1alpha3
-networking:
- apiServerAddress: 0.0.0.0
- apiServerPort: 19090
-nodes:
- - role: control-plane
- - role: worker
-kubeadmConfigPatchesJson6902:
- - group: kubeadm.k8s.io
- version: v1beta2
- kind: ClusterConfiguration
- patch: |
- - op: add
- path: /apiServer/certSANs/-
- value: docker
+apiVersion: v1
+kind: Service
+metadata:
+ name: airflow-webserver-node-port
+spec:
+ type: NodePort
+ selector:
+ component: webserver
+ release: airflow
+ tier: airflow
+ ports:
+ - port: 8080
+ targetPort: 8080
+ nodePort: 30007
diff --git a/scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh b/scripts/ci/kubernetes/redeploy_airflow.sh
similarity index 86%
rename from scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
rename to scripts/ci/kubernetes/redeploy_airflow.sh
index 2a7455a..7803d7c 100755
--- a/scripts/ci/kubernetes/ci_deploy_app_to_kubernetes.sh
+++ b/scripts/ci/kubernetes/redeploy_airflow.sh
@@ -23,10 +23,6 @@ traps::add_trap "kind::dump_kind_logs" EXIT HUP INT TERM
kind::make_sure_kubernetes_tools_are_installed
kind::get_kind_cluster_name
-build_images::prepare_prod_build
-build_images::build_prod_images
-kind::build_image_for_kubernetes_tests
-kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
-kind::forward_port_to_kind_webserver
kind::deploy_test_kubernetes_resources
+kind::wait_for_webserver_healthy
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 6194742..defa4de 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -16,14 +16,16 @@
# specific language governing permissions and limitations
# under the License.
-
-function kind::get_kind_cluster_name(){
+function kind::get_kind_cluster_name() {
# Name of the KinD cluster to connect to
export KIND_CLUSTER_NAME=${KIND_CLUSTER_NAME:="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"}
readonly KIND_CLUSTER_NAME
# Name of the KinD cluster to connect to when referred to via kubectl
export KUBECTL_CLUSTER_NAME=kind-${KIND_CLUSTER_NAME}
readonly KUBECTL_CLUSTER_NAME
+ export KUBECONFIG="${BUILD_CACHE_DIR}/.kube/config"
+ mkdir -pv "${BUILD_CACHE_DIR}/.kube/"
+ touch "${KUBECONFIG}"
}
function kind::dump_kind_logs() {
@@ -40,7 +42,7 @@ function kind::dump_kind_logs() {
}
function kind::make_sure_kubernetes_tools_are_installed() {
- SYSTEM=$(uname -s| tr '[:upper:]' '[:lower:]')
+ SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]')
KIND_URL="https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-${SYSTEM}-amd64"
mkdir -pv "${BUILD_CACHE_DIR}/bin"
@@ -48,7 +50,7 @@ function kind::make_sure_kubernetes_tools_are_installed() {
DOWNLOADED_KIND_VERSION=v"$(${KIND_BINARY_PATH} --version | awk '{ print $3 }')"
echo "Currently downloaded kind version = ${DOWNLOADED_KIND_VERSION}"
fi
- if [[ ! -f "${KIND_BINARY_PATH}" || ${DOWNLOADED_KIND_VERSION} != "${KIND_VERSION}" ]]; then
+ if [[ ! -f "${KIND_BINARY_PATH}" || ${DOWNLOADED_KIND_VERSION} != "${KIND_VERSION}" ]]; then
echo
echo "Downloading Kind version ${KIND_VERSION}"
repeats::run_with_retry 4 \
@@ -94,24 +96,10 @@ function kind::make_sure_kubernetes_tools_are_installed() {
}
function kind::create_cluster() {
- if [[ "${TRAVIS:="false"}" == "true" ]]; then
- # Travis CI does not handle the nice output of Kind well, so we need to capture it
- # And display only if kind fails to start
- start_output_heartbeat "Creating kubernetes cluster" 10
- set +e
- if ! OUTPUT=$(kind create cluster \
- --name "${KIND_CLUSTER_NAME}" \
- --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
- --image "kindest/node:${KUBERNETES_VERSION}" 2>&1); then
- echo "${OUTPUT}"
- fi
- stop_output_heartbeat
- else
- kind create cluster \
- --name "${KIND_CLUSTER_NAME}" \
- --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
- --image "kindest/node:${KUBERNETES_VERSION}"
- fi
+ kind create cluster \
+ --name "${KIND_CLUSTER_NAME}" \
+ --config "${AIRFLOW_SOURCES}/scripts/ci/kubernetes/kind-cluster-conf.yaml" \
+ --image "kindest/node:${KUBERNETES_VERSION}"
echo
echo "Created cluster ${KIND_CLUSTER_NAME}"
echo
@@ -125,9 +113,12 @@ function kind::delete_cluster() {
rm -rf "${HOME}/.kube/*"
}
-function kind::perform_kind_cluster_operation() {
- ALLOWED_KIND_OPERATIONS="[ start restart stop deploy test shell recreate ]"
+function kind::set_current_context() {
+ kubectl config set-context --current --namespace=airflow
+}
+function kind::perform_kind_cluster_operation() {
+ ALLOWED_KIND_OPERATIONS="[ start restart stop deploy test shell recreate k9s]"
set +u
if [[ -z "${1=}" ]]; then
echo >&2
@@ -170,6 +161,7 @@ function kind::perform_kind_cluster_operation() {
echo
kind::delete_cluster
kind::create_cluster
+ kind::set_current_context
elif [[ ${OPERATION} == "stop" ]]; then
echo
echo "Deleting cluster"
@@ -181,20 +173,35 @@ function kind::perform_kind_cluster_operation() {
echo "Deploying Airflow to KinD"
echo
kind::build_image_for_kubernetes_tests
+ kind::get_kind_cluster_name
kind::load_image_to_kind_cluster
kind::deploy_airflow_with_helm
- kind::forward_port_to_kind_webserver
kind::deploy_test_kubernetes_resources
+ kind::wait_for_webserver_healthy
elif [[ ${OPERATION} == "test" ]]; then
echo
echo "Testing with KinD"
echo
+ kind::set_current_context
"${AIRFLOW_SOURCES}/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh"
elif [[ ${OPERATION} == "shell" ]]; then
echo
echo "Entering an interactive shell for kubernetes testing"
echo
+ kind::set_current_context
"${AIRFLOW_SOURCES}/scripts/ci/kubernetes/ci_run_kubernetes_tests.sh" "-i"
+ elif [[ ${OPERATION} == "k9s" ]]; then
+ echo
+ echo "Starting k9s CLI"
+ echo
+ export TERM=xterm-256color
+ export EDITOR=vim
+ export K9S_EDITOR=vim
+ kind::set_current_context
+ exec docker run --rm -it --network host \
+ -e COLUMNS="$(tput cols)" -e LINES="$(tput lines)" \
+ -e EDITOR -e K9S_EDITOR \
+ -v "${KUBECONFIG}:/root/.kube/config" quay.io/derailed/k9s
else
echo >&2
echo >&2 "Wrong cluster operation: ${OPERATION}. Should be one of: ${ALLOWED_KIND_OPERATIONS}"
@@ -213,8 +220,7 @@ function kind::perform_kind_cluster_operation() {
echo "Creating cluster"
echo
kind::create_cluster
- elif [[ ${OPERATION} == "stop" || ${OPERATION} == "deploy" || \
- ${OPERATION} == "test" || ${OPERATION} == "shell" ]]; then
+ elif [[ ${OPERATION} == "stop" || ${OPERATION} == "deploy" || ${OPERATION} == "test" || ${OPERATION} == "shell" ]]; then
echo >&2
echo >&2 "Cluster ${KIND_CLUSTER_NAME} does not exist. It should exist for ${OPERATION} operation"
echo >&2
@@ -245,7 +251,6 @@ function kind::check_cluster_ready_for_airflow() {
kubectl create namespace test-namespace --cluster "${KUBECTL_CLUSTER_NAME}"
}
-
function kind::build_image_for_kubernetes_tests() {
cd "${AIRFLOW_SOURCES}" || exit 1
docker build --tag "${AIRFLOW_PROD_IMAGE_KUBERNETES}" . -f - <<EOF
@@ -268,61 +273,37 @@ function kind::load_image_to_kind_cluster() {
kind load docker-image --name "${KIND_CLUSTER_NAME}" "${AIRFLOW_PROD_IMAGE_KUBERNETES}"
}
-MAX_NUM_TRIES_FOR_PORT_FORWARD=12
-readonly MAX_NUM_TRIES_FOR_PORT_FORWARD
+MAX_NUM_TRIES_FOR_HEALTH_CHECK=12
+readonly MAX_NUM_TRIES_FOR_HEALTH_CHECK
-SLEEP_TIME_FOR_PORT_FORWARD=10
-readonly SLEEP_TIME_FOR_PORT_FORWARD
-
-forwarded_port_number=8080
-
-function kind::start_kubectl_forward() {
- echo
- echo "Trying to forward port ${forwarded_port_number} to 8080 on server"
- echo
- kubectl port-forward svc/airflow-webserver "${forwarded_port_number}:8080" --namespace airflow >/dev/null &
-}
+SLEEP_TIME_FOR_HEALTH_CHECK=10
+readonly SLEEP_TIME_FOR_HEALTH_CHECK
-function kind::stop_kubectl() {
- echo
- echo "Stops all kubectl instances"
- echo
- killall kubectl || true
- sleep 10
- killall -s KILL kubectl || true
+FORWARDED_PORT_NUMBER=8080
+readonly FORWARDED_PORT_NUMBER
-}
-function kind::forward_port_to_kind_webserver() {
+function kind::wait_for_webserver_healthy() {
num_tries=0
set +e
- kind::start_kubectl_forward
- sleep "${SLEEP_TIME_FOR_PORT_FORWARD}"
- while ! curl "http://localhost:${forwarded_port_number}/health" -s | grep -q healthy; do
+ sleep "${SLEEP_TIME_FOR_HEALTH_CHECK}"
+ while ! curl "http://localhost:${FORWARDED_PORT_NUMBER}/health" -s | grep -q healthy; do
echo
- echo "Trying to establish port forwarding to 'airflow webserver'"
+ echo "Sleeping ${SLEEP_TIME_FOR_HEALTH_CHECK} while waiting for webserver being ready"
echo
- if [[ ${INCREASE_PORT_NUMBER_FOR_KUBERNETES=} == "true" ]] ; then
- forwarded_port_number=$(( forwarded_port_number + 1 ))
+ sleep "${SLEEP_TIME_FOR_HEALTH_CHECK}"
+ num_tries=$((num_tries + 1))
+ if [[ ${num_tries} == "${MAX_NUM_TRIES_FOR_HEALTH_CHECK}" ]]; then
+ >&2 echo
+ >&2 echo "Timeout while waiting for the webserver health check"
+ >&2 echo
fi
- if [[ ${num_tries} == "${MAX_NUM_TRIES_FOR_PORT_FORWARD}" ]]; then
- echo >&2
- echo >&2 "ERROR! Could not setup a forward port to Airflow's webserver after ${num_tries}! Exiting."
- echo >&2
- exit 1
- fi
- echo
- echo "Trying to establish port forwarding to 'airflow webserver'"
- echo
- kind::start_kubectl_forward
- sleep "${SLEEP_TIME_FOR_PORT_FORWARD}"
- num_tries=$(( num_tries + 1))
done
echo
- echo "Connection to 'airflow webserver' established on port ${forwarded_port_number}"
+ echo "Connection to 'airflow webserver' established on port ${FORWARDED_PORT_NUMBER}"
echo
- initialization::ga_env CLUSTER_FORWARDED_PORT "${forwarded_port_number}"
- export CLUSTER_FORWARDED_PORT="${forwarded_port_number}"
+ initialization::ga_env CLUSTER_FORWARDED_PORT "${FORWARDED_PORT_NUMBER}"
+ export CLUSTER_FORWARDED_PORT="${FORWARDED_PORT_NUMBER}"
set -e
}
@@ -348,16 +329,15 @@ function kind::deploy_airflow_with_helm() {
popd || exit 1
}
-
function kind::deploy_test_kubernetes_resources() {
echo
echo "Deploying Custom kubernetes resources"
echo
kubectl apply -f "scripts/ci/kubernetes/volumes.yaml" --namespace default
kubectl apply -f "scripts/ci/kubernetes/secrets.yaml" --namespace default
+ kubectl apply -f "scripts/ci/kubernetes/nodeport.yaml" --namespace airflow
}
-
function kind::dump_kubernetes_logs() {
POD=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
--cluster "${KUBECTL_CLUSTER_NAME}" | grep airflow | head -1)
[airflow] 32/44: Fixes timeout in helm chart tests (#12209)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6e74f1b6d0c3d188aed013633c0666955e07e14a
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 9 21:18:58 2020 +0100
Fixes timeout in helm chart tests (#12209)
(cherry picked from commit e77867aeab1899a87e3d212ff69ccae0a8447adb)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c217bdc..6c854a1 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -281,7 +281,7 @@ jobs:
path: "./files/documentation"
tests-helm:
- timeout-minutes: 5
+ timeout-minutes: 20
name: "Python unit tests for helm chart"
runs-on: ubuntu-latest
needs: [build-info, ci-images]
[airflow] 31/44: Fixed path of the test_core.py file in docs
(#12191)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 10700af913b3415a97acc6bd7161dd1f2a3c2885
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 9 10:34:06 2020 +0100
Fixed path of the test_core.py file in docs (#12191)
The test_core.py has been used as example in Breeze and it's
location changed to tests/core folder. This PR fixes references
to the changed location.
(cherry picked from commit 57b273a0b1b8af30ed017c2b24c498deb9010247)
---
BREEZE.rst | 2 +-
TESTING.rst | 8 ++++----
breeze | 2 +-
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 5e481f7..b66c7d2 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1894,7 +1894,7 @@ This is the current syntax for `./breeze <./breeze>`_:
as additional options passed to pytest. You can pass 'tests' as target to
run all tests. For example:
- 'breeze tests tests/test_core.py -- --logging-level=DEBUG'
+ 'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
'breeze tests tests
Flags:
diff --git a/TESTING.rst b/TESTING.rst
index f4f790b..c8b170b 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -111,20 +111,20 @@ This can also be done by specifying a full path to the test:
.. code-block:: bash
- pytest tests/test_core.py::TestCore::test_check_operators
+ pytest tests/core/test_core.py::TestCore::test_check_operators
To run the whole test class, enter:
.. code-block:: bash
- pytest tests/test_core.py::TestCore
+ pytest tests/core/test_core.py::TestCore
You can use all available ``pytest`` flags. For example, to increase a log level
for debugging purposes, enter:
.. code-block:: bash
- pytest --log-level=DEBUG tests/test_core.py::TestCore
+ pytest --log-level=DEBUG tests/core/test_core.py::TestCore
Running Tests for a Specified Target Using Breeze from the Host
@@ -149,7 +149,7 @@ You can also specify individual tests or a group of tests:
.. code-block:: bash
- ./breeze tests --db-reset tests/test_core.py::TestCore
+ ./breeze tests --db-reset tests/core/test_core.py::TestCore
Running Tests of a specified type from the Host
diff --git a/breeze b/breeze
index 1d8ffea..3498c64 100755
--- a/breeze
+++ b/breeze
@@ -1763,7 +1763,7 @@ ${CMDNAME} tests [FLAGS] [TEST_TARGET ..] [-- <EXTRA_ARGS>]
as additional options passed to pytest. You can pass 'tests' as target to
run all tests. For example:
- '${CMDNAME} tests tests/test_core.py -- --logging-level=DEBUG'
+ '${CMDNAME} tests tests/core/test_core.py -- --logging-level=DEBUG'
'${CMDNAME} tests tests
Flags:
[airflow] 03/44: Pin `kubernetes` to a max version of 11.0.0.
(#11974)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d5e7b5115055fd3ccdb7896bb45dfe48db83c6da
Author: Jed Cunningham <66...@users.noreply.github.com>
AuthorDate: Fri Oct 30 11:56:58 2020 -0600
Pin `kubernetes` to a max version of 11.0.0. (#11974)
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.
(cherry picked from commit 9687b3bc812a394f9f1debe48ea17bbfe78c270f)
---
setup.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/setup.py b/setup.py
index 50818d4..3ace091 100644
--- a/setup.py
+++ b/setup.py
@@ -305,7 +305,7 @@ kerberos = [
]
kubernetes = [
'cryptography>=2.0.0',
- 'kubernetes>=3.0.0',
+ 'kubernetes>=3.0.0, <12.0.0',
]
ldap = [
'ldap3>=2.5.1',
[airflow] 40/44: Python base image is shared between CI and PROD
image (#12280)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 76f51e15fd4afbaad77b22f01768cc1a795e988b
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Thu Nov 12 12:31:14 2020 +0100
Python base image is shared between CI and PROD image (#12280)
When you are building CI images locally you use the CI
base images from apache:airflow/python* now to maintain
consistency and avoid often rebuilds. But when you build
prod images, you would accidentaly override it with the
python base image available in python repo which might be
different (newer and not yet tested in CI). This PR
changes it to use the same base image which is now
tagged in Apache Airflow's dockerhub repository.
(cherry picked from commit 3c2c29187afffb1a1dd1327803097651cdc079ee)
---
scripts/ci/libraries/_push_pull_remove_images.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/ci/libraries/_push_pull_remove_images.sh b/scripts/ci/libraries/_push_pull_remove_images.sh
index 4c51f0a..7c65db1 100644
--- a/scripts/ci/libraries/_push_pull_remove_images.sh
+++ b/scripts/ci/libraries/_push_pull_remove_images.sh
@@ -154,7 +154,8 @@ function push_pull_remove_images::pull_prod_images_if_needed() {
fi
push_pull_remove_images::pull_image_github_dockerhub "${PYTHON_BASE_IMAGE}" "${GITHUB_REGISTRY_PYTHON_BASE_IMAGE}${PYTHON_TAG_SUFFIX}"
else
- docker pull "${PYTHON_BASE_IMAGE}"
+ docker pull "${AIRFLOW_CI_PYTHON_IMAGE}"
+ docker tag "${AIRFLOW_CI_PYTHON_IMAGE}" "${PYTHON_BASE_IMAGE}"
fi
echo
fi
[airflow] 35/44: Beautify Output of setup-installation pre-commit
(#12218)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit eb7d914492d580dd06c05b5c114b9f9792b6de38
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Nov 10 00:54:47 2020 +0000
Beautify Output of setup-installation pre-commit (#12218)
(cherry picked from commit 08d67add52c293333f9f29f6e786e2f4e6591cdf)
---
.pre-commit-config.yaml | 1 +
.../pre_commit_check_setup_installation.py | 29 ++++++++++++++--------
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index cb5c753..e626cd4 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -219,6 +219,7 @@ repos:
files: ^setup.py$|^docs/installation.rst$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+ additional_dependencies: ['rich==9.2.0']
- id: update-breeze-file
name: Update output of breeze command in BREEZE.rst
entry: "./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh"
diff --git a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
index b4f3281..e386461 100755
--- a/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
+++ b/scripts/ci/pre_commit/pre_commit_check_setup_installation.py
@@ -26,6 +26,10 @@ import sys
from os.path import dirname
from typing import Dict, List
+from rich import print as rprint
+from rich.console import Console
+from rich.table import Table
+
AIRFLOW_SOURCES_DIR = os.path.join(dirname(__file__), os.pardir, os.pardir, os.pardir)
SETUP_PY_FILE = 'setup.py'
DOCS_FILE = 'installation.rst'
@@ -80,30 +84,33 @@ if __name__ == '__main__':
setup_packages = get_extras_from_setup()
docs_packages = get_extras_from_docs()
- output_table = ""
+ table = Table()
+ table.add_column("NAME", justify="right", style="cyan")
+ table.add_column("SETUP", justify="center", style="magenta")
+ table.add_column("INSTALLATION", justify="center", style="green")
for extras in sorted(setup_packages.keys()):
if not set(setup_packages[extras]).intersection(docs_packages):
- output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "V", "")
+ table.add_row(extras, "V", "")
setup_packages_str = str(setup_packages)
for extras in sorted(docs_packages):
if f"'{extras}'" not in setup_packages_str:
- output_table += "| {:20} | {:^10} | {:^10} |\n".format(extras, "", "V")
+ table.add_row(extras, "", "V")
- if output_table == "":
+ if table.row_count == 0:
sys.exit(0)
- print(f"""
-ERROR
+ rprint(f"""\
+[red bold]ERROR!![/red bold]
-"EXTRAS_REQUIREMENTS" section in {SETUP_PY_FILE} should be synchronized
-with "Extra Packages" section in documentation file doc/{DOCS_FILE}.
+"EXTRAS_REQUIREMENTS" section in [bold yellow]{SETUP_PY_FILE}[/bold yellow] should be synchronized
+with "Extra Packages" section in documentation file [bold yellow]doc/{DOCS_FILE}[/bold yellow].
-here is a list of packages that are used but are not documented, or
+Here is a list of packages that are used but are not documented, or
documented although not used.
""")
- print(".{:_^22}.{:_^12}.{:_^12}.".format("NAME", "SETUP", "INSTALLATION"))
- print(output_table)
+ console = Console()
+ console.print(table)
sys.exit(1)
[airflow] 01/44: Migrate from helm-unittest to python unittest
(#11827)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 572387dbeb9d83a2cc4fbdbbbbe5f32a03b1f08f
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Fri Oct 30 15:48:22 2020 +0100
Migrate from helm-unittest to python unittest (#11827)
* Migrate from helm-unittest to python unittest
* fixup! Migrate from helm-unittest to python unittest
* fixup! fixup! Migrate from helm-unittest to python unittest
(cherry picked from commit 9322f3e46c3a06d2e5b891e399cc054e9b76ae72)
---
.github/workflows/ci.yml | 15 +-
.pre-commit-config.yaml | 17 +-
airflow/www_rbac/webpack.config.js | 2 +-
chart/tests/conftest.py | 6 +-
chart/tests/dags-persistent-volume-claim_test.yaml | 64 -----
chart/tests/git-sync-webserver_test.yaml | 66 -----
chart/tests/git-sync-worker_test.yaml | 70 -----
chart/tests/pod-template-file_test.yaml | 149 ----------
chart/tests/test_basic_helm_chart.py | 2 +-
chart/tests/test_celery_kubernetes_executor.py | 47 ++++
.../test_celery_kubernetes_pod_launcher_role.py | 23 +-
chart/tests/test_dags_persistent_volume_claim.py | 72 +++++
chart/tests/test_git_sync_scheduler.py | 311 ++++++++-------------
chart/tests/test_git_sync_webserver.py | 62 ++++
chart/tests/test_git_sync_worker.py | 75 +++++
.../tests/test_migrate_database_job.py | 23 +-
chart/tests/test_pod_template_file.py | 185 ++++++++++++
chart/tests/test_scheduler.py | 40 +++
chart/tests/test_worker.py | 40 +++
docs/static/exampleinclude.css | 2 +-
docs/static/jira-links.js | 2 +-
21 files changed, 686 insertions(+), 587 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 2295bd9..596ab3e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -153,19 +153,6 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- helm-tests:
- timeout-minutes: 5
- name: "Checks: Helm tests. Will soon be replaced with python tests"
- runs-on: ubuntu-latest
- needs: [build-info]
- if: >
- needs.build-info.outputs.needs-helm-tests == 'true'
- steps:
- - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }} )"
- uses: actions/checkout@v2
- - name: "Helm Tests"
- run: ./scripts/ci/kubernetes/ci_run_helm_testing.sh
-
ci-images:
timeout-minutes: 120
name: "Wait for CI images"
@@ -295,7 +282,7 @@ jobs:
name: airflow-documentation
path: "./files/documentation"
- helm-python-tests:
+ tests-helm:
timeout-minutes: 5
name: "Python unit tests for helm chart"
runs-on: ubuntu-latest
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index e4c0a86..1b5c436 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -58,12 +58,12 @@ repos:
- --fuzzy-match-generates-todo
files: \.rst$
- id: insert-license
- name: Add license for all JS/CSS files
- files: \.(js|css)$
+ name: Add license for all JS/CSS/PUML files
+ files: \.(js|css|puml)$
exclude: ^\.github/.*$|^airflow/www/static/.*|^airflow/www_rbac/static/.*$
args:
- --comment-style
- - "/**| *| */"
+ - "/*!| *| */"
- --license-filepath
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
@@ -121,7 +121,7 @@ repos:
- id: insert-license
name: Add license for all md files
files: \.md$
- exclude: ^\.github/.*$
+ exclude: ^\.github/.*$|PROVIDER_CHANGES.*\.md
args:
- --comment-style
- "<!--|| -->"
@@ -133,7 +133,7 @@ repos:
hooks:
- id: doctoc
name: Add TOC for md files
- files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.md$|^dev/README.md$
+ files: ^README\.md$|^CONTRIBUTING\.md$|^UPDATING.*.md$|^dev/README\.md$|^dev/PROVIDER_PACKAGES.md$
args:
- "--maxlevel"
- "2"
@@ -141,7 +141,7 @@ repos:
hooks:
- id: check-hooks-apply
- repo: https://github.com/pre-commit/pre-commit-hooks
- rev: v3.2.0
+ rev: v3.3.0
hooks:
- id: check-merge-conflict
- id: debug-statements
@@ -157,13 +157,14 @@ repos:
- id: rst-backticks
- id: python-no-log-warn
- repo: https://github.com/adrienverge/yamllint
- rev: v1.24.2
+ rev: v1.25.0
hooks:
- id: yamllint
name: Check yaml files with yamllint
entry: yamllint -c yamllint-config.yml --strict
types: [yaml]
- exclude: ^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/templates/.*\.yaml$
+ exclude:
+ ^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/(?:templates|files)/.*\.yaml
##
## Dear committer.
##
diff --git a/airflow/www_rbac/webpack.config.js b/airflow/www_rbac/webpack.config.js
index 2daf628..d58c089 100644
--- a/airflow/www_rbac/webpack.config.js
+++ b/airflow/www_rbac/webpack.config.js
@@ -1,4 +1,4 @@
-/**
+/*!
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
diff --git a/chart/tests/conftest.py b/chart/tests/conftest.py
index c1a06d8..db1f22f 100644
--- a/chart/tests/conftest.py
+++ b/chart/tests/conftest.py
@@ -15,14 +15,10 @@
# specific language governing permissions and limitations
# under the License.
-import os
import subprocess
import sys
-import pytest
-# We should set these before loading _any_ of the rest of airflow so that the
-# unit test mode config is set as early as possible.
-tests_directory = os.path.dirname(os.path.realpath(__file__))
+import pytest
@pytest.fixture(autouse=True, scope="session")
diff --git a/chart/tests/dags-persistent-volume-claim_test.yaml b/chart/tests/dags-persistent-volume-claim_test.yaml
deleted file mode 100644
index 2e24615..0000000
--- a/chart/tests/dags-persistent-volume-claim_test.yaml
+++ /dev/null
@@ -1,64 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-templates:
- - dags-persistent-volume-claim.yaml
-tests:
- - it: should not generate a document if persistence is disabled
- set:
- dags:
- persistence:
- enabled: false
- asserts:
- - hasDocuments:
- count: 0
- - it: should not generate a document when using an existingClaim
- set:
- dags:
- persistence:
- enabled: true
- existingClaim: test-claim
- asserts:
- - hasDocuments:
- count: 0
- - it: should generate a document if persistence is enabled & not using an existingClaim
- set:
- dags:
- persistence:
- enabled: true
- existingClaim: ~
- asserts:
- - hasDocuments:
- count: 1
- - it: should set PVC details correctly
- set:
- dags:
- persistence:
- enabled: true
- size: 1G
- existingClaim: ~
- storageClass: "MyStorageClass"
- accessMode: ReadWriteMany
- asserts:
- - equal:
- path: spec
- value:
- accessModes: ["ReadWriteMany"]
- resources:
- requests:
- storage: 1G
- storageClassName: "MyStorageClass"
diff --git a/chart/tests/git-sync-webserver_test.yaml b/chart/tests/git-sync-webserver_test.yaml
deleted file mode 100644
index bdbb7c6..0000000
--- a/chart/tests/git-sync-webserver_test.yaml
+++ /dev/null
@@ -1,66 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-templates:
- - webserver/webserver-deployment.yaml
-tests:
- - it: should add dags volume to the webserver if git sync & peristence is enabled
- set:
- dags:
- gitSync:
- enabled: true
- persistence:
- enabled: true
- asserts:
- - equal:
- path: spec.template.spec.volumes[1].name
- value: dags
- - it: should add dags volume to the webserver if git sync is enabled & peristence is disabled
- set:
- dags:
- gitSync:
- enabled: true
- persistence:
- enabled: false
- asserts:
- - equal:
- path: spec.template.spec.volumes[1].name
- value: dags
- - it: should add git sync container to webserver if persistence is not enabled, but git sync is
- set:
- dags:
- gitSync:
- enabled: true
- containerName: git-sync
- persistence:
- enabled: false
- asserts:
- - equal:
- path: spec.template.spec.containers[0].name
- value: git-sync
- - it: should not add sync container to webserver if git sync and persistence are enabled
- set:
- dags:
- gitSync:
- enabled: true
- container_name: git-sync
- persistence:
- enabled: true
- asserts:
- - notEqual:
- path: spec.template.spec.containers[0].name
- value: git-sync
diff --git a/chart/tests/git-sync-worker_test.yaml b/chart/tests/git-sync-worker_test.yaml
deleted file mode 100644
index 847a4dc..0000000
--- a/chart/tests/git-sync-worker_test.yaml
+++ /dev/null
@@ -1,70 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-templates:
- - workers/worker-deployment.yaml
-tests:
- - it: should add dags volume to the worker if git sync & peristence is enabled
- set:
- executor: CeleryExecutor
- dags:
- persistence:
- enabled: true
- gitSync:
- enabled: true
- asserts:
- - equal:
- path: spec.template.spec.volumes[1].name
- value: dags
- - it: should add dags volume to the worker if git sync is enabled & peristence is disabled
- set:
- executor: CeleryExecutor
- dags:
- gitSync:
- enabled: true
- persistence:
- enabled: false
- asserts:
- - equal:
- path: spec.template.spec.volumes[1].name
- value: dags
- - it: should add git sync container to worker if persistence is not enabled, but git sync is
- set:
- executor: CeleryExecutor
- dags:
- gitSync:
- enabled: true
- containerName: git-sync
- persistence:
- enabled: false
- asserts:
- - equal:
- path: spec.template.spec.containers[0].name
- value: git-sync
- - it: should not add sync container to worker if git sync and persistence are enabled
- set:
- executor: CeleryExecutor
- dags:
- gitSync:
- enabled: true
- containerName: git-sync
- persistence:
- enabled: true
- asserts:
- - notEqual:
- path: spec.template.spec.containers[0].name
- value: git-sync
diff --git a/chart/tests/pod-template-file_test.yaml b/chart/tests/pod-template-file_test.yaml
deleted file mode 100644
index 64e99f8..0000000
--- a/chart/tests/pod-template-file_test.yaml
+++ /dev/null
@@ -1,149 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-templates:
- - pod-template-file.yaml
-tests:
- - it: should work
- asserts:
- - isKind:
- of: Pod
- - equal:
- path: spec.containers[0].image
- value: dummy_image
- - equal:
- path: spec.containers[0].name
- value: base
- - it: should add an initContainer if gitSync is true
- set:
- dags:
- gitSync:
- enabled: true
- containerName: git-sync-test
- containerTag: test-tag
- containerRepository: test-registry/test-repo
- wait: 66
- maxFailures: 70
- subPath: "path1/path2"
- dest: "test-dest"
- root: "/git-root"
- rev: HEAD
- depth: 1
- repo: https://github.com/apache/airflow.git
- branch: test-branch
- sshKeySecret: ~
- credentialsSecret: ~
- knownHosts: ~
- asserts:
- - isKind:
- of: Pod
- - equal:
- path: spec.initContainers[0]
- value:
- name: git-sync-test
- image: test-registry/test-repo:test-tag
- env:
- - name: GIT_SYNC_REV
- value: HEAD
- - name: GIT_SYNC_BRANCH
- value: test-branch
- - name: GIT_SYNC_REPO
- value: https://github.com/apache/airflow.git
- - name: GIT_SYNC_DEPTH
- value: "1"
- - name: GIT_SYNC_ROOT
- value: /git-root
- - name: GIT_SYNC_DEST
- value: test-dest
- - name: GIT_SYNC_ADD_USER
- value: "true"
- - name: GIT_SYNC_WAIT
- value: "66"
- - name: GIT_SYNC_MAX_SYNC_FAILURES
- value: "70"
- volumeMounts:
- - mountPath: /git-root
- name: dags
- - it: validate if ssh params are added
- set:
- dags:
- gitSync:
- enabled: true
- containerName: git-sync-test
- sshKeySecret: ssh-secret
- knownHosts: ~
- branch: test-branch
- asserts:
- - contains:
- path: spec.initContainers[0].env
- content:
- name: GIT_SSH_KEY_FILE
- value: "/etc/git-secret/ssh"
- - contains:
- path: spec.initContainers[0].env
- content:
- name: GIT_SYNC_SSH
- value: "true"
- - contains:
- path: spec.initContainers[0].env
- content:
- name: GIT_KNOWN_HOSTS
- value: "false"
- - contains:
- path: spec.volumes
- content:
- name: git-sync-ssh-key
- secret:
- secretName: ssh-secret
- defaultMode: 288
- - it: should set username and pass env variables
- set:
- dags:
- gitSync:
- enabled: true
- credentialsSecret: user-pass-secret
- sshKeySecret: ~
- asserts:
- - contains:
- path: spec.initContainers[0].env
- content:
- name: GIT_SYNC_USERNAME
- valueFrom:
- secretKeyRef:
- name: user-pass-secret
- key: GIT_SYNC_USERNAME
- - contains:
- path: spec.initContainers[0].env
- content:
- name: GIT_SYNC_PASSWORD
- valueFrom:
- secretKeyRef:
- name: user-pass-secret
- key: GIT_SYNC_PASSWORD
- - it: should set the volume claim correctly when using an existing claim
- set:
- dags:
- persistence:
- enabled: true
- existingClaim: test-claim
- asserts:
- - contains:
- path: spec.volumes
- content:
- name: dags
- persistentVolumeClaim:
- claimName: test-claim
diff --git a/chart/tests/test_basic_helm_chart.py b/chart/tests/test_basic_helm_chart.py
index ea02c91..dcf20cf 100644
--- a/chart/tests/test_basic_helm_chart.py
+++ b/chart/tests/test_basic_helm_chart.py
@@ -17,7 +17,7 @@
import unittest
-from .helm_template_generator import render_chart
+from tests.helm_template_generator import render_chart
OBJECT_COUNT_IN_BASIC_DEPLOYMENT = 22
diff --git a/chart/tests/test_celery_kubernetes_executor.py b/chart/tests/test_celery_kubernetes_executor.py
new file mode 100644
index 0000000..7eae16e
--- /dev/null
+++ b/chart/tests/test_celery_kubernetes_executor.py
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class CeleryKubernetesExecutorTest(unittest.TestCase):
+ def test_should_create_a_worker_deployment_with_the_celery_executo(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "dags": {"persistence": {"enabled": True}, "gitSync": {"enabled": True}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+
+ def test_should_create_a_worker_deployment_with_the_celery_kubernetes_executor(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryKubernetesExecutor",
+ "dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
diff --git a/scripts/ci/kubernetes/ci_run_helm_testing.sh b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
old mode 100755
new mode 100644
similarity index 55%
copy from scripts/ci/kubernetes/ci_run_helm_testing.sh
copy to chart/tests/test_celery_kubernetes_pod_launcher_role.py
index 93d8337..535be11
--- a/scripts/ci/kubernetes/ci_run_helm_testing.sh
+++ b/chart/tests/test_celery_kubernetes_pod_launcher_role.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.
-echo "Running helm tests"
+import unittest
-chart_directory="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../../../chart/"
+import jmespath
+from tests.helm_template_generator import render_chart
-cat chart/files/pod-template-file.kubernetes-helm-yaml > chart/templates/pod-template-file.yaml
-docker run -w /airflow-chart -v "$chart_directory":/airflow-chart \
- --entrypoint /bin/sh \
- aneeshkj/helm-unittest \
- -c "helm repo add stable https://kubernetes-charts.storage.googleapis.com; helm dependency update ; helm unittest ." \
- && rm chart/templates/pod-template-file.yaml
+class CeleryKubernetesPodLauncherRole(unittest.TestCase):
+ def test_should_allow_both_scheduler_pod_launching_and_worker_pod_launching(self):
+ docs = render_chart(
+ values={"executor": "CeleryKubernetesExecutor"},
+ show_only=[
+ "templates/rbac/pod-launcher-rolebinding.yaml",
+ ],
+ )
+
+ self.assertEqual(jmespath.search("subjects[0].name", docs[0]), "RELEASE-NAME-scheduler")
+ self.assertEqual(jmespath.search("subjects[1].name", docs[0]), "RELEASE-NAME-worker")
diff --git a/chart/tests/test_dags_persistent_volume_claim.py b/chart/tests/test_dags_persistent_volume_claim.py
new file mode 100644
index 0000000..069a0cd
--- /dev/null
+++ b/chart/tests/test_dags_persistent_volume_claim.py
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class DagsPersistentVolumeClaimTest(unittest.TestCase):
+ def test_should_not_generate_a_document_if_persistence_is_disabled(self):
+ docs = render_chart(
+ values={"dags": {"persistence": {"enabled": False}}},
+ show_only=["templates/dags-persistent-volume-claim.yaml"],
+ )
+
+ self.assertEqual(0, len(docs))
+
+ def test_should_not_generate_a_document_when_using_an_existing_claim(self):
+ docs = render_chart(
+ values={"dags": {"persistence": {"enabled": True, "existingClaim": "test-claim"}}},
+ show_only=["templates/dags-persistent-volume-claim.yaml"],
+ )
+
+ self.assertEqual(0, len(docs))
+
+ def test_should_generate_a_document_if_persistence_is_enabled_and_not_using_an_existing_claim(self):
+ docs = render_chart(
+ values={"dags": {"persistence": {"enabled": True, "existingClaim": None}}},
+ show_only=["templates/dags-persistent-volume-claim.yaml"],
+ )
+
+ self.assertEqual(1, len(docs))
+
+ def test_should_set_pvc_details_correctly(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "persistence": {
+ "enabled": True,
+ "size": "1G",
+ "existingClaim": None,
+ "storageClassName": "MyStorageClass",
+ "accessMode": "ReadWriteMany",
+ }
+ }
+ },
+ show_only=["templates/dags-persistent-volume-claim.yaml"],
+ )
+
+ self.assertEqual(
+ {
+ "accessModes": ["ReadWriteMany"],
+ "resources": {"requests": {"storage": "1G"}},
+ "storageClassName": "MyStorageClass",
+ },
+ jmespath.search("spec", docs[0]),
+ )
diff --git a/chart/tests/test_git_sync_scheduler.py b/chart/tests/test_git_sync_scheduler.py
index 116ae10..a01c0f2 100644
--- a/chart/tests/test_git_sync_scheduler.py
+++ b/chart/tests/test_git_sync_scheduler.py
@@ -17,210 +17,143 @@
import unittest
-import yaml
-from kubernetes.client import models as k8s
+import jmespath
+from tests.helm_template_generator import render_chart
-from .helm_template_generator import render_chart, render_k8s_object
-OBJECT_COUNT_IN_BASIC_DEPLOYMENT = 22
-
-git_sync_basic = """
-dags:
- gitSync:
- enabled: true
-"""
-
-git_sync_existing_claim = """
-dags:
- persistence:
- enabled: true
- existingClaim: test-claim
-"""
-
-git_sync_ssh_params = """
-dags:
- gitSync:
- enabled: true
- containerName: git-sync-test
- sshKeySecret: ssh-secret
- knownHosts: ~
- branch: test-branch
-"""
-
-git_sync_username = """
-dags:
- gitSync:
- enabled: true
- credentialsSecret: user-pass-secret
- sshKeySecret: ~
-"""
-
-git_sync_container_spec = """
-images:
- gitSync:
- repository: test-registry/test-repo
- tag: test-tag
-dags:
- gitSync:
- enabled: true
- containerName: git-sync-test
- wait: 66
- maxFailures: 70
- subPath: "path1/path2"
- dest: "test-dest"
- root: "/git-root"
- rev: HEAD
- depth: 1
- repo: https://github.com/apache/airflow.git
- branch: test-branch
- sshKeySecret: ~
- credentialsSecret: ~
- knownHosts: ~
- persistence:
- enabled: true
-"""
-
-
-class TestGitSyncScheduler(unittest.TestCase):
- def test_basic(self):
- helm_settings = yaml.safe_load(git_sync_basic)
- res = render_chart(
- 'GIT-SYNC', helm_settings, show_only=["templates/scheduler/scheduler-deployment.yaml"]
+class GitSyncSchedulerTest(unittest.TestCase):
+ def test_should_add_dags_volume(self):
+ docs = render_chart(
+ values={"dags": {"gitSync": {"enabled": True}}},
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
)
- dep = render_k8s_object(res[0], k8s.V1Deployment)
- self.assertEqual("dags", dep.spec.template.spec.volumes[1].name)
- def test_git_container_spec(self):
- helm_settings = yaml.safe_load(git_sync_container_spec)
- res = render_chart(
- 'GIT-SYNC', helm_settings, show_only=["templates/scheduler/scheduler-deployment.yaml"]
- )
- dep = render_k8s_object(res[0], k8s.V1Deployment)
- git_sync_container = dep.spec.template.spec.containers[1]
- self.assertEqual(git_sync_container.image, "test-registry/test-repo:test-tag")
- self.assertEqual(git_sync_container.name, "git-sync-test")
- self.assertEqual(git_sync_container.security_context.run_as_user, 65533)
- env_dict = [e.to_dict() for e in git_sync_container.env]
- self.assertEqual(
- env_dict,
- [
- {'name': 'GIT_SYNC_REV', 'value': 'HEAD', 'value_from': None},
- {'name': 'GIT_SYNC_BRANCH', 'value': 'test-branch', 'value_from': None},
- {
- 'name': 'GIT_SYNC_REPO',
- 'value': 'https://github.com/apache/airflow.git',
- 'value_from': None,
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+
+ def test_validate_the_git_sync_container_spec(self):
+ docs = render_chart(
+ values={
+ "images": {
+ "gitSync": {
+ "repository": "test-registry/test-repo",
+ "tag": "test-tag",
+ "pullPolicy": "Allways",
+ }
+ },
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "containerName": "git-sync-test",
+ "wait": 66,
+ "maxFailures": 70,
+ "subPath": "path1/path2",
+ "dest": "test-dest",
+ "root": "/git-root",
+ "rev": "HEAD",
+ "depth": 1,
+ "repo": "https://github.com/apache/airflow.git",
+ "branch": "test-branch",
+ "sshKeySecret": None,
+ "credentialsSecret": None,
+ "knownHosts": None,
+ },
+ "persistence": {"enabled": True},
},
- {'name': 'GIT_SYNC_DEPTH', 'value': '1', 'value_from': None},
- {'name': 'GIT_SYNC_ROOT', 'value': '/git-root', 'value_from': None},
- {'name': 'GIT_SYNC_DEST', 'value': 'test-dest', 'value_from': None},
- {'name': 'GIT_SYNC_ADD_USER', 'value': 'true', 'value_from': None},
- {'name': 'GIT_SYNC_WAIT', 'value': '66', 'value_from': None},
- {'name': 'GIT_SYNC_MAX_SYNC_FAILURES', 'value': '70', 'value_from': None},
- ],
+ },
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
)
self.assertEqual(
- git_sync_container.volume_mounts, [k8s.V1VolumeMount(name="dags", mount_path="/git-root")]
+ {
+ "name": "git-sync-test",
+ "securityContext": {"runAsUser": 65533},
+ "image": "test-registry/test-repo:test-tag",
+ "imagePullPolicy": "Allways",
+ "env": [
+ {"name": "GIT_SYNC_REV", "value": "HEAD"},
+ {"name": "GIT_SYNC_BRANCH", "value": "test-branch"},
+ {"name": "GIT_SYNC_REPO", "value": "https://github.com/apache/airflow.git"},
+ {"name": "GIT_SYNC_DEPTH", "value": "1"},
+ {"name": "GIT_SYNC_ROOT", "value": "/git-root"},
+ {"name": "GIT_SYNC_DEST", "value": "test-dest"},
+ {"name": "GIT_SYNC_ADD_USER", "value": "true"},
+ {"name": "GIT_SYNC_WAIT", "value": "66"},
+ {"name": "GIT_SYNC_MAX_SYNC_FAILURES", "value": "70"},
+ ],
+ "volumeMounts": [{"mountPath": "/git-root", "name": "dags"}],
+ },
+ jmespath.search("spec.template.spec.containers[1]", docs[0]),
)
- def test_ssh_params_added(self):
- helm_settings = yaml.safe_load(git_sync_ssh_params)
- res = render_chart(
- 'GIT-SYNC', helm_settings, show_only=["templates/scheduler/scheduler-deployment.yaml"]
+ def test_validate_if_ssh_params_are_added(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "containerName": "git-sync-test",
+ "sshKeySecret": "ssh-secret",
+ "knownHosts": None,
+ "branch": "test-branch",
+ }
+ }
+ },
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
)
- dep = render_k8s_object(res[0], k8s.V1Deployment)
- git_sync_container = dep.spec.template.spec.containers[1]
- env_dict = [e.to_dict() for e in git_sync_container.env]
- self.assertEqual(
- env_dict,
- [
- {'name': 'GIT_SSH_KEY_FILE', 'value': '/etc/git-secret/ssh', 'value_from': None},
- {'name': 'GIT_SYNC_SSH', 'value': 'true', 'value_from': None},
- {'name': 'GIT_KNOWN_HOSTS', 'value': 'false', 'value_from': None},
- {'name': 'GIT_SYNC_REV', 'value': 'HEAD', 'value_from': None},
- {'name': 'GIT_SYNC_BRANCH', 'value': 'test-branch', 'value_from': None},
- {
- 'name': 'GIT_SYNC_REPO',
- 'value': 'https://github.com/apache/airflow.git',
- 'value_from': None,
- },
- {'name': 'GIT_SYNC_DEPTH', 'value': '1', 'value_from': None},
- {'name': 'GIT_SYNC_ROOT', 'value': '/git', 'value_from': None},
- {'name': 'GIT_SYNC_DEST', 'value': 'repo', 'value_from': None},
- {'name': 'GIT_SYNC_ADD_USER', 'value': 'true', 'value_from': None},
- {'name': 'GIT_SYNC_WAIT', 'value': '60', 'value_from': None},
- {'name': 'GIT_SYNC_MAX_SYNC_FAILURES', 'value': '0', 'value_from': None},
- ],
+
+ self.assertIn(
+ {"name": "GIT_SSH_KEY_FILE", "value": "/etc/git-secret/ssh"},
+ jmespath.search("spec.template.spec.containers[1].env", docs[0]),
+ )
+ self.assertIn(
+ {"name": "GIT_SYNC_SSH", "value": "true"},
+ jmespath.search("spec.template.spec.containers[1].env", docs[0]),
+ )
+ self.assertIn(
+ {"name": "GIT_KNOWN_HOSTS", "value": "false"},
+ jmespath.search("spec.template.spec.containers[1].env", docs[0]),
+ )
+ self.assertIn(
+ {"name": "git-sync-ssh-key", "secret": {"secretName": "ssh-secret", "defaultMode": 288}},
+ jmespath.search("spec.template.spec.volumes", docs[0]),
)
- def test_adds_git_username(self):
- helm_settings = yaml.safe_load(git_sync_username)
- res = render_chart(
- 'GIT-SYNC', helm_settings, show_only=["templates/scheduler/scheduler-deployment.yaml"]
+ def test_should_set_username_and_pass_env_variables(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "credentialsSecret": "user-pass-secret",
+ "sshKeySecret": None,
+ }
+ }
+ },
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
)
- dep = render_k8s_object(res[0], k8s.V1Deployment)
- git_sync_container = dep.spec.template.spec.containers[1]
- env_dict = [e.to_dict() for e in git_sync_container.env]
- self.assertEqual(
- env_dict,
- [
- {
- 'name': 'GIT_SYNC_USERNAME',
- 'value': None,
- 'value_from': {
- 'config_map_key_ref': None,
- 'field_ref': None,
- 'resource_field_ref': None,
- 'secret_key_ref': {
- 'key': 'GIT_SYNC_USERNAME',
- 'name': 'user-pass-secret',
- 'optional': None,
- },
- },
- },
- {
- 'name': 'GIT_SYNC_PASSWORD',
- 'value': None,
- 'value_from': {
- 'config_map_key_ref': None,
- 'field_ref': None,
- 'resource_field_ref': None,
- 'secret_key_ref': {
- 'key': 'GIT_SYNC_PASSWORD',
- 'name': 'user-pass-secret',
- 'optional': None,
- },
- },
- },
- {'name': 'GIT_SYNC_REV', 'value': 'HEAD', 'value_from': None},
- {'name': 'GIT_SYNC_BRANCH', 'value': 'v1-10-stable', 'value_from': None},
- {
- 'name': 'GIT_SYNC_REPO',
- 'value': 'https://github.com/apache/airflow.git',
- 'value_from': None,
- },
- {'name': 'GIT_SYNC_DEPTH', 'value': '1', 'value_from': None},
- {'name': 'GIT_SYNC_ROOT', 'value': '/git', 'value_from': None},
- {'name': 'GIT_SYNC_DEST', 'value': 'repo', 'value_from': None},
- {'name': 'GIT_SYNC_ADD_USER', 'value': 'true', 'value_from': None},
- {'name': 'GIT_SYNC_WAIT', 'value': '60', 'value_from': None},
- {'name': 'GIT_SYNC_MAX_SYNC_FAILURES', 'value': '0', 'value_from': None},
- ],
+
+ self.assertIn(
+ {
+ "name": "GIT_SYNC_USERNAME",
+ "valueFrom": {"secretKeyRef": {"name": "user-pass-secret", "key": "GIT_SYNC_USERNAME"}},
+ },
+ jmespath.search("spec.template.spec.containers[1].env", docs[0]),
+ )
+ self.assertIn(
+ {
+ "name": "GIT_SYNC_PASSWORD",
+ "valueFrom": {"secretKeyRef": {"name": "user-pass-secret", "key": "GIT_SYNC_PASSWORD"}},
+ },
+ jmespath.search("spec.template.spec.containers[1].env", docs[0]),
)
- def test_set_volume_claim_to_existing_claim(self):
- helm_settings = yaml.safe_load(git_sync_existing_claim)
- res = render_chart(
- 'GIT-SYNC', helm_settings, show_only=["templates/scheduler/scheduler-deployment.yaml"]
+ def test_should_set_the_volume_claim_correctly_when_using_an_existing_claim(self):
+ docs = render_chart(
+ values={"dags": {"persistence": {"enabled": True, "existingClaim": "test-claim"}}},
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
)
- dep = render_k8s_object(res[0], k8s.V1Deployment)
- volume_map = {vol.name: vol for vol in dep.spec.template.spec.volumes}
- dag_volume = volume_map['dags']
- self.assertEqual(
- dag_volume,
- k8s.V1Volume(
- name="dags",
- persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='test-claim'),
- ),
+
+ self.assertIn(
+ {"name": "dags", "persistentVolumeClaim": {"claimName": "test-claim"}},
+ jmespath.search("spec.template.spec.volumes", docs[0]),
)
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
new file mode 100644
index 0000000..30c3f33
--- /dev/null
+++ b/chart/tests/test_git_sync_webserver.py
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class GitSyncWebserverTest(unittest.TestCase):
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_and_peristence_is_enabled(self):
+ docs = render_chart(
+ values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}}},
+ show_only=["templates/webserver/webserver-deployment.yaml"],
+ )
+
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+
+ def test_should_add_dags_volume_to_the_webserver_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ docs = render_chart(
+ values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}}},
+ show_only=["templates/webserver/webserver-deployment.yaml"],
+ )
+
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+
+ def test_should_add_git_sync_container_to_webserver_if_persistence_is_not_enabled_but_git_sync_is(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "gitSync": {"enabled": True, "containerName": "git-sync"},
+ "persistence": {"enabled": False},
+ }
+ },
+ show_only=["templates/webserver/webserver-deployment.yaml"],
+ )
+
+ self.assertEqual("git-sync", jmespath.search("spec.template.spec.containers[0].name", docs[0]))
+
+ def test_should_have_service_account_defined(self):
+ docs = render_chart(
+ values={"dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": True}}},
+ show_only=["templates/webserver/webserver-deployment.yaml"],
+ )
+
+ self.assertEqual(
+ "RELEASE-NAME-webserver", jmespath.search("spec.template.spec.serviceAccountName", docs[0])
+ )
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
new file mode 100644
index 0000000..714f385
--- /dev/null
+++ b/chart/tests/test_git_sync_worker.py
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class GitSyncWorkerTest(unittest.TestCase):
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_and_peristence_is_enabled(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "dags": {"persistence": {"enabled": True}, "gitSync": {"enabled": True}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+
+ def test_should_add_dags_volume_to_the_worker_if_git_sync_is_enabled_and_peristence_is_disabled(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "dags": {"gitSync": {"enabled": True}, "persistence": {"enabled": False}},
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("config", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+ self.assertEqual("dags", jmespath.search("spec.template.spec.volumes[2].name", docs[0]))
+
+ def test_should_add_git_sync_container_to_worker_if_persistence_is_not_enabled_but_git_sync_is(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "dags": {
+ "gitSync": {"enabled": True, "containerName": "git-sync"},
+ "persistence": {"enabled": False},
+ },
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("git-sync", jmespath.search("spec.template.spec.containers[0].name", docs[0]))
+
+ def test_should_not_add_sync_container_to_worker_if_git_sync_and_persistence_are_enabled(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "dags": {
+ "gitSync": {"enabled": True, "containerName": "git-sync"},
+ "persistence": {"enabled": True},
+ },
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertNotEqual("git-sync", jmespath.search("spec.template.spec.containers[0].name", docs[0]))
diff --git a/scripts/ci/kubernetes/ci_run_helm_testing.sh b/chart/tests/test_migrate_database_job.py
old mode 100755
new mode 100644
similarity index 60%
rename from scripts/ci/kubernetes/ci_run_helm_testing.sh
rename to chart/tests/test_migrate_database_job.py
index 93d8337..0524315
--- a/scripts/ci/kubernetes/ci_run_helm_testing.sh
+++ b/chart/tests/test_migrate_database_job.py
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.
-echo "Running helm tests"
+import unittest
-chart_directory="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../../../chart/"
+import jmespath
+from tests.helm_template_generator import render_chart
-cat chart/files/pod-template-file.kubernetes-helm-yaml > chart/templates/pod-template-file.yaml
-docker run -w /airflow-chart -v "$chart_directory":/airflow-chart \
- --entrypoint /bin/sh \
- aneeshkj/helm-unittest \
- -c "helm repo add stable https://kubernetes-charts.storage.googleapis.com; helm dependency update ; helm unittest ." \
- && rm chart/templates/pod-template-file.yaml
+class MigrateDatabaseJobTest(unittest.TestCase):
+ def test_should_run_by_default(self):
+ docs = render_chart(
+ values={},
+ show_only=["templates/migrate-database-job.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Job")
+ self.assertEqual(
+ "run-airflow-migrations", jmespath.search("spec.template.spec.containers[0].name", docs[0])
+ )
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
new file mode 100644
index 0000000..0673e08
--- /dev/null
+++ b/chart/tests/test_pod_template_file.py
@@ -0,0 +1,185 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from os import remove
+from os.path import realpath, dirname
+from shutil import copyfile
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+ROOT_FOLDER = realpath(dirname(realpath(__file__)) + "/..")
+
+
+class PodTemplateFileTest(unittest.TestCase):
+ def setUp(self):
+ copyfile(
+ ROOT_FOLDER + "/files/pod-template-file.kubernetes-helm-yaml",
+ ROOT_FOLDER + "/templates/pod-template-file.yaml",
+ )
+
+ def tearDown(self):
+ remove(ROOT_FOLDER + "/templates/pod-template-file.yaml")
+
+ def test_should_work(self):
+ docs = render_chart(
+ values={},
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertIsNotNone(jmespath.search("spec.containers[0].image", docs[0]))
+ self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
+
+ def test_should_add_an_init_container_if_git_sync_is_true(self):
+ docs = render_chart(
+ values={
+ "images": {
+ "gitSync": {
+ "repository": "test-registry/test-repo",
+ "tag": "test-tag",
+ "pullPolicy": "Allways",
+ }
+ },
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "containerName": "git-sync-test",
+ "wait": 66,
+ "maxFailures": 70,
+ "subPath": "path1/path2",
+ "dest": "test-dest",
+ "root": "/git-root",
+ "rev": "HEAD",
+ "depth": 1,
+ "repo": "https://github.com/apache/airflow.git",
+ "branch": "test-branch",
+ "sshKeySecret": None,
+ "credentialsSecret": None,
+ "knownHosts": None,
+ }
+ },
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertEqual(
+ {
+ "name": "git-sync-test",
+ "securityContext": {"runAsUser": 65533},
+ "image": "test-registry/test-repo:test-tag",
+ "imagePullPolicy": "Allways",
+ "env": [
+ {"name": "GIT_SYNC_REV", "value": "HEAD"},
+ {"name": "GIT_SYNC_BRANCH", "value": "test-branch"},
+ {"name": "GIT_SYNC_REPO", "value": "https://github.com/apache/airflow.git"},
+ {"name": "GIT_SYNC_DEPTH", "value": "1"},
+ {"name": "GIT_SYNC_ROOT", "value": "/git-root"},
+ {"name": "GIT_SYNC_DEST", "value": "test-dest"},
+ {"name": "GIT_SYNC_ADD_USER", "value": "true"},
+ {"name": "GIT_SYNC_WAIT", "value": "66"},
+ {"name": "GIT_SYNC_MAX_SYNC_FAILURES", "value": "70"},
+ {"name": "GIT_SYNC_ONE_TIME", "value": "true"},
+ ],
+ "volumeMounts": [{"mountPath": "/git-root", "name": "dags"}],
+ },
+ jmespath.search("spec.initContainers[0]", docs[0]),
+ )
+
+ def test_validate_if_ssh_params_are_added(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "containerName": "git-sync-test",
+ "sshKeySecret": "ssh-secret",
+ "knownHosts": None,
+ "branch": "test-branch",
+ }
+ }
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertIn(
+ {"name": "GIT_SSH_KEY_FILE", "value": "/etc/git-secret/ssh"},
+ jmespath.search("spec.initContainers[0].env", docs[0]),
+ )
+ self.assertIn(
+ {"name": "GIT_SYNC_SSH", "value": "true"}, jmespath.search("spec.initContainers[0].env", docs[0])
+ )
+ self.assertIn(
+ {"name": "GIT_KNOWN_HOSTS", "value": "false"},
+ jmespath.search("spec.initContainers[0].env", docs[0]),
+ )
+ self.assertIn(
+ {"name": "git-sync-ssh-key", "secret": {"secretName": "ssh-secret", "defaultMode": 288}},
+ jmespath.search("spec.volumes", docs[0]),
+ )
+
+ def test_should_set_username_and_pass_env_variables(self):
+ docs = render_chart(
+ values={
+ "dags": {
+ "gitSync": {
+ "enabled": True,
+ "credentialsSecret": "user-pass-secret",
+ "sshKeySecret": None,
+ }
+ }
+ },
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertIn(
+ {
+ "name": "GIT_SYNC_USERNAME",
+ "valueFrom": {"secretKeyRef": {"name": "user-pass-secret", "key": "GIT_SYNC_USERNAME"}},
+ },
+ jmespath.search("spec.initContainers[0].env", docs[0]),
+ )
+ self.assertIn(
+ {
+ "name": "GIT_SYNC_PASSWORD",
+ "valueFrom": {"secretKeyRef": {"name": "user-pass-secret", "key": "GIT_SYNC_PASSWORD"}},
+ },
+ jmespath.search("spec.initContainers[0].env", docs[0]),
+ )
+
+ def test_should_set_the_volume_claim_correctly_when_using_an_existing_claim(self):
+ docs = render_chart(
+ values={"dags": {"persistence": {"enabled": True, "existingClaim": "test-claim"}}},
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertIn(
+ {"name": "dags", "persistentVolumeClaim": {"claimName": "test-claim"}},
+ jmespath.search("spec.volumes", docs[0]),
+ )
+
+ def test_should_set_a_custom_image_in_pod_template(self):
+ docs = render_chart(
+ values={"images": {"pod_template": {"repository": "dummy_image", "tag": "latest"}}},
+ show_only=["templates/pod-template-file.yaml"],
+ )
+
+ self.assertRegex(docs[0]["kind"], "Pod")
+ self.assertEqual("dummy_image:latest", jmespath.search("spec.containers[0].image", docs[0]))
+ self.assertEqual("base", jmespath.search("spec.containers[0].name", docs[0]))
diff --git a/chart/tests/test_scheduler.py b/chart/tests/test_scheduler.py
new file mode 100644
index 0000000..9769848
--- /dev/null
+++ b/chart/tests/test_scheduler.py
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class SchedulerTest(unittest.TestCase):
+ def test_should_add_extra_volume_and_extra_volume_mount(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "scheduler": {
+ "extraVolumes": [{"name": "test-volume", "emptyDir": {}}],
+ "extraVolumeMounts": [{"name": "test-volume", "mountPath": "/opt/test"}],
+ },
+ },
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
+ )
+
+ self.assertEqual("test-volume", jmespath.search("spec.template.spec.volumes[1].name", docs[0]))
+ self.assertEqual(
+ "test-volume", jmespath.search("spec.template.spec.containers[0].volumeMounts[3].name", docs[0])
+ )
diff --git a/chart/tests/test_worker.py b/chart/tests/test_worker.py
new file mode 100644
index 0000000..2fc6d17
--- /dev/null
+++ b/chart/tests/test_worker.py
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from tests.helm_template_generator import render_chart
+
+
+class WorkerTest(unittest.TestCase):
+ def test_should_add_extra_volume_and_extra_volume_mount(self):
+ docs = render_chart(
+ values={
+ "executor": "CeleryExecutor",
+ "workers": {
+ "extraVolumes": [{"name": "test-volume", "emptyDir": {}}],
+ "extraVolumeMounts": [{"name": "test-volume", "mountPath": "/opt/test"}],
+ },
+ },
+ show_only=["templates/workers/worker-deployment.yaml"],
+ )
+
+ self.assertEqual("test-volume", jmespath.search("spec.template.spec.volumes[0].name", docs[0]))
+ self.assertEqual(
+ "test-volume", jmespath.search("spec.template.spec.containers[0].volumeMounts[0].name", docs[0])
+ )
diff --git a/docs/static/exampleinclude.css b/docs/static/exampleinclude.css
index 722ffec..b4f2a42 100644
--- a/docs/static/exampleinclude.css
+++ b/docs/static/exampleinclude.css
@@ -1,4 +1,4 @@
-/**
+/*!
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
diff --git a/docs/static/jira-links.js b/docs/static/jira-links.js
index 8dc4394..7d6d7f7 100644
--- a/docs/static/jira-links.js
+++ b/docs/static/jira-links.js
@@ -1,4 +1,4 @@
-/**
+/*!
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
[airflow] 07/44: Fix K8S CI job name rendering (#12007)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 33a14e7e666c6c9b92027a18c7c5941e52ad506f
Author: Xiaodong DENG <xd...@hotmail.com>
AuthorDate: Sun Nov 1 00:09:19 2020 +0100
Fix K8S CI job name rendering (#12007)
(cherry picked from commit 72f3fcb293bbaee405133a77a8c1d6cab87b9e62)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 596ab3e..630b5d6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -613,7 +613,7 @@ jobs:
tests-kubernetes:
timeout-minutes: 50
- name: "K8s: ${{matrix.python-version}} ${{matrix.kubernetes-version}} ${{matrix.kubernetes-mode}}"
+ name: K8s ${{matrix.python-version}} ${{matrix.kubernetes-version}} ${{matrix.kubernetes-mode}}
runs-on: ubuntu-latest
needs: [build-info, prod-images]
strategy:
[airflow] 11/44: Revise "Project Focus" copy (#12011)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 887c231332d2d425d7f81ba0973dc12c508d8a2c
Author: Ry Walker <42...@users.noreply.github.com>
AuthorDate: Sun Nov 1 12:34:34 2020 -0500
Revise "Project Focus" copy (#12011)
(cherry picked from commit 77ae32e311ebd9f6ab236cbeba42f7de3dd1e211)
---
README.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index ccac802..70bf2d8 100644
--- a/README.md
+++ b/README.md
@@ -59,11 +59,11 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
## Project Focus
-Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/).
+Airflow works best with workflows that are mostly static and slowly changing. When DAG structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](https://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/).
-Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work.
+Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e. results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external servic [...]
-Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space.
+Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.
## Principles
[airflow] 02/44: Updated documentation for the CI with mermaid
sequence diagrams (#10380)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 132e899a1132988e99281215fd6d9e9c37bd0369
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Aug 24 22:45:28 2020 +0200
Updated documentation for the CI with mermaid sequence diagrams (#10380)
(cherry picked from commit f2da6b419fde71d335d133031f2c0fac28b87289)
---
.pre-commit-config.yaml | 11 ++-
.rat-excludes | 1 +
images/ci/CI.png | Bin 0 -> 243004 bytes
images/ci/pull_request_ci_flow.md5 | 1 +
images/ci/pull_request_ci_flow.mermaid | 108 +++++++++++++++++++++++++
images/ci/pull_request_ci_flow.png | Bin 0 -> 195487 bytes
images/ci/push_ci_flow.md5 | 1 +
images/ci/push_ci_flow.mermaid | 117 ++++++++++++++++++++++++++++
images/ci/push_ci_flow.png | Bin 0 -> 209013 bytes
images/ci/scheduled_ci_flow.md5 | 1 +
images/ci/scheduled_ci_flow.mermaid | 111 ++++++++++++++++++++++++++
images/ci/scheduled_ci_flow.png | Bin 0 -> 201540 bytes
scripts/ci/pre_commit/pre_commit_mermaid.sh | 97 +++++++++++++++++++++++
13 files changed, 447 insertions(+), 1 deletion(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 1b5c436..6763164 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -128,6 +128,15 @@ repos:
- --license-filepath
- license-templates/LICENSE.txt
- --fuzzy-match-generates-todo
+ - id: insert-license
+ name: Add license for all mermaid files
+ args:
+ - --comment-style
+ - "|%%|"
+ - --license-filepath
+ - license-templates/LICENSE.txt
+ - --fuzzy-match-generates-todo
+ files: \.mermaid$
- repo: https://github.com/thlorenz/doctoc.git
rev: v1.4.0
hooks:
@@ -277,7 +286,7 @@ repos:
entry: ./scripts/ci/pre_commit/pre_commit_check_integrations.sh
language: system
pass_filenames: false
- files: ^airflow/scripts/ci/libraries/_initialization.sh$|^breeze-complete$
+ files: ^common/_common_values.sh$|^breeze-complete$
- id: check-apache-license
name: Check if licenses are OK for Apache
entry: "./scripts/ci/pre_commit/pre_commit_check_license.sh"
diff --git a/.rat-excludes b/.rat-excludes
index 497d7ed..69c8ccc 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -28,6 +28,7 @@ metastore_db
.*sql
.*svg
.*csv
+.*md5
CHANGELOG.txt
.*zip
.*lock
diff --git a/images/ci/CI.png b/images/ci/CI.png
new file mode 100644
index 0000000..7123519
Binary files /dev/null and b/images/ci/CI.png differ
diff --git a/images/ci/pull_request_ci_flow.md5 b/images/ci/pull_request_ci_flow.md5
new file mode 100644
index 0000000..3fe05dd
--- /dev/null
+++ b/images/ci/pull_request_ci_flow.md5
@@ -0,0 +1 @@
+e3aa93e8b46ca97b4aad06de23a10ad5 images/ci/pull_request_ci_flow.mermaid
diff --git a/images/ci/pull_request_ci_flow.mermaid b/images/ci/pull_request_ci_flow.mermaid
new file mode 100644
index 0000000..2e4bf9c
--- /dev/null
+++ b/images/ci/pull_request_ci_flow.mermaid
@@ -0,0 +1,108 @@
+%% Licensed to the Apache Software Foundation (ASF) under one
+%% or more contributor license agreements. See the NOTICE file
+%% distributed with this work for additional information
+%% regarding copyright ownership. The ASF licenses this file
+%% to you under the Apache License, Version 2.0 (the
+%% "License"); you may not use this file except in compliance
+%% with the License. You may obtain a copy of the License at
+%%
+%% http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing,
+%% software distributed under the License is distributed on an
+%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+%% KIND, either express or implied. See the License for the
+%% specific language governing permissions and limitations
+%% under the License.
+
+sequenceDiagram
+ Contributor -->> Fork: Submit PR
+ Fork -->> Airflow Repo: Start Pull Request
+ activate Airflow Repo
+ Note over Airflow Repo: Trigger PR build
+ Note over CI Build: pull_request<br>[Read Token]
+ Note over Build Images: workflow_run<br>[Write Token]
+ Airflow Repo -->> CI Build: Trigger 'pull_request' event
+ activate CI Build
+ CI Build -->> Build Images: Trigger 'workflow_run' event
+ activate Build Images
+ Note over Build Images: Find Duplicate<br>[CI Build]
+ opt Duplicated builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find Duplicates<br>[Build Image]
+ opt Duplicated builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Find failed<br>[CI Build]
+ opt Failed builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find failed<br>[Build Image]
+ opt Failed builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Build info
+ par 3.6, 3.7, 3.8
+ Note over Build Images: Build PROD Images<br>[GITHUB_RUN_ID]
+ and
+ Note over Build Images: Build CI Images<br>[GITHUB_RUN_ID]
+ end
+ par No CI image
+ Note over CI Build: Build info
+ and
+ Note over CI Build: Should trigger tests?
+ and
+ Note over CI Build: Helm tests
+ and
+ Note over CI Build: OpenAPI client gen
+ end
+ par 3.6, 3.7, 3.8
+ Build Images ->> GitHub Registry: Push CI Images
+ activate GitHub Registry
+ and
+ Build Images ->> GitHub Registry: Push PROD Images
+ end
+ opt If any step failed
+ Build Images -->> CI Build: Cancel triggering run
+ end
+ deactivate Build Images
+ Note over GitHub Registry: Tagged Images<br>[GITHUB_RUN_ID]
+ loop Wait for images
+ par 3.6, 3.7, 3.8
+ CI Build ->> CI Build: Pull CI Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ and
+ CI Build ->> CI Build: Pull PROD Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ end
+ end
+ par 3.6, 3.7, 3.8
+ GitHub Registry ->> CI Build: Pull CI Images
+ and
+ GitHub Registry ->> CI Build: Pull PROD Images
+ end
+ deactivate GitHub Registry
+ par 3.6, 3.7, 3.8
+ Note over CI Build: Run static checks
+ and
+ Note over CI Build: Build docs
+ and
+ Note over CI Build: Spell check docs
+ and
+ Note over CI Build: Backport packages
+ and
+ opt Triggered?
+ Note over CI Build: Tests
+ end
+ and
+ opt Triggered?
+ Note over CI Build: Kubernetes Tests
+ end
+ end
+ Note over CI Build: Quarantined tests
+ Note over CI Build: Merge Coverage
+ CI Build -->> Coverage.io: Upload Coverage
+ CI Build -->> Airflow Repo: Status Check for CI Build
+ deactivate Airflow Repo
+ deactivate CI Build
diff --git a/images/ci/pull_request_ci_flow.png b/images/ci/pull_request_ci_flow.png
new file mode 100644
index 0000000..7fec85a
Binary files /dev/null and b/images/ci/pull_request_ci_flow.png differ
diff --git a/images/ci/push_ci_flow.md5 b/images/ci/push_ci_flow.md5
new file mode 100644
index 0000000..edaa0cc
--- /dev/null
+++ b/images/ci/push_ci_flow.md5
@@ -0,0 +1 @@
+95db6b17e315a47318eaf6260b3f0fd3 images/ci/push_ci_flow.mermaid
diff --git a/images/ci/push_ci_flow.mermaid b/images/ci/push_ci_flow.mermaid
new file mode 100644
index 0000000..790f800
--- /dev/null
+++ b/images/ci/push_ci_flow.mermaid
@@ -0,0 +1,117 @@
+%% Licensed to the Apache Software Foundation (ASF) under one
+%% or more contributor license agreements. See the NOTICE file
+%% distributed with this work for additional information
+%% regarding copyright ownership. The ASF licenses this file
+%% to you under the Apache License, Version 2.0 (the
+%% "License"); you may not use this file except in compliance
+%% with the License. You may obtain a copy of the License at
+%%
+%% http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing,
+%% software distributed under the License is distributed on an
+%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+%% KIND, either express or implied. See the License for the
+%% specific language governing permissions and limitations
+%% under the License.
+
+sequenceDiagram
+ Committer -->> Airflow Repo: Merge PR
+ activate Airflow Repo
+ Note over Airflow Repo: Trigger PR build
+ Note over CI Build: push<br>[Write Token]
+ Note over Build Images: workflow_run<br>[Write Token]
+ Airflow Repo -->> CI Build: Trigger 'pull_request' event
+ activate CI Build
+ CI Build -->> Build Images: Trigger 'workflow_run' event
+ activate Build Images
+ Note over Build Images: Find Duplicate<br>[CI Build]
+ opt Duplicated builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find Duplicates<br>[Build Image]
+ opt Duplicated builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Find failed<br>[CI Build]
+ opt Failed builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find failed<br>[Build Image]
+ opt Failed builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Build info
+ par 3.6, 3.7, 3.8
+ Note over Build Images: Update constraints
+ Note over Build Images: Build PROD Images<br>[GITHUB_RUN_ID]
+ and
+ Note over Build Images: Update constraints
+ Note over Build Images: Build CI Images<br>[GITHUB_RUN_ID]
+ end
+ par No CI image
+ Note over CI Build: Build info
+ and
+ Note over CI Build: Test always!
+ and
+ Note over CI Build: Helm tests
+ and
+ Note over CI Build: OpenAPI client gen
+ end
+ par 3.6, 3.7, 3.8
+ Build Images ->> GitHub Registry: Push CI Images
+ activate GitHub Registry
+ and
+ Build Images ->> GitHub Registry: Push PROD Images
+ end
+ opt If any step failed
+ Build Images -->> CI Build: Cancel triggering run
+ end
+ deactivate Build Images
+ Note over GitHub Registry: Tagged Images<br>[GITHUB_RUN_ID]
+ loop Wait for images
+ par 3.6, 3.7, 3.8
+ CI Build ->> CI Build: Pull CI Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ and
+ CI Build ->> CI Build: Pull PROD Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ end
+ end
+ par 3.6, 3.7, 3.8
+ GitHub Registry ->> CI Build: Pull CI Images
+ and
+ GitHub Registry ->> CI Build: Pull PROD Images
+ end
+ deactivate GitHub Registry
+ par 3.6, 3.7, 3.8
+ Note over CI Build: Run static checks
+ and
+ Note over CI Build: Build docs
+ and
+ Note over CI Build: Spell check docs
+ and
+ Note over CI Build: Backport packages
+ and
+ Note over CI Build: Tests
+ and
+ Note over CI Build: Kubernetes Tests
+ end
+ Note over CI Build: Quarantined tests
+ Note over CI Build: Merge Coverage
+ CI Build -->> Coverage.io: Upload Coverage
+ Note over CI Build: Tag image<br>[COMMIT_SHA]
+ par 3.6, 3.7, 3.8
+ CI Build ->> GitHub Registry: Push CI Images
+ activate GitHub Registry
+ and
+ CI Build ->> GitHub Registry: Push PROD Images
+ end
+ Note over GitHub Registry: Tagged Images<br>[COMMIT_SHA]
+ par 3.6, 3.7, 3.8
+ CI Build ->> Airflow Repo: Push Constraints
+ end
+ deactivate GitHub Registry
+ CI Build -->> Airflow Repo: Status Check for CI Build
+ deactivate Airflow Repo
+ deactivate CI Build
diff --git a/images/ci/push_ci_flow.png b/images/ci/push_ci_flow.png
new file mode 100644
index 0000000..81c19bb
Binary files /dev/null and b/images/ci/push_ci_flow.png differ
diff --git a/images/ci/scheduled_ci_flow.md5 b/images/ci/scheduled_ci_flow.md5
new file mode 100644
index 0000000..8ae6938
--- /dev/null
+++ b/images/ci/scheduled_ci_flow.md5
@@ -0,0 +1 @@
+f6ea87f0b4d990fd48a9dbec2a2a4f2d images/ci/scheduled_ci_flow.mermaid
diff --git a/images/ci/scheduled_ci_flow.mermaid b/images/ci/scheduled_ci_flow.mermaid
new file mode 100644
index 0000000..b892a7a
--- /dev/null
+++ b/images/ci/scheduled_ci_flow.mermaid
@@ -0,0 +1,111 @@
+%% Licensed to the Apache Software Foundation (ASF) under one
+%% or more contributor license agreements. See the NOTICE file
+%% distributed with this work for additional information
+%% regarding copyright ownership. The ASF licenses this file
+%% to you under the Apache License, Version 2.0 (the
+%% "License"); you may not use this file except in compliance
+%% with the License. You may obtain a copy of the License at
+%%
+%% http://www.apache.org/licenses/LICENSE-2.0
+%%
+%% Unless required by applicable law or agreed to in writing,
+%% software distributed under the License is distributed on an
+%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+%% KIND, either express or implied. See the License for the
+%% specific language governing permissions and limitations
+%% under the License.
+
+sequenceDiagram
+ CRON job -->> Airflow Repo: Nightly run
+ activate Airflow Repo
+ Note over Airflow Repo: Trigger PR build
+ Note over CI Build: push<br>[Write Token]
+ Note over Build Images: workflow_run<br>[Write Token]
+ Airflow Repo -->> CI Build: Trigger 'pull_request' event
+ activate CI Build
+ CI Build -->> Build Images: Trigger 'workflow_run' event
+ activate Build Images
+ Note over Build Images: Find Duplicate<br>[CI Build]
+ opt Duplicated builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find Duplicates<br>[Build Image]
+ opt Duplicated builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Find failed<br>[CI Build]
+ opt Failed builds
+ Build Images -->> CI Build: Cancel
+ end
+ Note over Build Images: Find failed<br>[Build Image]
+ opt Failed builds
+ Build Images -->> Build Images: Cancel
+ end
+ Note over Build Images: Build info
+ par 3.6, 3.7, 3.8
+ Note over Build Images: Build from scratch
+ Note over Build Images: Update constraints
+ Note over Build Images: Build PROD Images<br>[GITHUB_RUN_ID]
+ and
+ Note over Build Images: Build from scratch
+ Note over Build Images: Update constraints
+ Note over Build Images: Build CI Images<br>[GITHUB_RUN_ID]
+ end
+ par No CI image
+ Note over CI Build: Build info
+ and
+ Note over CI Build: Test always!
+ and
+ Note over CI Build: Helm tests
+ and
+ Note over CI Build: OpenAPI client gen
+ end
+ par 3.6, 3.7, 3.8
+ Build Images ->> GitHub Registry: Push CI Images
+ activate GitHub Registry
+ and
+ Build Images ->> GitHub Registry: Push PROD Images
+ end
+ opt If any step failed
+ Build Images -->> CI Build: Cancel triggering run
+ end
+ deactivate Build Images
+ Note over GitHub Registry: Tagged Images<br>[GITHUB_RUN_ID]
+ loop Wait for images
+ par 3.6, 3.7, 3.8
+ CI Build ->> CI Build: Pull CI Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ and
+ CI Build ->> CI Build: Pull PROD Images
+ Note over CI Build: Wait for<br>[GITHUB_RUN_ID]
+ end
+ end
+ par 3.6, 3.7, 3.8
+ GitHub Registry ->> CI Build: Pull CI Images
+ and
+ GitHub Registry ->> CI Build: Pull PROD Images
+ end
+ deactivate GitHub Registry
+ par 3.6, 3.7, 3.8
+ Note over CI Build: Run static checks
+ and
+ Note over CI Build: Build docs
+ and
+ Note over CI Build: Spell check docs
+ and
+ Note over CI Build: Backport packages
+ and
+ Note over CI Build: Tests
+ and
+ Note over CI Build: Kubernetes Tests
+ end
+ Note over CI Build: Quarantined tests
+ Note over CI Build: Merge Coverage
+ CI Build -->> Coverage.io: Upload Coverage
+ par 3.6, 3.7, 3.8
+ CI Build ->> Airflow Repo: Push Constraints
+ end
+ CI Build ->> Airflow Repo: Push Nightly Tag
+ CI Build -->> Airflow Repo: Status Check for CI Build
+ deactivate Airflow Repo
+ deactivate CI Build
diff --git a/images/ci/scheduled_ci_flow.png b/images/ci/scheduled_ci_flow.png
new file mode 100644
index 0000000..553ccfc
Binary files /dev/null and b/images/ci/scheduled_ci_flow.png differ
diff --git a/scripts/ci/pre_commit/pre_commit_mermaid.sh b/scripts/ci/pre_commit/pre_commit_mermaid.sh
new file mode 100755
index 0000000..fc52c91
--- /dev/null
+++ b/scripts/ci/pre_commit/pre_commit_mermaid.sh
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -euo pipefail
+AIRFLOW_SOURCES="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && cd ../../../ && pwd )"
+export NO_TERMINAL_OUTPUT_FROM_SCRIPTS="true"
+
+if ! command -v npm; then
+ echo 'You need to have npm installed in order to generate .mermaid graphs automatically.'
+ echo
+ # Do not fail. This is no problem if those images are not regenerated.
+ exit 0
+fi
+
+tmp_file=$(mktemp)
+
+cd "${AIRFLOW_SOURCES}"
+
+MERMAID_INSTALLATION_DIR="${AIRFLOW_SOURCES}/.build/mermaid/"
+MERMAID_CONFIG_FILE="${MERMAID_INSTALLATION_DIR}/mermaid-config.json"
+MERMAID_CLI="${MERMAID_INSTALLATION_DIR}/node_modules/.bin/mmdc"
+export NODE_VIRTUAL_ENV="${MERMAID_INSTALLATION_DIR}"
+
+if [[ -f "${MERMAID_CLI}" ]]; then
+ MERMAID_INSTALLED="true"
+else
+ MERMAID_INSTALLED="false"
+fi
+
+# shellcheck disable=SC2064
+trap "rm -rf ${tmp_file}" EXIT
+
+for file in "${@}"
+do
+ basename_file=${AIRFLOW_SOURCES}/"$(dirname "${file}")/$(basename "${file}" .mermaid)"
+ md5sum_file="${basename_file}.md5"
+ if ! diff "${md5sum_file}" <(md5sum "${file}"); then
+ if [[ ${MERMAID_INSTALLED} != "true" ]]; then
+ echo "Installing mermaid"
+ mkdir -p "${MERMAID_INSTALLATION_DIR}/node_modules"
+ pushd "${MERMAID_INSTALLATION_DIR}"
+ npm install mermaid.cli
+ cat >"${MERMAID_CONFIG_FILE}" <<EOF
+{
+ "theme": "default",
+ "themeCSS": ".label foreignObject { overflow: visible; }"
+}
+EOF
+ MERMAID_INSTALLED="true"
+ popd
+ fi
+ echo "Running generation for ${file}"
+ rm -f "${basename_file}.png"
+ rm -f "${basename_file}.md5"
+ # unfortunately mermaid does not handle well multiline comments and we need licence comment
+ # Stripping them manually :(. Multiline comments are coming in the future
+ # https://github.com/mermaid-js/mermaid/issues/1249
+ grep -v "^%%" <"${file}" > "${tmp_file}"
+ mkdir -p "${MERMAID_INSTALLATION_DIR}"
+
+ "${MERMAID_CLI}" \
+ -i "${tmp_file}" \
+ -w 2048 \
+ -o "${basename_file}.png" \
+ -c "${MERMAID_CONFIG_FILE}"
+ if [ -f "${basename_file}.png" ]; then
+ md5sum "${file}" >"${md5sum_file}"
+ echo
+ echo "Successfully generated: ${basename_file}.png"
+ echo "Successfully updated: ${basename_file}.md5"
+ echo
+ echo "Please add both files and commit them to repository"
+ echo
+ else
+ 1>&2 echo
+ 1>&2 echo "ERROR: Could not generate ${basename_file}.png"
+ 1>&2 echo
+ exit 1
+ fi
+ else
+ echo "Skip regenerating file ${file} -> it's hash did not change in ${md5sum_file}"
+ fi
+done
[airflow] 44/44: Synchronize INTHEWILD.md with master
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 00f320fb50f72c8458e139340ab14ab526f88a81
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Nov 14 16:07:25 2020 +0100
Synchronize INTHEWILD.md with master
---
INTHEWILD.md | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/INTHEWILD.md b/INTHEWILD.md
index d1c914c..fe65c23 100644
--- a/INTHEWILD.md
+++ b/INTHEWILD.md
@@ -40,10 +40,12 @@ Currently, **officially** using Airflow:
1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)]
1. [Airfinity](https://www.airfinity.com) [[@sibowyer](https://github.com/sibowyer)]
1. [Airtel](https://www.airtel.in/) [[@harishbisht](https://github.com/harishbisht)]
+1. [Akamai](https://www.akamai.com/) [[@anirudhbagri](https://github.com/anirudhbagri)]
1. [Akamas](https://akamas.io) [[@GiovanniPaoloGibilisco](https://github.com/GiovanniPaoloGibilisco), [@lucacavazzana](https://github.com/lucacavazzana)]
1. [Alan](https://alan.eu) [[@charles-go](https://github.com/charles-go)]
1. [AloPeyk](https://alopeyk.com) [[@blcksrx](https://github.com/blcksrx), [@AloPeyk](https://github.com/AloPeyk)]
1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)]
+1. [American Family Insurance](https://www.amfam.com/about) [[@di1eep](https://github.com/di1eep)]
1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)]
1. [Arquivei](https://www.arquivei.com.br/) [[@arquivei](https://github.com/arquivei)]
1. [Arrive](https://www.arrive.com/)
@@ -51,7 +53,9 @@ Currently, **officially** using Airflow:
1. [Astronomer](http://www.astronomer.io) [[@schnie](https://github.com/schnie), [@ashb](https://github.com/ashb), [@kaxil](https://github.com/kaxil), [@dimberman](https://github.com/dimberman), [@andriisoldatenko](https://github.com/andriisoldatenko), [@ryw](https://github.com/ryw), [@ryanahamilton](https://github.com/ryanahamilton), [@jhtimmins](https://github.com/jhtimmins), [@vikramkoka](https://github.com/vikramkoka)]
1. [Auth0](https://auth0.com) [[@scottypate](https://github.com/scottypate)], [[@dm03514](https://github.com/dm03514)], [[@karangale](https://github.com/karangale)]
1. [Automattic](https://automattic.com/) [[@anandnalya](https://github.com/anandnalya), [@bperson](https://github.com/bperson), [@khrol](https://github.com/Khrol), [@xyu](https://github.com/xyu)]
+1. [Avesta Technologies](https://avestatechnologies.com) [[@TheRum](https://github.com/TheRum)]
1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]
+1. [Axesor designeted activity company](https://www.axesor.es/)
1. [Azri Solutions](http://www.azrisolutions.com/) [[@userimack](https://github.com/userimack)]
1. [BBM](https://www.bbm.com/)
1. [Bagelcode](https://site.bagelcode.com/)
@@ -71,6 +75,7 @@ Currently, **officially** using Airflow:
1. [Blacklane](https://www.blacklane.com) [[@serkef](https://github.com/serkef)]
1. [Bloc](https://www.bloc.io) [[@dpaola2](https://github.com/dpaola2)]
1. [Bloomberg](https://www.techatbloomberg.com) [[@dimberman](https://github.com/dimberman)]
+1. [Bloomreach](https://www.bloomreach.com/) [[@neelborooah](https://github.com/neelborooah) & [@debodirno](https://github.com/debodirno) & [@ayushmnnit](https://github.com/ayushmnnit)]
1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)]
1. [BlueApron](https://www.blueapron.com) [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)]
1. [Bluecore](https://www.bluecore.com) [[@JLDLaughlin](https://github.com/JLDLaughlin)]
@@ -102,6 +107,7 @@ Currently, **officially** using Airflow:
1. [Clairvoyant](https://clairvoyantsoft.com) [[@shekharv](https://github.com/shekharv)]
1. [Classmethod, Inc.](https://classmethod.jp/) [[@shoito](https://github.com/shoito)]
1. [Cleartax](https://cleartax.in/) [[@anks](https://github.com/anks) & [@codebuff](https://github.com/codebuff)]
+1. [Cloudera](https://www.cloudera.com/) [[@phraniiac](https://github.com/phraniiac) & [@VivekPemawat](https://github.com/VivekPemawat)]
1. [Clover Health](https://www.cloverhealth.com) [[@gwax](https://github.com/gwax) & [@vansivallab](https://github.com/vansivallab)]
1. [Colgate-Palmolive](https://www.colgatepalmolive.com/) [[@fhoda](https://github.com/fhoda)]
1. [Collectivehealth Inc.](https://www.collectivehealth.com) [[@retornam](https://github.com/retornam)]
@@ -125,6 +131,7 @@ Currently, **officially** using Airflow:
1. [DataFox](https://www.datafox.com/) [[@sudowork](https://github.com/sudowork)]
1. [DataSprints](https://datasprints.com/) [[@lopesdiego12](https://github.com/lopesdiego12) & [@rafaelsantanaep](https://github.com/rafaelsantanaep)]
1. [Datamaran](https://www.datamaran.com) [[@valexharo](https://github.com/valexharo)]
+1. [Datumo](https://datumo.io) [[@michalmisiewicz](https://github.com/michalmisiewicz)]
1. [Dentsu Inc.](http://www.dentsu.com/) [[@bryan831](https://github.com/bryan831) & [@loozhengyuan](https://github.com/loozhengyuan)]
1. [Deseret Digital Media](http://deseretdigital.com/) [[@formigone](https://github.com/formigone)
1. [Digital First Media](http://www.digitalfirstmedia.com/) [[@duffn](https://github.com/duffn) & [@mschmo](https://github.com/mschmo) & [@seanmuth](https://github.com/seanmuth)]
@@ -136,12 +143,16 @@ Currently, **officially** using Airflow:
1. [Dynata](https://www.dynata.com) [[@neil3handari](https://github.com/neil3handari)]
1. [EBANX](https://www.ebanx.com/) [[@estevammr](https://github.com/estevammr) & [@nathangngencissk](https://github.com/nathangngencissk) & [@raafaadg](https://github.com/raafaadg) & [@whrocha](https://github.com/whrocha)]
1. [Easy Taxi](http://www.easytaxi.com/) [[@caique-lima](https://github.com/caique-lima) & [@diraol](https://github.com/diraol)]
+1. [Elai Data](https://www.elaidata.com/) [[@lgov](https://github.com/lgov)]
1. [EllisDon](http://www.ellisdon.com/) [[@d2kalra](https://github.com/d2kalra) & [@zbasama](https://github.com/zbasama)]
1. [Endesa](https://www.endesa.com) [[@drexpp](https://github.com/drexpp)]
1. [Enigma](https://www.enigma.com) [[@hydrosquall](https://github.com/hydrosquall)]
1. [Etsy](https://www.etsy.com) [[@mchalek](https://github.com/mchalek)]
+1. [Everis](https://www.everis.com) [[@diegobenedicto](https://github.com/diegobenedicto)]
1. [Everlane](https://everlane.com) [[@NickBenthem](https://github.com/NickBenthem)]
1. [Experity (formerly DocuTAP)](https://www.experityhealth.com/) [[@cloneluke](https://github.com/cloneluke) & [@tobyjoliver](https://github.com/tobyjoliver)]
+1. [FanDuel](https://www.fanduel.com/)
+1. [Farfetch](https://github.com/farfetch) [[@davidmarques78](https://github.com/davidmarques78)]
1. [Fathom Health](https://www.fathomhealth.co/)
1. [Firestone Inventing](https://www.hsmap.com/) [[@zihengCat](https://github.com/zihengCat)]
1. [Flipp](https://www.flipp.com) [[@sethwilsonwishabi](https://github.com/sethwilsonwishabi)]
@@ -239,6 +250,7 @@ Currently, **officially** using Airflow:
1. [Multiply](https://www.multiply.com) [[@nrhvyc](https://github.com/nrhvyc)]
1. [NEXT Trucking](https://www.nexttrucking.com/) [[@earthmancash2](https://github.com/earthmancash2), [@kppullin](https://github.com/kppullin)]
1. [National Bank of Canada](https://nbc.ca) [[@brilhana](https://github.com/brilhana)]
+1. [Nav, Inc.](https://nav.com/) [[@tigerjz32](https://github.com/tigerjz32)]
1. [Neoway](https://www.neoway.com.br/) [[@neowaylabs](https://github.com/orgs/NeowayLabs/people)]
1. [Nerdwallet](https://www.nerdwallet.com)
1. [New Relic](https://www.newrelic.com) [[@marcweil](https://github.com/marcweil)]
@@ -271,7 +283,7 @@ Currently, **officially** using Airflow:
1. [Plaid](https://www.plaid.com/) [[@plaid](https://github.com/plaid), [@AustinBGibbons](https://github.com/AustinBGibbons) & [@jeeyoungk](https://github.com/jeeyoungk)]
1. [Playbuzz](https://www.playbuzz.com/) [[@clintonboys](https://github.com/clintonboys) & [@dbn](https://github.com/dbn)]
1. [Playsimple Games](https://playsimple.in/) [[@joshi95](https://github.com/joshi95)]
-1. [Polidea](https://www.polidea.com/) [[@potiuk](https://github.com/potiuk), [@mschickensoup](https://github.com/mschickensoup), [@mik-laj](https://github.com/mik-laj), [@turbaszek](https://github.com/turbaszek), [@michalslowikowski00](https://github.com/michalslowikowski00), [@olchas](https://github.com/olchas)], [@debek](https://github.com/debek)
+1. [Polidea](https://www.polidea.com/) [[@potiuk](https://github.com/potiuk), [@mschickensoup](https://github.com/mschickensoup), [@mik-laj](https://github.com/mik-laj), [@turbaszek](https://github.com/turbaszek), [@michalslowikowski00](https://github.com/michalslowikowski00), [@olchas](https://github.com/olchas), [@debek](https://github.com/debek), [@FHoffmannCode](https://github.com/FHoffmannCode), [@TobKed](https://github.com/TobKed)]
1. [Poshmark](https://www.poshmark.com)
1. [Postmates](http://www.postmates.com) [[@syeoryn](https://github.com/syeoryn)]
1. [Premise](http://www.premise.com) [[@jmccallum-premise](https://github.com/jmccallum-premise)]
@@ -286,6 +298,7 @@ Currently, **officially** using Airflow:
1. [Quizlet](https://quizlet.com) [[@quizlet](https://github.com/quizlet)]
1. [Quora](https://www.quora.com/)
1. [REA Group](https://www.rea-group.com/)
+1. [Raisin](https://www.raisin.com/) [[@davido912](https://github.com/davido912)]
1. [Rakuten](https://www.rakuten.com)
1. [Rapido](https://rapido.bike/) [[@ChethanUK](https://github.com/ChethanUK)]
1. [Raízen](https://www.raizen.com.br/) [[@rudlac](https://github.com/rudlac) & [@guifneves](https://github.com/guifneves)]
@@ -320,6 +333,7 @@ Currently, **officially** using Airflow:
1. [Stripe](https://stripe.com) [[@jbalogh](https://github.com/jbalogh)]
1. [Strongmind](https://www.strongmind.com) [[@tomchapin](https://github.com/tomchapin) & [@wongstein](https://github.com/wongstein)]
1. [Surfline](https://www.surfline.com/) [[@jawang35](https://github.com/jawang35)]
+1. [Syapse](https://www.syapse.com/) [[@zedmor](https://github.com/zedmor)]
1. [T2 Systems](http://t2systems.com) [[@unclaimedpants](https://github.com/unclaimedpants)]
1. [TEK](https://www.tek.fi/en) [[@telac](https://github.com/telac)]
1. [THE ICONIC](https://www.theiconic.com.au/) [[@revathijay](https://github.com/revathijay), [@ilikedata](https://github.com/ilikedata)]
@@ -348,6 +362,7 @@ Currently, **officially** using Airflow:
1. [USC Graduate School, University of Southern California](https://graduateschool.usc.edu/) [[@abhilash1in](https://github.com/abhilash1in), [@sudarshansunder](https://github.com/sudarshansunder)]
1. [Ubisoft](https://www.ubisoft.com/) [[@Walkoss](https://github.com/Walkoss)]
1. [Udacity](https://www.udacity.com/) [[@dandikunited](https://github.com/DandikUnited), [@simon-uc](https://github.com/simon-uc)]
+1. [Umami Collective](https://umamicollective.com) [[@juanuicich](https://github.com/juanuicich)]
1. [United Airlines](https://www.united.com/) [[@ilopezfr](https://github.com/ilopezfr)]
1. [Upsight](https://www.upsight.com)
1. [VeeR VR](https://veer.tv) [[@pishilong](https://github.com/pishilong)]
@@ -364,6 +379,8 @@ Currently, **officially** using Airflow:
1. [Whistle Labs](http://www.whistle.com) [[@ananya77041](https://github.com/ananya77041)]
1. [Wildlifestudios](https://wildlifestudios.com/)
1. [WiseBanyan](https://wisebanyan.com/)
+1. [WixAnswers](https://www.wixanswers.com/) [[@eladkal](https://github.com/eladkal)]
+1. [Wix](https://www.wix.com/) [[@eladkal](https://github.com/eladkal)]
1. [Wooga](https://www.wooga.com/)
1. [WorldRemit](https://www.worldremit.com/) [[@boittega](https://github.com/boittega)]
1. [Wrike](https://www.wrike.com) [[@eliseealex](https://github.com/eliseealex) & [teoretic6](https://github.com/Teoretic6)]
@@ -387,3 +404,4 @@ Currently, **officially** using Airflow:
1. [imgix](https://www.imgix.com/) [[@dclubb](https://github.com/dclubb)]
1. [liligo](http://liligo.com/) [[@tromika](https://github.com/tromika)]
1. [proton.ai](https://proton.ai/) [[@prmsolutions](https://github.com/prmsolutions)]
+1. [uSmart Securities](https://www.usmartsecurities.com/hk/en/) [[@yangrong688](https://github.com/yangrong688)]
[airflow] 26/44: Update to new helm stable repo (#12137)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 2ddafd037a1f919aaabff4708c69b9766d2225b2
Author: Ash Berlin-Taylor <as...@firemirror.com>
AuthorDate: Fri Nov 6 16:05:18 2020 +0000
Update to new helm stable repo (#12137)
Switch out deprecated helm repo for new stable repo.
- https://www.cncf.io/blog/2020/11/05/helm-chart-repository-deprecation-update/
- https://helm.sh/docs/faq/#i-am-getting-a-warning-about-unable-to-get-an-update-from-the-stable-chart-repository
(cherry picked from commit 128c9918b5f79cb46a563b77e803c29548c4319c)
---
chart/README.md | 2 +-
chart/requirements.lock | 6 +++---
chart/tests/conftest.py | 4 +---
scripts/ci/libraries/_kind.sh | 2 +-
4 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/chart/README.md b/chart/README.md
index e7ef973..7d23ce3 100644
--- a/chart/README.md
+++ b/chart/README.md
@@ -37,7 +37,7 @@ cluster using the [Helm](https://helm.sh) package manager.
To install this repository from source (using helm 3)
```bash
kubectl create namespace airflow
-helm repo add stable https://kubernetes-charts.storage.googleapis.com
+helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace airflow
```
diff --git a/chart/requirements.lock b/chart/requirements.lock
index 715458e..3f3c34a 100644
--- a/chart/requirements.lock
+++ b/chart/requirements.lock
@@ -1,6 +1,6 @@
dependencies:
- name: postgresql
- repository: https://kubernetes-charts.storage.googleapis.com/
+ repository: https://charts.helm.sh/stable/
version: 6.3.12
-digest: sha256:e8d53453861c590e6ae176331634c9268a11cf894be17ed580fa2b347101be97
-generated: "2020-10-27T21:16:13.0063538Z"
+digest: sha256:58d88cf56e78b2380091e9e16cc6ccf58b88b3abe4a1886dd47cd9faef5309af
+generated: "2020-11-04T15:59:36.967913-08:00"
diff --git a/chart/tests/conftest.py b/chart/tests/conftest.py
index db1f22f..789f311 100644
--- a/chart/tests/conftest.py
+++ b/chart/tests/conftest.py
@@ -26,7 +26,5 @@ def upgrade_helm():
"""
Upgrade Helm repo
"""
- subprocess.check_output(
- ["helm", "repo", "add", "stable", "https://kubernetes-charts.storage.googleapis.com/"]
- )
+ subprocess.check_output(["helm", "repo", "add", "stable", "https://charts.helm.sh/stable/"])
subprocess.check_output(["helm", "dep", "update", sys.path[0]])
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 41048fc..6194742 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -336,7 +336,7 @@ function kind::deploy_airflow_with_helm() {
kubectl create namespace "${HELM_AIRFLOW_NAMESPACE}"
kubectl create namespace "test-namespace"
pushd "${AIRFLOW_SOURCES}/chart" || exit 1
- helm repo add stable https://kubernetes-charts.storage.googleapis.com
+ helm repo add stable https://charts.helm.sh/stable/
helm dep update
helm install airflow . --namespace "${HELM_AIRFLOW_NAMESPACE}" \
--set "defaultAirflowRepository=${DOCKERHUB_USER}/${DOCKERHUB_REPO}" \
[airflow] 14/44: Turns failure of PR label when approved action
into warning (#12017)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8e582911f9576fad1827d613a7980700c3c0080e
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 2 00:01:20 2020 +0100
Turns failure of PR label when approved action into warning (#12017)
Sometimes (quite often really) when PR gets approved, the PR
gets merged rather quickly, without waiting for result of this
action. Or a new PR gets pushed quickly. In those cases PR will
not be found. But this is usually not a problem then and rather
than failing, we should simply print a warning and exit.
(cherry picked from commit a7a7cf22172ca92edfb62c373993ed974a298496)
---
.github/workflows/label_when_reviewed_workflow_run.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/label_when_reviewed_workflow_run.yml b/.github/workflows/label_when_reviewed_workflow_run.yml
index 0fcab12..56bc835 100644
--- a/.github/workflows/label_when_reviewed_workflow_run.yml
+++ b/.github/workflows/label_when_reviewed_workflow_run.yml
@@ -72,7 +72,7 @@ jobs:
./scripts/ci/selective_ci_checks.sh
fi
- name: "Label when approved by committers for PRs that require full tests"
- uses: TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 # v1.2
+ uses: TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 # v1.3
id: label-full-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -101,7 +101,7 @@ jobs:
[the run](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
"}
- name: "Label when approved by committers for PRs that do not require full tests"
- uses: TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 # v1.2
+ uses: TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 # v1.3
id: label-simple-test-prs-when-approved-by-commiters
if: >
steps.selective-checks.outputs.run-tests == 'true' &&
@@ -116,7 +116,7 @@ jobs:
Airflow. The committers might merge it or can add a label 'full tests needed' and re-run it
to run all tests if they see it is needed!
- name: "Label when approved by committers for PRs that do not require tests at all"
- uses: TobKed/label-when-approved-action@7872312da76508d29f98d4fa68843ea91754cc59 # v1.2
+ uses: TobKed/label-when-approved-action@4c5190fec5661e98d83f50bbd4ef9ebb48bd1194 # v1.3
id: label-no-test-prs-when-approved-by-commiters
if: steps.selective-checks.outputs.run-tests != 'true'
with:
[airflow] 16/44: Fixes documentation-only selective checks (#12038)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ad8aa653f874141084973b00d1f73b1f1f6e5402
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 2 15:16:24 2020 +0100
Fixes documentation-only selective checks (#12038)
There was a problem that documentation-only checks triggered
selective checks without docs build (they resulted in
basic-checks-only and no images being built.
This occured for example in #12025
This PR fixes it by adding image-build and docs-build as two
separate outputs.
(cherry picked from commit adbf764ade6916b505c3238697bac10f98bfa6eb)
---
.github/workflows/build-images-workflow-run.yml | 4 +-
.github/workflows/ci.yml | 41 +++++----------
scripts/ci/selective_ci_checks.sh | 69 +++++++++++++++++--------
3 files changed, 63 insertions(+), 51 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index 6099727..a3277a7 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -201,7 +201,7 @@ jobs:
defaultPythonVersion: ${{ steps.selective-checks.outputs.default-python-version }}
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
- basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only }}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
if: >
needs.cancel-workflow-runs.outputs.buildImages == 'true'
steps:
@@ -266,7 +266,7 @@ jobs:
image-type: [CI, PROD]
fail-fast: true
if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
+ needs.build-info.outputs.image-build == 'true' &&
needs.cancel-workflow-runs.outputs.buildImages == 'true'
env:
BACKEND: postgres
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a5457c0..c217bdc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -92,6 +92,8 @@ jobs:
run-tests: ${{ steps.selective-checks.outputs.run-tests }}
run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
basic-checks-only: ${{ steps.selective-checks.outputs.basic-checks-only }}
+ image-build: ${{ steps.selective-checks.outputs.image-build }}
+ docs-build: ${{ steps.selective-checks.outputs.docs-build }}
needs-helm-tests: ${{ steps.selective-checks.outputs.needs-helm-tests }}
needs-api-tests: ${{ steps.selective-checks.outputs.needs-api-tests }}
pullRequestNumber: ${{ steps.source-run-info.outputs.pullRequestNumber }}
@@ -158,8 +160,7 @@ jobs:
name: "Wait for CI images"
runs-on: ubuntu-latest
needs: [build-info]
- if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+ if: needs.build-info.outputs.image-build == 'true'
env:
BACKEND: sqlite
steps:
@@ -196,8 +197,7 @@ jobs:
env:
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+ if: needs.build-info.outputs.basic-checks-only == 'false'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }} )"
uses: actions/checkout@v2
@@ -233,8 +233,7 @@ jobs:
SKIP: "build,mypy,flake8,pylint,bats-in-container-tests"
MOUNT_LOCAL_SOURCES: "true"
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- if: >
- needs.build-info.outputs.basic-checks-only == 'true'
+ if: needs.build-info.outputs.basic-checks-only == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -266,8 +265,7 @@ jobs:
name: "Build docs"
runs-on: ubuntu-latest
needs: [build-info, ci-images]
- if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+ if: needs.build-info.outputs.docs-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -348,8 +346,7 @@ jobs:
RUN_TESTS: true
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
TEST_TYPE: ""
- if: >
- needs.build-info.outputs.run-tests == 'true'
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -401,8 +398,7 @@ jobs:
RUN_TESTS: true
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
TEST_TYPE: ""
- if: >
- needs.build-info.outputs.run-tests == 'true'
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -451,8 +447,7 @@ jobs:
RUN_TESTS: true
TEST_TYPES: "${{needs.build-info.outputs.testTypes}}"
TEST_TYPE: ""
- if: >
- needs.build-info.outputs.run-tests == 'true'
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -506,8 +501,7 @@ jobs:
TEST_TYPE: ""
NUM_RUNS: 10
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: >
- needs.build-info.outputs.run-tests == 'true'
+ if: needs.build-info.outputs.run-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -588,8 +582,7 @@ jobs:
env:
BACKEND: sqlite
PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
- if: >
- needs.build-info.outputs.basic-checks-only == 'false'
+ if: needs.build-info.outputs.image-build == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
@@ -633,8 +626,7 @@ jobs:
KUBERNETES_VERSION: "${{ matrix.kubernetes-version }}"
KIND_VERSION: "${{ needs.build-info.outputs.defaultKindVersion }}"
HELM_VERSION: "${{ needs.build-info.outputs.defaultHelmVersion }}"
- if: >
- needs.build-info.outputs.run-kubernetes-tests == 'true'
+ if: needs.build-info.outputs.run-kubernetes-tests == 'true'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -744,7 +736,6 @@ jobs:
- ci-images
- docs
if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
(github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' ) &&
github.event_name != 'schedule'
strategy:
@@ -780,9 +771,7 @@ jobs:
- ci-images
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
- (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' )
+ if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} : merge commit ${{ github.merge_commit_sha }})"
uses: actions/checkout@v2
@@ -815,9 +804,7 @@ jobs:
- tests-mysql
- tests-postgres
- tests-kubernetes
- if: >
- needs.build-info.outputs.basic-checks-only == 'false' &&
- (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test' )
+ if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/v1-10-test'
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
diff --git a/scripts/ci/selective_ci_checks.sh b/scripts/ci/selective_ci_checks.sh
index 10b74df..0faa541 100755
--- a/scripts/ci/selective_ci_checks.sh
+++ b/scripts/ci/selective_ci_checks.sh
@@ -139,7 +139,6 @@ function get_changed_files() {
readonly CHANGED_FILES
}
-
function run_tests() {
initialization::ga_output run-tests "${@}"
}
@@ -160,11 +159,18 @@ function needs_python_scans() {
initialization::ga_output needs-python-scans "${@}"
}
-
function set_test_types() {
initialization::ga_output test-types "${@}"
}
+function set_docs_build() {
+ initialization::ga_output docs-build "${@}"
+}
+
+function set_image_build() {
+ initialization::ga_output image-build "${@}"
+}
+
function set_basic_checks_only() {
initialization::ga_output basic-checks-only "${@}"
}
@@ -181,6 +187,8 @@ function set_outputs_run_everything_and_exit() {
run_kubernetes_tests "true"
set_test_types "${ALL_TESTS}"
set_basic_checks_only "false"
+ set_docs_build "true"
+ set_image_build "true"
exit
}
@@ -189,9 +197,10 @@ function set_outputs_run_all_tests() {
run_kubernetes_tests "true"
set_test_types "${ALL_TESTS}"
set_basic_checks_only "false"
+ set_image_build "true"
}
-function set_output_skip_all_tests_and_exit() {
+function set_output_skip_all_tests_and_docs_and_exit() {
needs_api_tests "false"
needs_helm_tests "false"
needs_javascript_scans "false"
@@ -200,6 +209,22 @@ function set_output_skip_all_tests_and_exit() {
run_kubernetes_tests "false"
set_test_types ""
set_basic_checks_only "true"
+ set_docs_build "false"
+ set_image_build "false"
+ exit
+}
+
+function set_output_skip_tests_but_build_images_and_exit() {
+ needs_api_tests "false"
+ needs_helm_tests "false"
+ needs_javascript_scans "false"
+ needs_python_scans "false"
+ run_tests "false"
+ run_kubernetes_tests "false"
+ set_test_types ""
+ set_basic_checks_only "true"
+ set_docs_build "true"
+ set_image_build "true"
exit
}
@@ -292,14 +317,15 @@ function check_if_docs_should_be_generated() {
echo "None of the docs changed"
else
image_build_needed="true"
+ docs_build_needed="true"
fi
}
AIRFLOW_SOURCES_TRIGGERING_TESTS=(
- "^airflow"
- "^chart"
- "^tests"
- "^kubernetes_tests"
+ "^airflow"
+ "^chart"
+ "^tests"
+ "^kubernetes_tests"
)
readonly AIRFLOW_SOURCES_TRIGGERING_TESTS
@@ -308,8 +334,13 @@ function check_if_tests_are_needed_at_all() {
show_changed_files
if [[ $(count_changed_files) == "0" ]]; then
- echo "None of the important files changed, Skipping tests"
- set_output_skip_all_tests_and_exit
+ if [[ ${image_build_needed} == "true" ]]; then
+ echo "No tests needed, Skipping tests but building images."
+ set_output_skip_tests_but_build_images_and_exit
+ else
+ echo "None of the important files changed, Skipping tests"
+ set_output_skip_all_tests_and_docs_and_exit
+ fi
else
image_build_needed="true"
tests_needed="true"
@@ -398,6 +429,7 @@ readonly FULL_TESTS_NEEDED
output_all_basic_variables
image_build_needed="false"
+docs_build_needed="false"
tests_needed="false"
kubernetes_tests_needed="false"
@@ -405,27 +437,20 @@ get_changed_files "${1}"
run_all_tests_if_environment_files_changed
check_if_docs_should_be_generated
check_if_helm_tests_should_be_run
-check_if_tests_are_needed_at_all
check_if_javascript_security_scans_should_be_run
check_if_python_security_scans_should_be_run
+check_if_tests_are_needed_at_all
get_count_all_files
get_count_kubernetes_files
calculate_test_types_to_run
+set_image_build "${image_build_needed}"
if [[ ${image_build_needed} == "true" ]]; then
set_basic_checks_only "false"
else
set_basic_checks_only "true"
fi
-
-if [[ ${tests_needed} == "true" ]]; then
- run_tests "true"
-else
- run_tests "false"
-fi
-
-if [[ ${kubernetes_tests_needed} == "true" ]]; then
- run_kubernetes_tests "true"
-else
- run_kubernetes_tests "false"
-fi
+set_basic_checks_only "${image_build_needed}"
+set_docs_build "${docs_build_needed}"
+run_tests "${tests_needed}"
+run_kubernetes_tests "${kubernetes_tests_needed}"
[airflow] 33/44: Remove popd which is a remnant from past (#12211)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 73187015e2dc64a76bf0d221e05ba7322eb436b8
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Nov 9 21:58:37 2020 +0100
Remove popd which is a remnant from past (#12211)
(cherry picked from commit d8abee690823b7973e5561f2725dc7b79467bfdf)
---
scripts/ci/libraries/_start_end.sh | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/ci/libraries/_start_end.sh b/scripts/ci/libraries/_start_end.sh
index f4d8bfe..8fc7e5d 100644
--- a/scripts/ci/libraries/_start_end.sh
+++ b/scripts/ci/libraries/_start_end.sh
@@ -53,7 +53,6 @@ function start_end::dump_container_logs() {
echo "###########################################################################################"
docker logs "${container}" > "${dump_file}"
echo " Container ${container} logs dumped to ${dump_file}"
- popd || exit 1
echo "###########################################################################################"
}
[airflow] 04/44: Move Project focus and Principles higher in the
README (#11973)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6f6b5600c371783d91a1f1234c627f895a3bbef9
Author: Ry Walker <42...@users.noreply.github.com>
AuthorDate: Fri Oct 30 15:10:30 2020 -0400
Move Project focus and Principles higher in the README (#11973)
(cherry picked from commit 3c723e35a58b274962dc47e21cbb05389263d97a)
---
README.md | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
diff --git a/README.md b/README.md
index 5e638a5..ccac802 100644
--- a/README.md
+++ b/README.md
@@ -40,13 +40,13 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of contents**
+- [Project Focus](#project-focus)
+- [Principles](#principles)
- [Requirements](#requirements)
- [Getting started](#getting-started)
- [Installing from PyPI](#installing-from-pypi)
- [Official source code](#official-source-code)
- [Convenience packages](#convenience-packages)
-- [Project Focus](#project-focus)
-- [Principles](#principles)
- [User Interface](#user-interface)
- [Contributing](#contributing)
- [Who uses Apache Airflow?](#who-uses-apache-airflow)
@@ -57,6 +57,21 @@ Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+## Project Focus
+
+Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/).
+
+Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work.
+
+Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space.
+
+## Principles
+
+- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
+- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
+- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
+- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
+
## Requirements
Apache Airflow is tested with:
@@ -150,21 +165,6 @@ All those artifacts are not official releases, but they are prepared using offic
Some of those artifacts are "development" or "pre-release" ones, and they are clearly marked as such
following the ASF Policy.
-## Project Focus
-
-Airflow works best with workflows that are mostly static and slowly changing. When the structure is similar from one run to the next, it allows for clarity around unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](http://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/).
-
-Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent, and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [Xcom feature](https://airflow.apache.org/docs/stable/concepts.html#xcoms)). For high-volume, data-intensive tasks, a best practice is to delegate to external services that specialize on that type of work.
-
-Airflow **is not** a streaming solution. Airflow is not in the [Spark Streaming](http://spark.apache.org/streaming/) or [Storm](https://storm.apache.org/) space.
-
-## Principles
-
-- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
-- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
-- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
-- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers.
-
## User Interface
- **DAGs**: Overview of all DAGs in your environment.