You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/01/21 20:19:43 UTC
[airflow] 18/44: Add changes from 1.10.14 (#12993)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 17d53ab7bc0d7d555479a0c96961cd714a82d16c
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Thu Dec 10 20:52:29 2020 +0000
Add changes from 1.10.14 (#12993)
This commit adds Changelog, Updating.md and replaces 1.10.13 to 1.10.14
across the codebase
(cherry picked from commit 969d3ea4f86a39807d696ca6f47c7cdf23526333)
---
.github/workflows/ci.yml | 1 +
BREEZE.rst | 117 ++++++++++++++++++++++++++++++++++++++++++++---
CONTRIBUTING.rst | 12 -----
IMAGES.rst | 67 +++++++++++++++++----------
README.md | 27 +++++++----
breeze | 3 +-
breeze-complete | 14 +++---
7 files changed, 181 insertions(+), 60 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ccfd50f..74670d3 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -289,6 +289,7 @@ jobs:
with:
name: airflow-documentation
path: "./files/documentation"
+ retention-days: 7
tests-helm:
timeout-minutes: 20
diff --git a/BREEZE.rst b/BREEZE.rst
index 43705f9..1ed3cfe 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -541,6 +541,74 @@ dependencies). If you work offline and do not want to rebuild the images when ne
``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section.
+Preparing packages
+------------------
+
+Breeze can also be used to prepare airflow packages - both "apache-airflow" main package and
+provider packages.
+
+You can read more about testing provider packages in
+`TESTING.rst <TESTING.rst#running-tests-with-packages>`_
+
+There are several commands that you can run in Breeze to manage and build packages:
+
+* preparing Provider Readme files
+* preparing Airflow packages
+* preparing Provider packages
+
+Preparing provider readme files is part of the release procedure by the release managers
+and it is described in detail in `dev <dev/README.md>`_ .
+
+You can prepare provider packages - by default regular provider packages are prepared, but with
+``--backport`` flag you can prepare backport packages.
+
+The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
+before running, so you should run it before generating airflow package below as it will be removed.
+
+The below example builds provider packages in the wheel format.
+
+.. code-block:: bash
+
+ ./breeze prepare-provider-packages
+
+If you run this command without packages, you will prepare all packages, you can however specify
+providers that you would like to build. By default only ``wheel`` packages are prepared,
+but you can change it providing optional --package-format flag.
+
+
+.. code-block:: bash
+
+ ./breeze prepare-provider-packages --package-format=both google amazon
+
+You can also prepare backport provider packages, if you specify ``--backport`` flag. You can read more
+about backport packages in `dev <dev/README.md>`_
+
+.. code-block:: bash
+
+ ./breeze prepare-provider-packages --backports --package-format=both google amazon
+
+You can see all providers available by running this command:
+
+.. code-block:: bash
+
+ ./breeze prepare-provider-packages -- --help
+
+
+You can also prepare airflow packages using breeze:
+
+.. code-block:: bash
+
+ ./breeze prepare-airflow-packages
+
+This prepares airflow .whl package in the dist folder.
+
+Again, you can specify optional ``--package-format`` flag to build airflow packages.
+
+.. code-block:: bash
+
+ ./breeze prepare-airflow-packages --package-format=bot
+
+
Building Production images
--------------------------
@@ -1067,6 +1135,7 @@ This is the current syntax for `./breeze <./breeze>`_:
generate-constraints Generates pinned constraint files
push-image Pushes images to registry
initialize-local-virtualenv Initializes local virtualenv
+ prepare-airflow-packages Prepares airflow packages
setup-autocomplete Sets up autocomplete for breeze
start-airflow Starts Scheduler and Webserver and enters the shell
stop Stops the docker-compose environment
@@ -1142,7 +1211,7 @@ This is the current syntax for `./breeze <./breeze>`_:
Detailed usage for command: build-docs
- breeze build-docs [-- <EXTRA_ARGS>]
+ breeze build-docs
Builds Airflow documentation. The documentation is build inside docker container - to
maintain the same build environment for everyone. Appropriate sources are mapped from
@@ -1150,8 +1219,6 @@ This is the current syntax for `./breeze <./breeze>`_:
is generated ('docs/_build') are also mounted to the container - this way results of
the documentation build is available in the host.
- The possible extra args are: --docs-only, --spellcheck-only, --help
-
####################################################################################################
@@ -1187,8 +1254,7 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version. This happens at
image building time in production image and at container entering time for CI image. One of:
- 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
- 1.10.2 wheel none
+ 1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
@@ -1553,6 +1619,44 @@ This is the current syntax for `./breeze <./breeze>`_:
####################################################################################################
+ Detailed usage for command: prepare-airflow-packages
+
+
+ breeze prepare-airflow-packages [FLAGS]
+
+ Prepares airflow packages (sdist and wheel) in dist folder. Note that
+ prepare-provider-packages command cleans up the dist folder, so if you want also
+ to generate provider packages, make sure you run prepare-provider-packages first,
+ and prepare-airflow-packages second.
+
+ General form:
+
+ 'breeze prepare-airflow-packages
+
+ Flags:
+
+ --package-format PACKAGE_FORMAT
+
+ Chooses format of packages to prepare.
+
+ One of:
+
+ wheel,sdist,both
+
+ Default:
+
+ -v, --verbose
+ Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
+ debugging - when you run breeze with --verbose flags you will be able to see the commands
+ executed under the hood and copy&paste them to your terminal to debug them more easily.
+
+ Note that you can further increase verbosity and see all the commands executed by breeze
+ by running 'export VERBOSE_COMMANDS="true"' before running breeze.
+
+
+ ####################################################################################################
+
+
Detailed usage for command: setup-autocomplete
@@ -2105,8 +2209,7 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version. This happens at
image building time in production image and at container entering time for CI image. One of:
- 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
- 1.10.2 wheel none
+ 1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 8c4bf35..bc5661c 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -818,18 +818,6 @@ You can only run one of the steps via ``--spellcheck-only`` or ``--docs-only``.
./breeze build-docs
-or just to run spell-check
-
-.. code-block:: bash
-
- ./breeze build-docs -- --spellcheck-only
-
-or just to run documentation building
-
-.. code-block:: bash
-
- ./breeze build-docs
-
Also documentation is available as downloadable artifact in GitHub Actions after the CI builds your PR.
**Known issues:**
diff --git a/IMAGES.rst b/IMAGES.rst
index 513f8cf..94ad6fd 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -24,8 +24,8 @@ Airflow has two images (build from Dockerfiles):
* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
You can read more about building and using the production image in the
- `Production Deployments <docs/production-deployment.rst>`_ document. The image is built using
- `Dockerfile <Dockerfile>`_
+ `Production Deployments <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_ document.
+ The image is built using `Dockerfile <Dockerfile>`_
* CI image (Dockerfile.ci) - used for running tests and local development. The image is built using
`Dockerfile.ci <Dockerfile.ci>`_
@@ -64,7 +64,7 @@ The easiest way to build those images is to use `<BREEZE.rst>`_.
Note! Breeze by default builds production image from local sources. You can change it's behaviour by
providing ``--install-airflow-version`` parameter, where you can specify the
-tag/branch used to download Airflow package from in github repository. You can
+tag/branch used to download Airflow package from in GitHub repository. You can
also change the repository itself by adding ``--dockerhub-user`` and ``--dockerhub-repo`` flag values.
You can build the CI image using this command:
@@ -117,6 +117,16 @@ parameter to Breeze:
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=1.10.14
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
+ does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
+ ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
This will build the image using command similar to:
.. code-block:: bash
@@ -125,14 +135,6 @@ This will build the image using command similar to:
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
-.. note::
- On 30th of November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver.
- This resolver does not yet work with Apache Airflow and might leads to errors in installation -
- depends on your choice of extras. In order to install Airflow you need to either downgrade
- pip to version 20.2.4 ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
- ``--use-deprecated legacy-resolver`` to your pip install command.
-
-
You can also build production images from specific Git version via providing ``--install-airflow-reference``
parameter to Breeze (this time constraints are taken from the ``constraints-master`` branch which is the
HEAD of development for constraints):
@@ -142,6 +144,15 @@ HEAD of development for constraints):
pip install "https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt"
+You can also skip installing airflow by providing ``--install-airflow-version none`` parameter to Breeze:
+
+.. code-block:: bash
+
+ ./breeze build-image --python 3.7 --additional-extras=presto \
+ --production-image --install-airflow-version=none --install-from-local-files-when-building
+
+In this case you usually install airflow and all packages in ``docker-context-files`` folder.
+
Using cache during builds
=========================
@@ -276,8 +287,8 @@ production image from the local sources.
The image is primarily optimised for size of the final image, but also for speed of rebuilds - the
'airflow-build-image' segment uses the same technique as the CI builds for pre-installing PIP dependencies.
-It first pre-installs them from the right github branch and only after that final airflow installation is
-done from either local sources or remote location (PIP or github repository).
+It first pre-installs them from the right GitHub branch and only after that final airflow installation is
+done from either local sources or remote location (PIP or GitHub repository).
Customizing the image
.....................
@@ -409,7 +420,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | file has to be in docker context so |
| | | it's best to place such file in |
| | | one of the folders included in |
-| | | dockerignore . for example in the |
+| | | .dockerignore. for example in the |
| | | 'docker-context-files'. Note that the |
| | | location does not work for the first |
| | | stage of installation when the |
@@ -418,21 +429,31 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | set to true. Default location from |
| | | GitHub is used in this case. |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_LOCAL_PIP_WHEELS`` | ``false`` | If set to true, Airflow and it's |
+| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | ``constraints-master`` | reference (branch or tag) from GitHub |
+| | | repository from which constraints are |
+| | | used. By default it is set to |
+| | | ``constraints-master`` but can be |
+| | | ``constraints-1-10`` for 1.10.* versions |
+| | | or it could point to specific version |
+| | | for example ``constraints-1.10.12`` |
++------------------------------------------+------------------------------------------+------------------------------------------+
+| ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` | ``false`` | If set to true, Airflow and it's |
| | | dependencies are installed from locally |
| | | downloaded .whl files placed in the |
| | | ``docker-context-files``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | ``all`` | extras to install |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``INSTALL_AIRFLOW_VIA_PIP`` | ``false`` | If set to true, Airflow is installed via |
-| | | pip install. if you want to install |
+| ``INSTALL_FROM_PYPI`` | ``true`` | If set to true, Airflow is installed |
+| | | from pypi. If you want to install |
| | | Airflow from externally provided binary |
| | | package you can set it to false, place |
| | | the package in ``docker-context-files`` |
-| | | and set ``AIRFLOW_LOCAL_PIP_WHEELS`` to |
-| | | true. You have to also set to true the |
+| | | and set |
+| | | ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` to |
+| | | true. For this you have to also set the |
| | | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` flag |
+| | | to false |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
| | | from the GitHub of Apache Airflow |
@@ -530,7 +551,7 @@ Production images
-----------------
You can find details about using, building, extending and customising the production images in the
-`Latest documentation <https://airflow.readthedocs.io/en/latest/production-deployment.html>`_
+`Latest documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_
Image manifests
@@ -615,10 +636,8 @@ The entrypoint performs those operations:
Using, customising, and extending the production image
======================================================
-You can read more about using, customising, and extending the production image in the documentation:
-
-* [Stable docs](https://airflow.apache.org/docs/stable/production-deployment.html)
-* [Latest docs from master branch](https://airflow.readthedocs.io/en/latest/production-deployment.html
+You can read more about using, customising, and extending the production image in the
+`documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_.
Alpha versions of 1.10.10 production-ready images
=================================================
diff --git a/README.md b/README.md
index 79cceed..2d7edd6 100644
--- a/README.md
+++ b/README.md
@@ -82,18 +82,22 @@ Apache Airflow is tested with:
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
| MySQL | 5.7, 8 | 5.6, 5.7 |
| SQLite | latest stable | latest stable |
-| Kubernetes | 1.16.2, 1.17.0 | 1.16.2, 1.17.0 |
+| Kubernetes | 1.16.9, 1.17.5, 1.18.6 | 1.16.9, 1.17.5, 1.18.6 |
-**Note:** SQLite is used primarily for development purpose.
+**Note:** MariaDB and MySQL 5.x are unable to or have limitations with
+running multiple schedulers -- please see the "Scheduler" docs.
+
+**Note:** SQLite is used in Airflow tests. Do not use it in production.
### Additional notes on Python version requirements
* Stable version [requires](https://github.com/apache/airflow/issues/8162) at least Python 3.5.3 when using Python 3
## Getting started
+
Visit the official Airflow website documentation (latest **stable** release) for help with [installing Airflow](https://airflow.apache.org/installation.html), [getting started](https://airflow.apache.org/start.html), or walking through a more complete [tutorial](https://airflow.apache.org/tutorial.html).
-> Note: If you're looking for documentation for master branch (latest development branch): you can find it on [ReadTheDocs](https://airflow.readthedocs.io/en/latest/).
+> Note: If you're looking for documentation for master branch (latest development branch): you can find it on [s.apache.org/airflow-docs](https://s.apache.org/airflow-docs/).
For more information on Airflow's Roadmap or Airflow Improvement Proposals (AIPs), visit the [Airflow Wiki](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home).
@@ -117,6 +121,15 @@ correct Airflow tag/version/branch and python versions in the URL.
1. Installing just Airflow:
+NOTE!!!
+
+On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
+does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
+of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
+`pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
+`--use-deprecated legacy-resolver` to your pip install command.
+
+
```bash
pip install apache-airflow==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"
@@ -130,14 +143,12 @@ depends on your choice of extras. In order to install Airflow you need to either
pip to version 20.2.4 `pip upgrade --pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.
-
-2. Installing with extras (for example postgres,gcp)
```bash
-pip install apache-airflow[postgres,gcp]==1.10.14 \
+pip install apache-airflow[postgres,google]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.7.txt"
```
-For information on installing backport providers check https://airflow.readthedocs.io/en/latest/backport-providers.html.
+For information on installing backport providers check [/docs/backport-providers.rst][/docs/backport-providers.rst].
## Official source code
@@ -165,7 +176,7 @@ Those are - in the order of most common ways people install Airflow:
- [Docker Images](https://hub.docker.com/repository/docker/apache/airflow) to install airflow via
`docker` tool, use them in Kubernetes, Helm Charts, `docker-compose`, `docker swarm` etc. You can
read more about using, customising, and extending the images in the
- [Latest docs](https://airflow.readthedocs.io/en/latest/production-deployment.html), and
+ [Latest docs](https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html), and
learn details on the internals in the [IMAGES.rst](IMAGES.rst) document.
- [Tags in GitHub](https://github.com/apache/airflow/tags) to retrieve the git project sources that
were used to generate official source packages via git
diff --git a/breeze b/breeze
index e9a9557..2565ad5 100755
--- a/breeze
+++ b/breeze
@@ -1567,7 +1567,7 @@ ${CMDNAME} exec [-- <EXTRA_ARGS>]
webserver, workers, database console and interactive terminal.
"
export DETAILED_USAGE_BUILD_DOCS="
-${CMDNAME} build-docs [-- <EXTRA_ARGS>]
+${CMDNAME} build-docs
Builds Airflow documentation. The documentation is build inside docker container - to
maintain the same build environment for everyone. Appropriate sources are mapped from
@@ -1575,7 +1575,6 @@ ${CMDNAME} build-docs [-- <EXTRA_ARGS>]
is generated ('docs/_build') are also mounted to the container - this way results of
the documentation build is available in the host.
- The possible extra args are: --docs-only, --spellcheck-only, --help
"
readonly DETAILED_USAGE_BUILD_DOCS
export DETAILED_USAGE_BUILD_IMAGE="
diff --git a/breeze-complete b/breeze-complete
index df81093..5502eec 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -34,6 +34,7 @@ _breeze_allowed_mysql_versions="5.6 5.7"
_breeze_allowed_postgres_versions="9.6 10 11 12 13"
_breeze_allowed_kind_operations="start stop restart status deploy test shell k9s"
_breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm"
+_breeze_allowed_package_formats="wheel sdist both"
# shellcheck disable=SC2034
{
@@ -46,21 +47,16 @@ _breeze_allowed_test_types="All Core Integration Heisentests Postgres MySQL Helm
_breeze_default_postgres_version=$(echo "${_breeze_allowed_postgres_versions}" | awk '{print $1}')
_breeze_default_mysql_version=$(echo "${_breeze_allowed_mysql_versions}" | awk '{print $1}')
_breeze_default_test_type=$(echo "${_breeze_allowed_test_types}" | awk '{print $1}')
+ _breeze_default_package_format=$(echo "${_breeze_allowed_package_formats}" | awk '{print $1}')
}
_breeze_allowed_install_airflow_versions=$(cat <<-EOF
+1.10.14
1.10.13
1.10.12
1.10.11
1.10.10
1.10.9
-1.10.8
-1.10.7
-1.10.6
-1.10.5
-1.10.4
-1.10.3
-1.10.2
wheel
none
EOF
@@ -162,6 +158,7 @@ exec
generate-constraints
push-image
initialize-local-virtualenv
+prepare-airflow-packages
setup-autocomplete
start-airflow
stop
@@ -265,6 +262,9 @@ function breeze_complete::get_known_values_breeze() {
--test-type)
_breeze_known_values="${_breeze_allowed_test_types}"
;;
+ --package-format)
+ _breeze_known_values="${_breeze_allowed_package_formats}"
+ ;;
*)
_breeze_known_values=""
;;